scadaproj/ZB.MOM.WW.LocalDb

ZB.MOM.WW.LocalDb

An embedded SQLite local cache with optional bidirectional async 2-node replication over gRPC, for the ZB.MOM.WW SCADA / OT family. It gives an application a fast, crash-safe local store that can optionally keep a second node in sync asynchronously — local writes never block on the peer, and a node that never enables replication is simply a fast local DB.

Change capture uses SQLite triggers writing a single HLC-stamped oplog inside the consumer's own write transaction (crash-consistent, non-bypassable). Convergence is last-writer-wins over a hybrid logical clock. All timestamps are UTC.

Packages (all 0.3.7; the feed carries up to 0.3.4 — 0.3.5 through 0.3.7 are unpublished)

Package	Contents
ZB.MOM.WW.LocalDb	Embedded DB host: ILocalDb SQL surface (ExecuteAsync / QueryAsync / transactions), RegisterReplicated(table[, baselineExistingRows]) (PK + BLOB + foreign-key validation, trigger install, oplog schema), DeregisterReplicated(table) (trigger removal + oplog/ledger prune), HybridLogicalClock, AddZbLocalDb() DI.
ZB.MOM.WW.LocalDb.Contracts	localdb_sync.v1 proto + generated gRPC types (packable; a future peer could be non-.NET).
ZB.MOM.WW.LocalDb.Replication	gRPC sync engine: passive LocalDbSyncService, initiating SyncBackgroundService, MaintenanceBackgroundService, LWW apply, watermark acks + pruning, snapshot resync, ISyncStatus, LocalDbMetrics, AddZbLocalDbReplication() / MapZbLocalDbSync().

Status: published to the Gitea feed (through 0.3.4); adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phases 1+2), both pinned to 0.3.4.

• 0.3.7 (unpublished) — a deleted row no longer comes back when the peer was offline longer than TombstoneRetention (scadaproj#16). Tombstones used to be pruned on age alone; once the delete's oplog row had also aged out, the snapshot (built from the row-version ledger) could not carry the delete, and the returning peer's older update re-created the row on both nodes. A tombstone is now pruned only when it is past retention and the peer holds it — no outstanding snapshot debt (database-wide or for its table) and, for a local delete, no longer in the unacked oplog — and its retention clock restarts when it is delivered (delta ack or snapshot discharge). New optional TombstoneHardCap (default off) bounds how long an undelivered tombstone is kept. Wire-compatible: no proto, lib_schema_version or on-disk schema change.
• 0.3.6 (unpublished) — code-review fixes (scadaproj#36–#39, #46, #50). Wire-compatible: no proto change, no lib_schema_version bump, no on-disk schema change.
  • The passive endpoint enforces ApiKey itself (Bearer, constant-time, before the handshake).
  • Snapshots deliver FK-linked rows in any order (parents first; FK-failing rows deferred and retried; true orphans dead-lettered at completion), with the held-back rows capped by the new MaxSnapshotDeferredRows.
  • Remote rows are upserted on the primary key (ON CONFLICT(<pk>) DO UPDATE), not INSERT OR REPLACE, so a secondary-UNIQUE clash is a dead-letter, not a silent delete.
  • FailClosedOnDrift covers snapshot rows.
  • Snapshot-path dead-letters are counted in localdb.sync.dead_lettered.
  • Column names containing ' replicate (they broke the apply/snapshot SQL).
  • RegisterReplicated refuses an AUTOINCREMENT primary key (see Limitations) — the one change that can refuse a schema an earlier version accepted.
• 0.3.5 (unpublished) — Microsoft.Data.Sqlite 10.0.7 → 10.0.13 (efcore#39009 pool fault) and the SQLitePCLRaw.lib.e_sqlite3 transitive pin 2.1.12 → 2.1.13. No wire or schema change.
• 0.3.4 (2026-08-20) — a converged pair no longer re-streams a full snapshot on every reconnect once the sender's oplog has been ack-pruned to empty. Found on the ScadaBridge docker rig (its register row 106): a rebuilt node (watermark 0) was healed by a snapshot stamped as_of_seq = MAX(seq) — which is 0 on an empty oplog — so its MAX-guarded watermark stayed 0, and the healthy side's empty-oplog horizon rule (0.1.2: streamable from last_acked_seq + 1) saw a gap on every later handshake and re-sent the whole database (127k rows per restart on the rig) until it happened to write a new row. The as-of is now max(MAX(seq), last_acked_seq): every delta pruned below last_acked_seq is already folded into the rows the snapshot carries, so that is the floor it may claim. Sender-side only, wire-compatible, no schema change; pinned by WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot.
• 0.3.3 (2026-08-20) — two latent apply-path hazards from the ScadaBridge adversarial architecture review (P4-7). Neither had a live trigger in any current consumer; both are the kind that surface as silent divergence, not as an error. Wire-compatible: no proto change, no lib_schema_version bump, no on-disk schema change.
  • A replicated table may no longer be the parent of a row-mutating ON DELETE foreign key. The apply path writes remote rows with INSERT OR REPLACE, and REPLACE resolves a PK conflict by DELETING the existing row first. Connections run PRAGMA foreign_keys=ON, so that implicit delete fires the child's ON DELETE CASCADE / SET NULL / SET DEFAULT — while the capture guard is raised. The child rows change on the applying node ONLY, produce no oplog entry, and the pair diverges with nothing able to detect it (LWW compares only rows it was told about). RegisterReplicated now inspects every foreign key in the database and refuses such a table with a LocalDbRegistrationException naming the offending child column and action. RESTRICT / NO ACTION remain fine — they abort the statement instead of mutating rows behind capture's back, which the apply path already handles as per-entry poison. Checked at registration; SQLite cannot add a foreign key to an existing table, so a child table created later is caught at the next startup's registration rather than at the DDL.
  • An entry's row write and its ledger write are now atomic with each other. They are two statements, and SQLite aborts only the failing one — so a statement-level fault between them (the case the per-entry poison catch exists for) committed a half-applied entry: a tombstone whose table row was gone while __localdb_row_version still called it live, which the snapshot streamer can only warn about and skip, so the peer never learned the row died; or an upsert whose new row carried the OLD version and therefore lost the next comparison it should have won. The pair is wrapped in a per-entry SAVEPOINT that is rolled back before dead-lettering, so a poison entry leaves both writes or neither and the rest of the batch is untouched.
• 0.3.2 (2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, no lib_schema_version bump, no on-disk schema change — the keepalive is transport-level and the rest is local policy.
  • A live session now ends itself when a cap prune eats deltas it has not sent. The pump reads seq > sentThruSeq, which resumes SILENTLY above a pruned hole; the peer acks the later seqs and the pair is durably divergent for the whole life of that session. needs_snapshot (set in the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares the oplog horizon (MIN(seq), one b-tree seek) against its sent watermark and throws LocalDbOplogGapException when deltas it still owes are gone. The horizon, not the flag, because the flag is also raised for reasons that do not invalidate the running session.
  • The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins. Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash or partition of the dialling node left the passive node's handler parked in MoveNext on a half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted peer's reconnects with AlreadyExists, indefinitely on an idle pair. The initiator now dials with KeepAlivePingDelay / KeepAlivePingTimeout (new options, 60s/20s, Always policy so idle connections ping too), and a fresh authenticated Sync cancels the incumbent and waits for it to unwind before starting — the guard serializes sessions, it was never a fence, and an authenticated same-key dialler is by definition the peer. An incumbent that will not yield within the takeover timeout still gets AlreadyExists, because two live sessions over one database is worse than a refused dial.
• 0.3.1 (2026-08-18) — a faulting session flushes its control lane before cancelling its writer, so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to reconnect-loop against a generic disconnect.
• 0.3.0 (2026-08-15) — the snapshot-resync flag is per-table. On-disk bookkeeping schema goes to v3 (new __localdb_snapshot_state, upgraded in place on open); wire-compatible with 0.1.x/0.2.x peers, and lib_schema_version deliberately stays 1 because it is compared fail-closed for equality.
  • RegisterReplicated(table, baselineExistingRows: true) now flags a resync for THAT TABLE. Through 0.2.x the flag was one bit for the whole database, so baselining one table re-streamed every registered table in full — and on a pair enabling replication late, both nodes did that to each other at once. A snapshot now streams only the flagged tables.
  • The database-wide flag survives, unchanged, for the cases that genuinely need it: an oplog cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of every table is sound. What moved is only where a baseline records its debt.
  • Additive wire extension, negotiated by capability, not by version. Handshake gains supports_partial_snapshot; SnapshotBegin gains a tables scope where empty means "all tables" — which is precisely what a pre-0.3.0 sender emits and what a pre-0.3.0 receiver assumes. A peer that does not advertise the capability is served a full snapshot, because a scoped one would make it advance last_applied_remote_seq past deltas it never received.
  • A partial snapshot moves no watermark on either side. The receiver keeps last_applied_remote_seq (it is one number for the whole peer oplog, and the untouched tables' deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The overlap re-delivers rows the snapshot already carried, which LWW absorbs.
  • Snapshot debt now clears on acknowledgement, not on enqueue (new SnapshotAck, exchanged only between 0.3.0 peers). A crash part-way through a snapshot re-streams the same scope on reconnect instead of losing it. Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is no ack to wait for.
• 0.2.1 (2026-08-14) — two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers; no proto change, no lib_schema_version bump, no on-disk schema change.
  • DeregisterReplicated no longer leaves the HLC without a crash anchor. Clock recovery on open is MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc), and hlc_high_water was written only by graceful disposal — while deregistration deletes the table's oplog AND ledger rows. A crash after a deregister boot could therefore recover the clock below stamps the peer already holds, and LWW discards those as older. The current clock is now flushed inside the deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never regress the anchor).
  • The sync inbox is bounded (64 messages) instead of unbounded, so inbound memory is capped at roughly 64 x MaxBatchBytes rather than "whatever the peer sends" — the bad case being a mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now stops draining the gRPC stream, which closes the peer's flow-control window.
  • A snapshot this node owes is streamed alongside a running receive loop, not from inside the handshake. That is what makes bounding safe: previously neither side drained its inbox while sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was already safe (snapshot rows and deltas both merge through LWW, the watermark write is MAX-guarded) — only the ordering changed.
  • Documented the cost of late baselining: the snapshot flag is per-database, so baselining one table re-streams every registered table, in both directions on a late-enabled pair. Trim high-volume tables (event logs) before turning replication on late.
• 0.2.0 (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review. All of it is wire-compatible with 0.1.x peers (no proto change, no lib_schema_version bump); the on-disk bookkeeping schema goes to v2 and is upgraded in place on open.
  • DeregisterReplicated(table) — drops a table's capture triggers and prunes its oplog and row-version rows. A node that does not replicate should not be paying for capture on every write, and until now there was no way to stop once a file had been registered. Idempotent, and usable on triggers left behind by an earlier build.
  • RegisterReplicated(table, baselineExistingRows: true) — closes the late-opt-in hole: rows that predate registration were invisible to the peer forever, because the snapshot streamer pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at an LWW-safe floor (HLC 0, this node's id) and flags a snapshot resync so they actually travel. Default is still off — the documented CDC contract is unchanged unless you ask.
  • O(1) backlog depth — MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1) replaces a COUNT(*) over the whole unacked backlog that ran once a minute AND on every metrics scrape (measured 58 VM steps vs 100,020 on a 50k backlog).
  • Byte-budget batching — MaxBatchBytes (default 2 MB) bounds delta and snapshot messages by summed serialized size, with MaxBatchSize demoted to a secondary row cap. Consumers no longer have to tune the row count against their widest row to stay under the gRPC message limit. Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
  • LwwApplier prepares once per statement instead of three SQL parses per applied row.
  • Dropped __localdb_oplog_hlc — an index no query ever used, costing a b-tree insert inside every captured consumer write. Existing files lose it on the next open.
• 0.1.3 (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN writes were silently dropped. last_applied_remote_seq is a watermark in the peer's seq space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's watermark, the rebuilt node started its pump above it, and its first N writes were never sent. Now the watermark is reset when the peer's node id changes, and a peer claiming to have applied more of our stream than we ever produced is treated as evidence that we were rebuilt (pump restarts from the beginning; LWW makes re-sending harmless). Found on the OtOpcUa docker-dev rig — it only became reachable once 0.1.2 made back-fill work.
• 0.1.2 (2026-07-21) — fixes snapshot resync for a wiped peer: a converged pair prunes its oplog to empty on ack, and the handshake read an empty oplog as "no gap possible", so a peer that came back with an empty database and a zero watermark was never snapshotted. The gap is now measured against last_acked_seq when the oplog is empty.
• 0.1.1 (2026-07-20) — creates the parent directory of LocalDb:Path (a missing directory was a hard boot failure, SQLite Error 14).

Quickstart

Local-only (no replication)

builder.Services.AddZbLocalDb(builder.Configuration, db =>
{
    // onReady: create your own tables, then opt them into (future) replication.
    db.ExecuteAsync(
        "CREATE TABLE IF NOT EXISTS widget (id INTEGER PRIMARY KEY, name TEXT, qty INTEGER)")
      .GetAwaiter().GetResult();
    db.RegisterReplicated("widget");   // PK required; installs capture triggers
    // The application assigns widget ids. A key SQLite assigns per node (AUTOINCREMENT, or a NULL
    // rowid) collides across a pair: AUTOINCREMENT is refused, prefer TEXT GUIDs. See Limitations.
});

// Consume ILocalDb anywhere:
await db.ExecuteAsync("INSERT INTO widget (id, name, qty) VALUES (@id, @n, @q)",
    new { id = 1, n = "bolt", q = 42 });
var rows = await db.QueryAsync("SELECT name, qty FROM widget",
    r => (Name: r.GetString(0), Qty: r.GetInt32(1)));

AddZbLocalDb is first-registration-wins: a second call is a no-op. RegisterReplicated installs capture triggers even without the replication package present — you can register tables now and turn on sync later.

Replicated (2 nodes)

Every node registers the passive side (so either can accept a stream) and the initiator (idle unless a PeerAddress is set). Configure exactly one side with the peer's address; the other stays passive.

// Both nodes:
builder.Services.AddZbLocalDb(builder.Configuration, db => { /* create + RegisterReplicated */ });
builder.Services.AddGrpc();
builder.Services.AddZbLocalDbReplication(builder.Configuration);   // passive host + initiator + maintenance

var app = builder.Build();
app.MapZbLocalDbSync();   // passive gRPC endpoint — host must have AddGrpc()
app.Run();

// Node A — the INITIATOR: dials node B
"LocalDb": {
  "Path": "nodeA.db",
  "Replication": { "PeerAddress": "https://nodeB:5001", "ApiKey": "…" }
}
// Node B — PASSIVE: no PeerAddress, its SyncBackgroundService idles and only serves the endpoint
"LocalDb": { "Path": "nodeB.db", "Replication": { } }

Both directions flow on the single bidirectional stream the initiator opens; the passive node still pushes its own deltas back. Only one side needs PeerAddress.

Configuration

LocalDb section (LocalDbOptions)

Key	Default	Meaning
Path	"" (required)	Filesystem path to the SQLite database file.
BusyTimeoutMs	5000	SQLite busy_timeout pragma, in milliseconds.
Synchronous	"NORMAL"	SQLite synchronous pragma (OFF / NORMAL / FULL / EXTRA, case-insensitive).

LocalDb:Replication section (ReplicationOptions)

Key	Default	Meaning
PeerAddress	""	gRPC address to dial. Empty ⇒ passive (initiator idles).
ApiKey	null	Shared pair key — set the same value on both nodes. The initiator sends it as Authorization: Bearer <key>; since 0.3.6 the passive endpoint requires it and refuses any other stream with Unauthenticated. Blank ⇒ no header sent and no check (any caller is accepted; a startup warning is logged).
FlushInterval	250 ms	How often the change pump flushes pending oplog deltas.
MaxBatchBytes	2 MB	Primary batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone.
MaxBatchSize	500	Secondary row-count cap per outbound batch (also the snapshot DB page size).
MaxOplogRows	1_000_000	Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows.
MaxOplogAge	7 days	Age ceiling; an unpruned entry older than this flags a snapshot resync.
TombstoneRetention	7 days	How long a deleted-row tombstone is kept after the peer has received it (0.3.7: pruned only once past this AND delivered — delta acked or snapshot discharged; the clock restarts at delivery). An undelivered tombstone is kept for the whole outage.
TombstoneHardCap	null (off)	Optional absolute ceiling on keeping an undelivered tombstone (0.3.7). Bounds ledger growth when the peer is gone for good; a peer that returns after more than this can resurrect deleted rows, so rebuild it instead. Must be ≥ TombstoneRetention.
ReconnectBackoffMax	60 s	Upper bound on exponential reconnect backoff (validator caps ≤ 1 day).
MaintenanceInterval	60 s	How often maintenance enforces caps + prunes expired tombstones.
MaxSnapshotDeferredRows	100_000	Ceiling on snapshot rows held in memory while their FK parent has not arrived (0.3.6). Past it the oldest are dead-lettered (and counted).
MaxHlcDriftAhead	5 min	Max tolerated HLC drift where the peer's clock runs ahead of ours.
FailClosedOnDrift	false	When true, an over-drift entry is dead-lettered instead of applied.

All options are validated at startup (ZB.MOM.WW.Configuration validators).

How it works

• Capture — three AFTER triggers per registered table write each change into one shared, HLC-stamped __localdb_oplog plus a __localdb_row_version ledger, in the consumer's own transaction. A DB-scoped __localdb_applying guard makes replicated applies skip re-capture.
• Convergence — last-writer-wins: compare incoming HLC, then node_id ordinal as the tie-break; the strict-greater winner is applied, the loser discarded (but still acked).
• Streaming — one long-lived bidirectional gRPC stream. The initiator dials; both sides push oplog batches above the peer's watermark and ack the applied watermark back, which drives pruning. Outbound uses a single writer fed by two lanes (an unbounded control lane for handshake/acks that always preempts a bounded data lane for delta/snapshot batches), so receiving never blocks on the ability to ack.
• Resync — when a peer's watermark falls below the pruned oplog horizon, it is sent a full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above the snapshot's as-of seq. When instead a single table was baselined, only that table is streamed and no watermark moves on either side — a partial snapshot speaks for nothing outside its own tables, so the deltas it overlaps are still pumped and merged by LWW. Each node decides from its OWN state whether it owes the peer a snapshot, and a peer that does not advertise supports_partial_snapshot is sent a full one. The snapshot is streamed after the handshake, alongside a running receive loop, so a node sending one still applies what the peer sends it — a mutual snapshot exchanges under flow control instead of being buffered whole in memory. Between 0.3.0 peers the sender's flags are cleared by the receiver's SnapshotAck, so a crash mid-snapshot re-streams it.
• Maintenance — a background service on both roles enforces the oplog row/age caps and prunes expired tombstones every MaintenanceInterval — only those the peer is known to hold (0.3.7; see Ops notes).

Semantics worth knowing

• PK-column updates are captured as delete-of-old-PK + insert-of-new — a tombstone on the old key plus a fresh insert (full delete+insert support).
• Upserts (INSERT … ON CONFLICT DO UPDATE) are fully supported (the row-version trigger uses an explicit ON CONFLICT clause so it is not rewritten by the outer statement's conflict algorithm).
• Clock-drift guard — warn-once per session when a peer entry's HLC physical time is more than MaxHlcDriftAhead ahead. With FailClosedOnDrift, each over-drift entry is dead-lettered; otherwise it is applied (the HLC merge absorbs the skew). The session never dies from drift — no livelock on reconnect. Since 0.3.6 the guard covers snapshot rows too (warn once per snapshot; dead-lettered under FailClosedOnDrift, and their HLC is neither observed nor persisted as last_seen_hlc). Up to 0.3.5 a snapshot applied every row unchecked, so one future-stamped row could push the receiver's clock ahead permanently.
• Writes never block on the peer; local commit is independent of replication.
• All timestamps are UTC (HLC physical component is UTC Unix milliseconds).

Limitations (honest list)

• Exactly two nodes. No mesh / N-node topologies.
• LWW only. No per-table pluggable conflict resolvers.
• Replicated primary keys must be unique across both nodes. LWW keys on the primary key, so two different rows inserted on the two nodes under the same key are read as two versions of one row, and one is lost on both nodes with no dead-letter and no metric. A key each node assigns on its own cannot replicate: RegisterReplicated refuses an AUTOINCREMENT key (0.3.6), and a plain INTEGER PRIMARY KEY is accepted only because the application may assign it from a node-scoped range — left NULL on insert it loses rows the same way. Use a TEXT GUID minted by the writer (what ScadaBridge's site_events and OtOpcUa's alarm store-and-forward did when they moved onto LocalDb).
• BLOB columns are rejected at registration (a column declared BLOB fails RegisterReplicated, because json_object cannot capture it). A typeless column can still hold a BLOB value at runtime, which then surfaces as a write-time error, not a registration error.
• A replicated table may not be the parent of a row-mutating ON DELETE foreign key (CASCADE / SET NULL / SET DEFAULT from any other table, or from itself). Applying a remote tombstone deletes the row (and up to 0.3.5 applying any remote row used INSERT OR REPLACE, whose implicit delete did the same), which fires that action with change capture suppressed — the child rows would change on the applying node only and never replicate, diverging the pair undetectably. RegisterReplicated refuses such a table (0.3.3); use ON DELETE NO ACTION/RESTRICT and delete children explicitly, or leave the parent unreplicated. The check runs at registration and reads the whole database's foreign keys, so a child table created after registration is caught at the next startup, not at the moment of the CREATE TABLE.
• Foreign keys between replicated tables (NO ACTION/RESTRICT) survive a snapshot as of 0.3.6. The sender streams referenced tables before the tables that reference them, and the receiver defers a snapshot row that fails only its FK check, retrying it as more rows land; one whose parent never arrives is dead-lettered at SnapshotComplete. Before 0.3.6 a child that arrived before its parent (name order, or a self-reference whose parent's key sorts later as pk_json text) was dead-lettered for good. Mixed pairs: a pre-0.3.6 receiver still loses self-referencing rows that arrive child-first, and a pre-0.3.6 sender's name order relies on the 0.3.6 receiver's deferral — so upgrade both nodes before adding FK-linked tables. Deferred rows are held in memory until the snapshot completes, up to MaxSnapshotDeferredRows (default 100,000); past that the oldest are dead-lettered rather than buffered without bound — an older, name-ordered sender whose child table sorts first can otherwise defer a whole table.
• Secondary UNIQUE keys must be unique across both nodes. Remote rows are applied with INSERT … ON CONFLICT(<pk>) DO UPDATE (0.3.6), so only the row with the same primary key is touched. If the same natural key is created under different primary keys on the two nodes (e.g. during a partition), each node keeps its own row and dead-letters the peer's — visible in __localdb_dead_letter, but the pair holds different rows until the conflict is resolved by hand. Up to 0.3.5 INSERT OR REPLACE silently deleted the local row instead and the pair diverged undetectably. Prefer GUID or node-scoped natural keys.
• CDC semantics — rows that existed before RegisterReplicated are not captured and are not snapshotted; only changes after registration are replicated. Opt out with RegisterReplicated(table, baselineExistingRows: true), which seeds the row-version ledger for existing rows at the baseline floor (HLC 0) and flags a snapshot resync for that table so they reach the peer. The floor is below every genuine stamp, so a baselined row loses every comparison against a real remote write and wins only where the peer has no version of that key at all. Two nodes that baseline the same key both hold HLC 0 and resolve by the node-id tie-break — convergent, but arbitrary as to which content survives.
• A snapshot still ships whole tables. The flag is per-table as of 0.3.0, so baselining one table streams that table and no other — but that table travels in full, and two cases still stream everything: an oplog cap prune / impossible peer ack (delta continuity is lost for the oplog as a whole), and any peer older than 0.3.0 (which cannot be told a snapshot is partial). So purge or trim high-volume tables such as event logs before baselining them, and expect one slow first sync. Bounded either way (batch budget out, bounded inbox in).
• REAL ±Infinity is carried in SQLite's JSON dialect on the wire (fine for the SQLite↔SQLite peers this library targets).
• AddZbLocalDb / AddZbLocalDbReplication are first-registration-wins (idempotent).
• Inbound buffering is bounded, not zero. The duplex inbox holds up to 64 messages, so peak inbound memory is roughly 64 x MaxBatchBytes (~128 MB at the 2 MB default, far less in practice — messages are only that large when the sender has that much backlog). Lower MaxBatchBytes to lower the ceiling. Beyond the bound the stream stops being drained and the peer is slowed by flow control.

Observability

Metrics publish under the single meter ZB.MOM.WW.LocalDb.Replication (all instruments null-guarded — replication runs identically whether or not metrics are collected):

Instrument	Kind	Notes
localdb.oplog.depth	ObservableGauge	Unacked oplog backlog (polled cheaply at scrape time).
localdb.sync.lag.seconds	ObservableGauge	Seconds since the last successful sync (no measurement when never synced).
localdb.sync.applied	Counter	Rows applied from inbound delta batches.
localdb.sync.conflicts_discarded	Counter	Inbound rows that lost LWW.
localdb.sync.dead_lettered	Counter	Rows routed to the dead-letter table — poison / over-drift rows from deltas and (since 0.3.6) snapshots, including FK orphans and rows spilled past MaxSnapshotDeferredRows.
localdb.sync.reconnects	Counter	Reconnect attempts (initiator).
localdb.sync.snapshots	Counter	Snapshots, tagged direction=sent/received (one increment per snapshot).

Counter split to note: rows merged from a snapshot are not counted under localdb.sync.applied; a completed snapshot instead increments localdb.sync.snapshots once (with its direction tag). Only delta-batch rows increment localdb.sync.applied.

ISyncStatus (registered singleton) surfaces live state for dashboards / health: Connected (true while any session is active — a node can run both roles at once), PeerNodeId, LastSyncUtc, OplogBacklog (nullable — null means unknown because no provider is wired or the poll threw, so a DB failure never reads as "0 backlog"), and ConnectionAttempts.

Ops notes

• Tombstones and long outages (0.3.7, scadaproj#16). A tombstone is what lets LWW reject an older write of a deleted row, so it is kept until the peer has it: while the database or its table owes the peer a snapshot (the snapshot is built from the ledger, so a pruned tombstone is a delete it can no longer carry), and while a local delete is still in the unacked oplog. Delivery (a delta ack covering the delete, or the snapshot debt being discharged) restamps tombstone_utc, and TombstoneRetention runs from there — the returning peer's own stale writes get a full retention to drain. The cost: while the peer is down, every delete's tombstone stays in __localdb_row_version (high-churn tables such as a store-and-forward queue grow the ledger for the whole outage). Set TombstoneHardCap to bound that; past it the pre-0.3.7 resurrection risk returns, so treat a peer offline longer than the cap as one to rebuild, not reconnect. Not covered by any setting: a peer restored from an old backup of its own file presents stale rows under its old identity and watermarks — rebuild it empty instead.

• When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the database-wide needs_snapshot (disk safety over delta continuity); the peer then gets a full snapshot. Per-table debt lives separately in __localdb_snapshot_state and is what baselining writes.

• Upgrading a pair to 0.3.0 needs no coordination: the two nodes negotiate scoped snapshots by capability, so a 0.3.0 node paired with a 0.2.x one simply keeps sending full snapshots until both are upgraded. Rolling the file forward is one-way — a 0.2.x build reopening a v3 file leaves __localdb_snapshot_state in place and ignores it, so any per-table debt recorded by 0.3.0 is invisible to it (roll forward again, or re-baseline).

• Turning a table off — DeregisterReplicated(table) drops its capture triggers and prunes its oplog + ledger rows. It must be applied symmetrically: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on neither. Re-enabling later should pass baselineExistingRows: true, or the rows that accumulated while capture was off stay invisible to the peer — but trim high-volume that table first (see Limitations: baselining streams that one table in full, not every table — only an oplog cap prune or a pre-0.3.0 peer streams everything). The HLC's durable anchor is flushed in the same transaction, so a crash after a deregister boot cannot bring the clock back below stamps peers already hold.

• On-disk schema upgrades are applied in place when the file is opened, in one transaction, and every step is safe to re-run. schema_version in __localdb_meta is local-file state only — it is NOT the wire lib_schema_version, so a pair mid-upgrade keeps syncing.

• The __localdb_dead_letter table holds poison rows (bad JSON, null-row non-tombstone entries), over-drift rejects (when FailClosedOnDrift), snapshot rows whose FK parent never arrived, and secondary-UNIQUE clashes. Every row written there, on the delta and the snapshot path, is counted in localdb.sync.dead_lettered (snapshot-path rows were not counted before 0.3.6). Inspect it to diagnose rejects.

• Peer auth (0.3.6+): with ApiKey set, the initiator sends Authorization: Bearer <key> and the passive LocalDbSyncService verifies it (constant-time) before the handshake and before it displaces the current stream — a caller without the key gets Unauthenticated, sees no table names or schema digests, and cannot evict the real peer. With ApiKey blank on a node, that node's endpoint accepts any caller (exactly the pre-0.3.6 behaviour) and logs a warning at startup; protect it by network isolation or a host interceptor. Up to 0.3.5 the library never checked the key, so ScadaBridge and OtOpcUa each added a fail-closed LocalDbSyncAuthInterceptor over the same key — those keep working unchanged and are now redundant with the library check (they still add fail-closed-when-blank, which the library deliberately does not, so a key-less pair keeps syncing).

  • Rollout: the wire is unchanged (the initiator has always sent the header when the key is set). Set the key on both nodes; a node with the key refuses a peer without it, so a pair with mismatched keys stops syncing until both agree. A pair where neither node has a key is unaffected.

Testing

259 tests, all offline / no live dependencies — unit (HLC ordering + restart persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL generation, LWW decision table, watermark/pruning, digest computation, schema-version upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence, delete/update races, offline-accumulate → reconnect, prune → snapshot resync, a peer offline past TombstoneRetention not resurrecting a newer delete, mutual snapshot over a capacity-1 transport, single-table baseline streaming only that table, mixed-version interop where a peer without supports_partial_snapshot gets a full one, snapshot debt surviving until the peer's SnapshotAck), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce). Family conventions apply (TreatWarningsAsErrors).

Build / test from this directory:

dotnet build ZB.MOM.WW.LocalDb.slnx
dotnet test  ZB.MOM.WW.LocalDb.slnx

See the design doc: docs/plans/2026-07-17-localdb-design.md.