# ZB.MOM.WW.LocalDb An embedded **SQLite local cache** with **optional bidirectional async 2-node replication over gRPC**, for the ZB.MOM.WW SCADA / OT family. It gives an application a fast, crash-safe local store that can optionally keep a second node in sync asynchronously — local writes **never** block on the peer, and a node that never enables replication is simply a fast local DB. Change capture uses SQLite triggers writing a single HLC-stamped oplog inside the consumer's own write transaction (crash-consistent, non-bypassable). Convergence is last-writer-wins over a hybrid logical clock. All timestamps are UTC. ## Packages (all `0.3.3`) | Package | Contents | |---|---| | `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK + BLOB + foreign-key validation, trigger install, oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. | | `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). | | `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. | > Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).** > > - `0.3.3` (2026-08-20) — two latent apply-path hazards from the ScadaBridge adversarial > architecture review (P4-7). Neither had a live trigger in any current consumer; both are the > kind that surface as silent divergence, not as an error. Wire-compatible: no proto change, no > `lib_schema_version` bump, no on-disk schema change. > - **A replicated table may no longer be the parent of a row-mutating `ON DELETE` foreign key.** > The apply path writes remote rows with `INSERT OR REPLACE`, and REPLACE resolves a PK conflict > by DELETING the existing row first. Connections run `PRAGMA foreign_keys=ON`, so that implicit > delete fires the child's `ON DELETE CASCADE` / `SET NULL` / `SET DEFAULT` — while the capture > guard is raised. The child rows change on the applying node ONLY, produce no oplog entry, and > the pair diverges with nothing able to detect it (LWW compares only rows it was told about). > `RegisterReplicated` now inspects every foreign key in the database and refuses such a table > with a `LocalDbRegistrationException` naming the offending child column and action. > `RESTRICT` / `NO ACTION` remain fine — they abort the statement instead of mutating rows > behind capture's back, which the apply path already handles as per-entry poison. Checked at > registration; SQLite cannot add a foreign key to an existing table, so a child table created > later is caught at the next startup's registration rather than at the DDL. > - **An entry's row write and its ledger write are now atomic with each other.** They are two > statements, and SQLite aborts only the failing one — so a statement-level fault between them > (the case the per-entry poison catch exists for) committed a half-applied entry: a tombstone > whose table row was gone while `__localdb_row_version` still called it live, which the snapshot > streamer can only warn about and skip, so the peer never learned the row died; or an upsert > whose new row carried the OLD version and therefore lost the next comparison it should have > won. The pair is wrapped in a per-entry `SAVEPOINT` that is rolled back before dead-lettering, > so a poison entry leaves both writes or neither and the rest of the batch is untouched. > - `0.3.2` (2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review > (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, no `lib_schema_version` > bump, no on-disk schema change — the keepalive is transport-level and the rest is local policy. > - **A live session now ends itself when a cap prune eats deltas it has not sent.** The pump reads > `seq > sentThruSeq`, which resumes SILENTLY above a pruned hole; the peer acks the later seqs > and the pair is durably divergent for the whole life of that session. `needs_snapshot` (set in > the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable > LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares > the oplog horizon (`MIN(seq)`, one b-tree seek) against its sent watermark and throws > `LocalDbOplogGapException` when deltas it still owes are gone. The horizon, not the flag, > because the flag is also raised for reasons that do not invalidate the running session. > - **The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins.** > Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash > or partition of the dialling node left the passive node's handler parked in `MoveNext` on a > half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted > peer's reconnects with `AlreadyExists`, indefinitely on an idle pair. The initiator now dials > with `KeepAlivePingDelay` / `KeepAlivePingTimeout` (new options, 60s/20s, `Always` policy so > idle connections ping too), and a fresh authenticated `Sync` cancels the incumbent and waits > for it to unwind before starting — the guard serializes sessions, it was never a fence, and an > authenticated same-key dialler is by definition the peer. An incumbent that will not yield > within the takeover timeout still gets `AlreadyExists`, because two live sessions over one > database is worse than a refused dial. > - `0.3.1` (2026-08-18) — a faulting session flushes its control lane before cancelling its writer, > so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to > reconnect-loop against a generic disconnect. > - `0.3.0` (2026-08-15) — **the snapshot-resync flag is per-table.** On-disk bookkeeping schema > goes to **v3** (new `__localdb_snapshot_state`, upgraded in place on open); wire-compatible with > `0.1.x`/`0.2.x` peers, and `lib_schema_version` deliberately stays `1` because it is compared > fail-closed for equality. > - **`RegisterReplicated(table, baselineExistingRows: true)` now flags a resync for THAT TABLE.** > Through `0.2.x` the flag was one bit for the whole database, so baselining one table > re-streamed every registered table in full — and on a pair enabling replication late, both > nodes did that to each other at once. A snapshot now streams only the flagged tables. > - **The database-wide flag survives, unchanged, for the cases that genuinely need it**: an oplog > cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of > every table is sound. What moved is only where a *baseline* records its debt. > - **Additive wire extension, negotiated by capability, not by version.** `Handshake` gains > `supports_partial_snapshot`; `SnapshotBegin` gains a `tables` scope where **empty means "all > tables"** — which is precisely what a pre-`0.3.0` sender emits and what a pre-`0.3.0` receiver > assumes. A peer that does not advertise the capability is served a **full** snapshot, because a > scoped one would make it advance `last_applied_remote_seq` past deltas it never received. > - **A partial snapshot moves no watermark on either side.** The receiver keeps > `last_applied_remote_seq` (it is one number for the whole peer oplog, and the untouched tables' > deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The > overlap re-delivers rows the snapshot already carried, which LWW absorbs. > - **Snapshot debt now clears on acknowledgement, not on enqueue** (new `SnapshotAck`, exchanged > only between `0.3.0` peers). A crash part-way through a snapshot re-streams the same scope on > reconnect instead of losing it. Against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue > behaviour is kept — there is no ack to wait for. > - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with > `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change. > - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on > open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was > written only by graceful disposal — while deregistration deletes the table's oplog AND ledger > rows. A crash after a deregister boot could therefore recover the clock below stamps the peer > already holds, and LWW discards those as older. The current clock is now flushed inside the > deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never > regress the anchor). > - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped > at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a > mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now > stops draining the gRPC stream, which closes the peer's flow-control window. > - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the > handshake. That is what makes bounding safe: previously neither side drained its inbox while > sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was > already safe (snapshot rows and deltas both merge through LWW, the watermark write is > MAX-guarded) — only the ordering changed. > - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining > one table re-streams every registered table, in both directions on a late-enabled pair. Trim > high-volume tables (event logs) before turning replication on late. > - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review. > All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump); > the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open. > - **`DeregisterReplicated(table)`** — drops a table's capture triggers and prunes its oplog and > row-version rows. A node that does not replicate should not be paying for capture on every > write, and until now there was no way to stop once a file had been registered. Idempotent, and > usable on triggers left behind by an earlier build. > - **`RegisterReplicated(table, baselineExistingRows: true)`** — closes the late-opt-in hole: rows > that predate registration were invisible to the peer forever, because the snapshot streamer > pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at > an LWW-safe floor (HLC `0`, this node's id) and flags a snapshot resync so they actually > travel. Default is still off — the documented CDC contract is unchanged unless you ask. > - **O(1) backlog depth** — `MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)` replaces a > `COUNT(*)` over the whole unacked backlog that ran once a minute AND on every metrics scrape > (measured 58 VM steps vs 100,020 on a 50k backlog). > - **Byte-budget batching** — `MaxBatchBytes` (default 2 MB) bounds delta and snapshot messages by > summed serialized size, with `MaxBatchSize` demoted to a secondary row cap. Consumers no longer > have to tune the row count against their widest row to stay under the gRPC message limit. > Inbound delta batches and acks already queued are coalesced into one transaction / one ack. > - **`LwwApplier` prepares once per statement** instead of three SQL parses per applied row. > - **Dropped `__localdb_oplog_hlc`** — an index no query ever used, costing a b-tree insert inside > every captured consumer write. Existing files lose it on the next open. > - `0.1.3` (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN > writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq > space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's > watermark, the rebuilt node started its pump above it, and its first N writes were never sent. > Now the watermark is reset when the peer's node id changes, and a peer claiming to have applied > more of our stream than we ever produced is treated as evidence that *we* were rebuilt (pump > restarts from the beginning; LWW makes re-sending harmless). Found on the OtOpcUa docker-dev rig > — it only became reachable once `0.1.2` made back-fill work. > - `0.1.2` (2026-07-21) — fixes snapshot resync for a **wiped peer**: a converged pair prunes its > oplog to empty on ack, and the handshake read an empty oplog as "no gap possible", so a peer > that came back with an empty database and a zero watermark was never snapshotted. The gap is > now measured against `last_acked_seq` when the oplog is empty. > - `0.1.1` (2026-07-20) — creates the parent directory of `LocalDb:Path` (a missing directory was > a hard boot failure, `SQLite Error 14`). ## Quickstart ### Local-only (no replication) ```csharp builder.Services.AddZbLocalDb(builder.Configuration, db => { // onReady: create your own tables, then opt them into (future) replication. db.ExecuteAsync( "CREATE TABLE IF NOT EXISTS widget (id INTEGER PRIMARY KEY, name TEXT, qty INTEGER)") .GetAwaiter().GetResult(); db.RegisterReplicated("widget"); // PK required; installs capture triggers }); // Consume ILocalDb anywhere: await db.ExecuteAsync("INSERT INTO widget (id, name, qty) VALUES (@id, @n, @q)", new { id = 1, n = "bolt", q = 42 }); var rows = await db.QueryAsync("SELECT name, qty FROM widget", r => (Name: r.GetString(0), Qty: r.GetInt32(1))); ``` `AddZbLocalDb` is first-registration-wins: a second call is a no-op. `RegisterReplicated` installs capture triggers even without the replication package present — you can register tables now and turn on sync later. ### Replicated (2 nodes) Every node registers the passive side (so either can accept a stream) and the initiator (idle unless a `PeerAddress` is set). Configure exactly **one** side with the peer's address; the other stays passive. ```csharp // Both nodes: builder.Services.AddZbLocalDb(builder.Configuration, db => { /* create + RegisterReplicated */ }); builder.Services.AddGrpc(); builder.Services.AddZbLocalDbReplication(builder.Configuration); // passive host + initiator + maintenance var app = builder.Build(); app.MapZbLocalDbSync(); // passive gRPC endpoint — host must have AddGrpc() app.Run(); ``` ```jsonc // Node A — the INITIATOR: dials node B "LocalDb": { "Path": "nodeA.db", "Replication": { "PeerAddress": "https://nodeB:5001", "ApiKey": "…" } } // Node B — PASSIVE: no PeerAddress, its SyncBackgroundService idles and only serves the endpoint "LocalDb": { "Path": "nodeB.db", "Replication": { } } ``` Both directions flow on the single bidirectional stream the initiator opens; the passive node still pushes its own deltas back. Only one side needs `PeerAddress`. ## Configuration ### `LocalDb` section (`LocalDbOptions`) | Key | Default | Meaning | |---|---|---| | `Path` | `""` (required) | Filesystem path to the SQLite database file. | | `BusyTimeoutMs` | `5000` | SQLite `busy_timeout` pragma, in milliseconds. | | `Synchronous` | `"NORMAL"` | SQLite `synchronous` pragma (`OFF` / `NORMAL` / `FULL`). | ### `LocalDb:Replication` section (`ReplicationOptions`) | Key | Default | Meaning | |---|---|---| | `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). | | `ApiKey` | `null` | Bearer token the initiator presents on its channel; `null` ⇒ no key header. | | `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. | | `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. | | `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). | | `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. | | `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. | | `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. | | `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). | | `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. | | `MaxHlcDriftAhead` | `5 min` | Max tolerated HLC drift where the peer's clock runs ahead of ours. | | `FailClosedOnDrift` | `false` | When true, an over-drift entry is dead-lettered instead of applied. | All options are validated at startup (`ZB.MOM.WW.Configuration` validators). ## How it works - **Capture** — three AFTER triggers per registered table write each change into one shared, HLC-stamped `__localdb_oplog` plus a `__localdb_row_version` ledger, in the consumer's own transaction. A DB-scoped `__localdb_applying` guard makes replicated applies skip re-capture. - **Convergence** — last-writer-wins: compare incoming HLC, then `node_id` ordinal as the tie-break; the strict-greater winner is applied, the loser discarded (but still acked). - **Streaming** — one long-lived bidirectional gRPC stream. The initiator dials; both sides push oplog batches above the peer's watermark and ack the applied watermark back, which drives pruning. Outbound uses a single writer fed by two lanes (an unbounded control lane for handshake/acks that always preempts a bounded data lane for delta/snapshot batches), so receiving never blocks on the ability to ack. - **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above the snapshot's as-of seq. When instead a single table was *baselined*, only that table is streamed and **no watermark moves on either side** — a partial snapshot speaks for nothing outside its own tables, so the deltas it overlaps are still pumped and merged by LWW. **Each node decides from its OWN state** whether it owes the peer a snapshot, and a peer that does not advertise `supports_partial_snapshot` is sent a full one. The snapshot is streamed *after* the handshake, alongside a running receive loop, so a node sending one still applies what the peer sends it — a mutual snapshot exchanges under flow control instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it. - **Maintenance** — a background service on **both** roles enforces the oplog row/age caps and prunes expired tombstones every `MaintenanceInterval`. ## Semantics worth knowing - **PK-column updates** are captured as delete-of-old-PK + insert-of-new — a tombstone on the old key plus a fresh insert (full delete+insert support). - **Upserts** (`INSERT … ON CONFLICT DO UPDATE`) are fully supported (the row-version trigger uses an explicit `ON CONFLICT` clause so it is not rewritten by the outer statement's conflict algorithm). - **Clock-drift guard** — warn-once per session when a peer entry's HLC physical time is more than `MaxHlcDriftAhead` ahead. With `FailClosedOnDrift`, each over-drift entry is dead-lettered; otherwise it is applied (the HLC merge absorbs the skew). The **session never dies from drift** — no livelock on reconnect. - **Writes never block on the peer**; local commit is independent of replication. - **All timestamps are UTC** (HLC physical component is UTC Unix milliseconds). ## Limitations (honest list) - **Exactly two nodes.** No mesh / N-node topologies. - **LWW only.** No per-table pluggable conflict resolvers. - **BLOB columns are rejected at registration** (a column declared `BLOB` fails `RegisterReplicated`, because `json_object` cannot capture it). A *typeless* column can still hold a BLOB value at runtime, which then surfaces as a write-time error, not a registration error. - **A replicated table may not be the parent of a row-mutating `ON DELETE` foreign key** (`CASCADE` / `SET NULL` / `SET DEFAULT` from any other table, or from itself). Applying a remote row uses `INSERT OR REPLACE`, whose implicit delete fires that action with change capture suppressed — the child rows would change on the applying node only and never replicate, diverging the pair undetectably. `RegisterReplicated` refuses such a table (`0.3.3`); use `ON DELETE NO ACTION`/`RESTRICT` and delete children explicitly, or leave the parent unreplicated. The check runs at registration and reads the whole database's foreign keys, so a child table created *after* registration is caught at the next startup, not at the moment of the `CREATE TABLE`. - **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured and are not snapshotted; only changes after registration are replicated. **Opt out with `RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync **for that table** so they reach the peer. The floor is below every genuine stamp, so a baselined row loses every comparison against a real remote write and wins only where the peer has no version of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by the node-id tie-break — convergent, but arbitrary as to which content survives. - **A snapshot still ships whole tables.** The flag is per-table as of `0.3.0`, so baselining one table streams that table and no other — but *that* table travels in full, and two cases still stream **everything**: an oplog cap prune / impossible peer ack (delta continuity is lost for the oplog as a whole), and any peer older than `0.3.0` (which cannot be told a snapshot is partial). So **purge or trim high-volume tables such as event logs before baselining them**, and expect one slow first sync. Bounded either way (batch budget out, bounded inbox in). - **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the SQLite↔SQLite peers this library targets). - `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent). - **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far less in practice — messages are only that large when the sender has that much backlog). Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being drained and the peer is slowed by flow control. ## Observability Metrics publish under the single meter **`ZB.MOM.WW.LocalDb.Replication`** (all instruments null-guarded — replication runs identically whether or not metrics are collected): | Instrument | Kind | Notes | |---|---|---| | `localdb.oplog.depth` | ObservableGauge | Unacked oplog backlog (polled cheaply at scrape time). | | `localdb.sync.lag.seconds` | ObservableGauge | Seconds since the last successful sync (no measurement when never synced). | | `localdb.sync.applied` | Counter | Rows applied from inbound **delta** batches. | | `localdb.sync.conflicts_discarded` | Counter | Inbound rows that lost LWW. | | `localdb.sync.dead_lettered` | Counter | Poison / over-drift rows routed to the dead-letter table. | | `localdb.sync.reconnects` | Counter | Reconnect attempts (initiator). | | `localdb.sync.snapshots` | Counter | Snapshots, tagged `direction=sent`/`received` (one increment per snapshot). | **Counter split to note:** rows merged from a **snapshot** are *not* counted under `localdb.sync.applied`; a completed snapshot instead increments `localdb.sync.snapshots` once (with its `direction` tag). Only delta-batch rows increment `localdb.sync.applied`. `ISyncStatus` (registered singleton) surfaces live state for dashboards / health: `Connected` (true while **any** session is active — a node can run both roles at once), `PeerNodeId`, `LastSyncUtc`, `OplogBacklog` (**nullable** — null means unknown because no provider is wired or the poll threw, so a DB failure never reads as "0 backlog"), and `ConnectionAttempts`. ## Ops notes - When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the **database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what baselining writes. - **Upgrading a pair to `0.3.0`** needs no coordination: the two nodes negotiate scoped snapshots by capability, so a `0.3.0` node paired with a `0.2.x` one simply keeps sending full snapshots until both are upgraded. Rolling the file forward is one-way — a `0.2.x` build reopening a v3 file leaves `__localdb_snapshot_state` in place and ignores it, so any per-table debt recorded by `0.3.0` is invisible to it (roll forward again, or re-baseline). - **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that accumulated while capture was off stay invisible to the peer — but trim high-volume tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's durable anchor is flushed in the same transaction, so a crash after a deregister boot cannot bring the clock back below stamps peers already hold. - **On-disk schema upgrades** are applied in place when the file is opened, in one transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade keeps syncing. - The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose rejects. - **Peer auth**: `ApiKey` is sent by the initiator as a static `Authorization: Bearer …` header on its channel. The library's passive service does **not** itself verify the key — the passive host enforces whatever gRPC auth the host application configures (e.g. `ZB.MOM.WW.Auth`). Configure host-side authorization to actually gate inbound streams. ## Testing 199 tests, all offline / no live dependencies — unit (HLC ordering + restart persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL generation, LWW decision table, watermark/pruning, digest computation, schema-version upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence, delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual snapshot over a capacity-1 transport, single-table baseline streaming only that table, mixed-version interop where a peer without `supports_partial_snapshot` gets a full one, snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce). Family conventions apply (`TreatWarningsAsErrors`). Build / test from this directory: ```bash dotnet build ZB.MOM.WW.LocalDb.slnx dotnet test ZB.MOM.WW.LocalDb.slnx ``` See the design doc: [`docs/plans/2026-07-17-localdb-design.md`](../docs/plans/2026-07-17-localdb-design.md).