scadaproj/ZB.MOM.WW.LocalDb/README.md

# ZB.MOM.WW.LocalDb

An embedded **SQLite local cache** with **optional bidirectional async 2-node
replication over gRPC**, for the ZB.MOM.WW SCADA / OT family. It gives an application a
fast, crash-safe local store that can optionally keep a second node in sync
asynchronously — local writes **never** block on the peer, and a node that never enables
replication is simply a fast local DB.

Change capture uses SQLite triggers writing a single HLC-stamped oplog inside the
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
last-writer-wins over a hybrid logical clock. All timestamps are UTC.

## Packages (all `0.3.3`)

| Package | Contents |
|---|---|
| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK + BLOB + foreign-key validation, trigger install, oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). |
| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. |

> Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
>
> - `0.3.3` (2026-08-20) — two latent apply-path hazards from the ScadaBridge adversarial
>   architecture review (P4-7). Neither had a live trigger in any current consumer; both are the
>   kind that surface as silent divergence, not as an error. Wire-compatible: no proto change, no
>   `lib_schema_version` bump, no on-disk schema change.
>   - **A replicated table may no longer be the parent of a row-mutating `ON DELETE` foreign key.**
>     The apply path writes remote rows with `INSERT OR REPLACE`, and REPLACE resolves a PK conflict
>     by DELETING the existing row first. Connections run `PRAGMA foreign_keys=ON`, so that implicit
>     delete fires the child's `ON DELETE CASCADE` / `SET NULL` / `SET DEFAULT` — while the capture
>     guard is raised. The child rows change on the applying node ONLY, produce no oplog entry, and
>     the pair diverges with nothing able to detect it (LWW compares only rows it was told about).
>     `RegisterReplicated` now inspects every foreign key in the database and refuses such a table
>     with a `LocalDbRegistrationException` naming the offending child column and action.
>     `RESTRICT` / `NO ACTION` remain fine — they abort the statement instead of mutating rows
>     behind capture's back, which the apply path already handles as per-entry poison. Checked at
>     registration; SQLite cannot add a foreign key to an existing table, so a child table created
>     later is caught at the next startup's registration rather than at the DDL.
>   - **An entry's row write and its ledger write are now atomic with each other.** They are two
>     statements, and SQLite aborts only the failing one — so a statement-level fault between them
>     (the case the per-entry poison catch exists for) committed a half-applied entry: a tombstone
>     whose table row was gone while `__localdb_row_version` still called it live, which the snapshot
>     streamer can only warn about and skip, so the peer never learned the row died; or an upsert
>     whose new row carried the OLD version and therefore lost the next comparison it should have
>     won. The pair is wrapped in a per-entry `SAVEPOINT` that is rolled back before dead-lettering,
>     so a poison entry leaves both writes or neither and the rest of the batch is untouched.
> - `0.3.2` (2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review
>   (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, no `lib_schema_version`
>   bump, no on-disk schema change — the keepalive is transport-level and the rest is local policy.
>   - **A live session now ends itself when a cap prune eats deltas it has not sent.** The pump reads
>     `seq > sentThruSeq`, which resumes SILENTLY above a pruned hole; the peer acks the later seqs
>     and the pair is durably divergent for the whole life of that session. `needs_snapshot` (set in
>     the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable
>     LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares
>     the oplog horizon (`MIN(seq)`, one b-tree seek) against its sent watermark and throws
>     `LocalDbOplogGapException` when deltas it still owes are gone. The horizon, not the flag,
>     because the flag is also raised for reasons that do not invalidate the running session.
>   - **The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins.**
>     Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash
>     or partition of the dialling node left the passive node's handler parked in `MoveNext` on a
>     half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted
>     peer's reconnects with `AlreadyExists`, indefinitely on an idle pair. The initiator now dials
>     with `KeepAlivePingDelay` / `KeepAlivePingTimeout` (new options, 60s/20s, `Always` policy so
>     idle connections ping too), and a fresh authenticated `Sync` cancels the incumbent and waits
>     for it to unwind before starting — the guard serializes sessions, it was never a fence, and an
>     authenticated same-key dialler is by definition the peer. An incumbent that will not yield
>     within the takeover timeout still gets `AlreadyExists`, because two live sessions over one
>     database is worse than a refused dial.
> - `0.3.1` (2026-08-18) — a faulting session flushes its control lane before cancelling its writer,
>   so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to
>   reconnect-loop against a generic disconnect.
> - `0.3.0` (2026-08-15) — **the snapshot-resync flag is per-table.** On-disk bookkeeping schema
>   goes to **v3** (new `__localdb_snapshot_state`, upgraded in place on open); wire-compatible with
>   `0.1.x`/`0.2.x` peers, and `lib_schema_version` deliberately stays `1` because it is compared
>   fail-closed for equality.
>   - **`RegisterReplicated(table, baselineExistingRows: true)` now flags a resync for THAT TABLE.**
>     Through `0.2.x` the flag was one bit for the whole database, so baselining one table
>     re-streamed every registered table in full — and on a pair enabling replication late, both
>     nodes did that to each other at once. A snapshot now streams only the flagged tables.
>   - **The database-wide flag survives, unchanged, for the cases that genuinely need it**: an oplog
>     cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of
>     every table is sound. What moved is only where a *baseline* records its debt.
>   - **Additive wire extension, negotiated by capability, not by version.** `Handshake` gains
>     `supports_partial_snapshot`; `SnapshotBegin` gains a `tables` scope where **empty means "all
>     tables"** — which is precisely what a pre-`0.3.0` sender emits and what a pre-`0.3.0` receiver
>     assumes. A peer that does not advertise the capability is served a **full** snapshot, because a
>     scoped one would make it advance `last_applied_remote_seq` past deltas it never received.
>   - **A partial snapshot moves no watermark on either side.** The receiver keeps
>     `last_applied_remote_seq` (it is one number for the whole peer oplog, and the untouched tables'
>     deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The
>     overlap re-delivers rows the snapshot already carried, which LWW absorbs.
>   - **Snapshot debt now clears on acknowledgement, not on enqueue** (new `SnapshotAck`, exchanged
>     only between `0.3.0` peers). A crash part-way through a snapshot re-streams the same scope on
>     reconnect instead of losing it. Against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue
>     behaviour is kept — there is no ack to wait for.
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
>   `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
>   - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
>     open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was
>     written only by graceful disposal — while deregistration deletes the table's oplog AND ledger
>     rows. A crash after a deregister boot could therefore recover the clock below stamps the peer
>     already holds, and LWW discards those as older. The current clock is now flushed inside the
>     deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never
>     regress the anchor).
>   - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped
>     at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a
>     mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now
>     stops draining the gRPC stream, which closes the peer's flow-control window.
>   - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the
>     handshake. That is what makes bounding safe: previously neither side drained its inbox while
>     sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was
>     already safe (snapshot rows and deltas both merge through LWW, the watermark write is
>     MAX-guarded) — only the ordering changed.
>   - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining
>     one table re-streams every registered table, in both directions on a late-enabled pair. Trim
>     high-volume tables (event logs) before turning replication on late.
> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
>   All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
>   the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
>   - **`DeregisterReplicated(table)`** — drops a table's capture triggers and prunes its oplog and
>     row-version rows. A node that does not replicate should not be paying for capture on every
>     write, and until now there was no way to stop once a file had been registered. Idempotent, and
>     usable on triggers left behind by an earlier build.
>   - **`RegisterReplicated(table, baselineExistingRows: true)`** — closes the late-opt-in hole: rows
>     that predate registration were invisible to the peer forever, because the snapshot streamer
>     pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at
>     an LWW-safe floor (HLC `0`, this node's id) and flags a snapshot resync so they actually
>     travel. Default is still off — the documented CDC contract is unchanged unless you ask.
>   - **O(1) backlog depth** — `MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)` replaces a
>     `COUNT(*)` over the whole unacked backlog that ran once a minute AND on every metrics scrape
>     (measured 58 VM steps vs 100,020 on a 50k backlog).
>   - **Byte-budget batching** — `MaxBatchBytes` (default 2 MB) bounds delta and snapshot messages by
>     summed serialized size, with `MaxBatchSize` demoted to a secondary row cap. Consumers no longer
>     have to tune the row count against their widest row to stay under the gRPC message limit.
>     Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
>   - **`LwwApplier` prepares once per statement** instead of three SQL parses per applied row.
>   - **Dropped `__localdb_oplog_hlc`** — an index no query ever used, costing a b-tree insert inside
>     every captured consumer write. Existing files lose it on the next open.
> - `0.1.3` (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN
>   writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq
>   space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's
>   watermark, the rebuilt node started its pump above it, and its first N writes were never sent.
>   Now the watermark is reset when the peer's node id changes, and a peer claiming to have applied
>   more of our stream than we ever produced is treated as evidence that *we* were rebuilt (pump
>   restarts from the beginning; LWW makes re-sending harmless). Found on the OtOpcUa docker-dev rig
>   — it only became reachable once `0.1.2` made back-fill work.
> - `0.1.2` (2026-07-21) — fixes snapshot resync for a **wiped peer**: a converged pair prunes its
>   oplog to empty on ack, and the handshake read an empty oplog as "no gap possible", so a peer
>   that came back with an empty database and a zero watermark was never snapshotted. The gap is
>   now measured against `last_acked_seq` when the oplog is empty.
> - `0.1.1` (2026-07-20) — creates the parent directory of `LocalDb:Path` (a missing directory was
>   a hard boot failure, `SQLite Error 14`).

## Quickstart

### Local-only (no replication)

```csharp
builder.Services.AddZbLocalDb(builder.Configuration, db =>
{
    // onReady: create your own tables, then opt them into (future) replication.
    db.ExecuteAsync(
        "CREATE TABLE IF NOT EXISTS widget (id INTEGER PRIMARY KEY, name TEXT, qty INTEGER)")
      .GetAwaiter().GetResult();
    db.RegisterReplicated("widget");   // PK required; installs capture triggers
});

// Consume ILocalDb anywhere:
await db.ExecuteAsync("INSERT INTO widget (id, name, qty) VALUES (@id, @n, @q)",
    new { id = 1, n = "bolt", q = 42 });
var rows = await db.QueryAsync("SELECT name, qty FROM widget",
    r => (Name: r.GetString(0), Qty: r.GetInt32(1)));
```

`AddZbLocalDb` is first-registration-wins: a second call is a no-op. `RegisterReplicated`
installs capture triggers even without the replication package present — you can register
tables now and turn on sync later.

### Replicated (2 nodes)

Every node registers the passive side (so either can accept a stream) and the initiator
(idle unless a `PeerAddress` is set). Configure exactly **one** side with the peer's
address; the other stays passive.

```csharp
// Both nodes:
builder.Services.AddZbLocalDb(builder.Configuration, db => { /* create + RegisterReplicated */ });
builder.Services.AddGrpc();
builder.Services.AddZbLocalDbReplication(builder.Configuration);   // passive host + initiator + maintenance

var app = builder.Build();
app.MapZbLocalDbSync();   // passive gRPC endpoint — host must have AddGrpc()
app.Run();
```

```jsonc
// Node A — the INITIATOR: dials node B
"LocalDb": {
  "Path": "nodeA.db",
  "Replication": { "PeerAddress": "https://nodeB:5001", "ApiKey": "…" }
}
// Node B — PASSIVE: no PeerAddress, its SyncBackgroundService idles and only serves the endpoint
"LocalDb": { "Path": "nodeB.db", "Replication": { } }
```

Both directions flow on the single bidirectional stream the initiator opens; the passive
node still pushes its own deltas back. Only one side needs `PeerAddress`.

## Configuration

### `LocalDb` section (`LocalDbOptions`)

| Key | Default | Meaning |
|---|---|---|
| `Path` | `""` (required) | Filesystem path to the SQLite database file. |
| `BusyTimeoutMs` | `5000` | SQLite `busy_timeout` pragma, in milliseconds. |
| `Synchronous` | `"NORMAL"` | SQLite `synchronous` pragma (`OFF` / `NORMAL` / `FULL`). |

### `LocalDb:Replication` section (`ReplicationOptions`)

| Key | Default | Meaning |
|---|---|---|
| `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). |
| `ApiKey` | `null` | Bearer token the initiator presents on its channel; `null` ⇒ no key header. |
| `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. |
| `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
| `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
| `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
| `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
| `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
| `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. |
| `MaxHlcDriftAhead` | `5 min` | Max tolerated HLC drift where the peer's clock runs ahead of ours. |
| `FailClosedOnDrift` | `false` | When true, an over-drift entry is dead-lettered instead of applied. |

All options are validated at startup (`ZB.MOM.WW.Configuration` validators).

## How it works

- **Capture** — three AFTER triggers per registered table write each change into one
  shared, HLC-stamped `__localdb_oplog` plus a `__localdb_row_version` ledger, in the
  consumer's own transaction. A DB-scoped `__localdb_applying` guard makes replicated
  applies skip re-capture.
- **Convergence** — last-writer-wins: compare incoming HLC, then `node_id` ordinal as the
  tie-break; the strict-greater winner is applied, the loser discarded (but still acked).
- **Streaming** — one long-lived bidirectional gRPC stream. The initiator dials; both
  sides push oplog batches above the peer's watermark and ack the applied watermark back,
  which drives pruning. Outbound uses a single writer fed by two lanes (an unbounded
  control lane for handshake/acks that always preempts a bounded data lane for
  delta/snapshot batches), so receiving never blocks on the ability to ack.
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
  full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
  the snapshot's as-of seq. When instead a single table was *baselined*, only that table is
  streamed and **no watermark moves on either side** — a partial snapshot speaks for nothing
  outside its own tables, so the deltas it overlaps are still pumped and merged by LWW.
  **Each node decides from its OWN state** whether it owes the peer a snapshot, and a peer
  that does not advertise `supports_partial_snapshot` is sent a full one. The snapshot is
  streamed *after* the handshake, alongside a running receive loop, so a node sending one
  still applies what the peer sends it — a mutual snapshot exchanges under flow control
  instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
  cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
  and prunes expired tombstones every `MaintenanceInterval`.

## Semantics worth knowing

- **PK-column updates** are captured as delete-of-old-PK + insert-of-new — a tombstone on
  the old key plus a fresh insert (full delete+insert support).
- **Upserts** (`INSERT … ON CONFLICT DO UPDATE`) are fully supported (the row-version
  trigger uses an explicit `ON CONFLICT` clause so it is not rewritten by the outer
  statement's conflict algorithm).
- **Clock-drift guard** — warn-once per session when a peer entry's HLC physical time is
  more than `MaxHlcDriftAhead` ahead. With `FailClosedOnDrift`, each over-drift entry is
  dead-lettered; otherwise it is applied (the HLC merge absorbs the skew). The **session
  never dies from drift** — no livelock on reconnect.
- **Writes never block on the peer**; local commit is independent of replication.
- **All timestamps are UTC** (HLC physical component is UTC Unix milliseconds).

## Limitations (honest list)

- **Exactly two nodes.** No mesh / N-node topologies.
- **LWW only.** No per-table pluggable conflict resolvers.
- **BLOB columns are rejected at registration** (a column declared `BLOB` fails
  `RegisterReplicated`, because `json_object` cannot capture it). A *typeless* column can
  still hold a BLOB value at runtime, which then surfaces as a write-time error, not a
  registration error.
- **A replicated table may not be the parent of a row-mutating `ON DELETE` foreign key**
  (`CASCADE` / `SET NULL` / `SET DEFAULT` from any other table, or from itself). Applying a
  remote row uses `INSERT OR REPLACE`, whose implicit delete fires that action with change
  capture suppressed — the child rows would change on the applying node only and never
  replicate, diverging the pair undetectably. `RegisterReplicated` refuses such a table
  (`0.3.3`); use `ON DELETE NO ACTION`/`RESTRICT` and delete children explicitly, or leave
  the parent unreplicated. The check runs at registration and reads the whole database's
  foreign keys, so a child table created *after* registration is caught at the next startup,
  not at the moment of the `CREATE TABLE`.
- **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
  and are not snapshotted; only changes after registration are replicated. **Opt out with
  `RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
  ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync
  **for that table** so they reach the peer. The floor is below every genuine stamp, so a baselined row loses
  every comparison against a real remote write and wins only where the peer has no version
  of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
  the node-id tie-break — convergent, but arbitrary as to which content survives.
- **A snapshot still ships whole tables.** The flag is per-table as of `0.3.0`, so baselining
  one table streams that table and no other — but *that* table travels in full, and two
  cases still stream **everything**: an oplog cap prune / impossible peer ack (delta
  continuity is lost for the oplog as a whole), and any peer older than `0.3.0` (which
  cannot be told a snapshot is partial). So **purge or trim high-volume tables such as event
  logs before baselining them**, and expect one slow first sync. Bounded either way (batch
  budget out, bounded inbox in).
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
  SQLite↔SQLite peers this library targets).
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
- **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so
  peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far
  less in practice — messages are only that large when the sender has that much backlog).
  Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being
  drained and the peer is slowed by flow control.

## Observability

Metrics publish under the single meter **`ZB.MOM.WW.LocalDb.Replication`** (all instruments
null-guarded — replication runs identically whether or not metrics are collected):

| Instrument | Kind | Notes |
|---|---|---|
| `localdb.oplog.depth` | ObservableGauge | Unacked oplog backlog (polled cheaply at scrape time). |
| `localdb.sync.lag.seconds` | ObservableGauge | Seconds since the last successful sync (no measurement when never synced). |
| `localdb.sync.applied` | Counter | Rows applied from inbound **delta** batches. |
| `localdb.sync.conflicts_discarded` | Counter | Inbound rows that lost LWW. |
| `localdb.sync.dead_lettered` | Counter | Poison / over-drift rows routed to the dead-letter table. |
| `localdb.sync.reconnects` | Counter | Reconnect attempts (initiator). |
| `localdb.sync.snapshots` | Counter | Snapshots, tagged `direction=sent`/`received` (one increment per snapshot). |

**Counter split to note:** rows merged from a **snapshot** are *not* counted under
`localdb.sync.applied`; a completed snapshot instead increments `localdb.sync.snapshots`
once (with its `direction` tag). Only delta-batch rows increment `localdb.sync.applied`.

`ISyncStatus` (registered singleton) surfaces live state for dashboards / health:
`Connected` (true while **any** session is active — a node can run both roles at once),
`PeerNodeId`, `LastSyncUtc`, `OplogBacklog` (**nullable** — null means unknown because no
provider is wired or the poll threw, so a DB failure never reads as "0 backlog"), and
`ConnectionAttempts`.

## Ops notes

- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
  **database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
  full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
  baselining writes.
- **Upgrading a pair to `0.3.0`** needs no coordination: the two nodes negotiate scoped snapshots
  by capability, so a `0.3.0` node paired with a `0.2.x` one simply keeps sending full snapshots
  until both are upgraded. Rolling the file forward is one-way — a `0.2.x` build reopening a v3
  file leaves `__localdb_snapshot_state` in place and ignores it, so any per-table debt recorded
  by `0.3.0` is invisible to it (roll forward again, or re-baseline).
- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
  its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
  registered-table digests fail-closed, so a node that drops a table its peer still
  replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
  neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
  accumulated while capture was off stay invisible to the peer — but trim high-volume
  tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's
  durable anchor is flushed in the same transaction, so a crash after a deregister boot
  cannot bring the clock back below stamps peers already hold.
- **On-disk schema upgrades** are applied in place when the file is opened, in one
  transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
  local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
  keeps syncing.
- The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
  entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
  rejects.
- **Peer auth**: `ApiKey` is sent by the initiator as a static `Authorization: Bearer …`
  header on its channel. The library's passive service does **not** itself verify the key
  — the passive host enforces whatever gRPC auth the host application configures (e.g.
  `ZB.MOM.WW.Auth`). Configure host-side authorization to actually gate inbound streams.

## Testing

199 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
generation, LWW decision table, watermark/pruning, digest computation, schema-version
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
snapshot over a capacity-1 transport, single-table baseline streaming only that table,
mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
Family conventions apply (`TreatWarningsAsErrors`).

Build / test from this directory:

```bash
dotnet build ZB.MOM.WW.LocalDb.slnx
dotnet test  ZB.MOM.WW.LocalDb.slnx
```

See the design doc: [`docs/plans/2026-07-17-localdb-design.md`](../docs/plans/2026-07-17-localdb-design.md).