scadaproj/ZB.MOM.WW.LocalDb/README.md

# ZB.MOM.WW.LocalDb

An embedded **SQLite local cache** with **optional bidirectional async 2-node
replication over gRPC**, for the ZB.MOM.WW SCADA / OT family. It gives an application a
fast, crash-safe local store that can optionally keep a second node in sync
asynchronously — local writes **never** block on the peer, and a node that never enables
replication is simply a fast local DB.

Change capture uses SQLite triggers writing a single HLC-stamped oplog inside the
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
last-writer-wins over a hybrid logical clock. All timestamps are UTC.

## Packages (all `0.3.7`; the feed carries up to `0.3.4` — `0.3.5` through `0.3.7` are unpublished)

| Package | Contents |
|---|---|
| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK + BLOB + foreign-key validation, trigger install, oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). |
| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. |

> Status: **published to the Gitea feed (through `0.3.4`); adopted by ScadaBridge (Phases 1+2) and
> OtOpcUa (Phases 1+2), both pinned to `0.3.4`.**
>
> - `0.3.7` (unpublished) — **a deleted row no longer comes back when the peer was offline longer
>   than `TombstoneRetention`** (scadaproj#16). Tombstones used to be pruned on age alone; once the
>   delete's oplog row had also aged out, the snapshot (built from the row-version ledger) could not
>   carry the delete, and the returning peer's older update re-created the row on **both** nodes. A
>   tombstone is now pruned only when it is past retention **and** the peer holds it — no outstanding
>   snapshot debt (database-wide or for its table) and, for a local delete, no longer in the unacked
>   oplog — and its retention clock restarts when it is delivered (delta ack or snapshot discharge).
>   New optional `TombstoneHardCap` (default off) bounds how long an undelivered tombstone is kept.
>   Wire-compatible: no proto, `lib_schema_version` or on-disk schema change.
> - `0.3.6` (unpublished) — code-review fixes (scadaproj#36–#39, #46, #50). Wire-compatible: no proto
>   change, no `lib_schema_version` bump, no on-disk schema change.
>   - The passive endpoint enforces `ApiKey` itself (Bearer, constant-time, before the handshake).
>   - Snapshots deliver FK-linked rows in any order (parents first; FK-failing rows deferred and
>     retried; true orphans dead-lettered at completion), with the held-back rows capped by the new
>     `MaxSnapshotDeferredRows`.
>   - Remote rows are upserted on the primary key (`ON CONFLICT(<pk>) DO UPDATE`), not
>     `INSERT OR REPLACE`, so a secondary-`UNIQUE` clash is a dead-letter, not a silent delete.
>   - `FailClosedOnDrift` covers snapshot rows.
>   - Snapshot-path dead-letters are counted in `localdb.sync.dead_lettered`.
>   - Column names containing `'` replicate (they broke the apply/snapshot SQL).
>   - **`RegisterReplicated` refuses an `AUTOINCREMENT` primary key** (see Limitations) — the one
>     change that can refuse a schema an earlier version accepted.
> - `0.3.5` (unpublished) — `Microsoft.Data.Sqlite` 10.0.7 → 10.0.13 (efcore#39009 pool fault) and
>   the `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 → 2.1.13. No wire or schema change.
> - `0.3.4` (2026-08-20) — **a converged pair no longer re-streams a full snapshot on every
>   reconnect once the sender's oplog has been ack-pruned to empty.** Found on the ScadaBridge
>   docker rig (its register row 106): a rebuilt node (watermark 0) was healed by a snapshot
>   stamped `as_of_seq = MAX(seq)` — which is 0 on an empty oplog — so its MAX-guarded watermark
>   stayed 0, and the healthy side's empty-oplog horizon rule (`0.1.2`: streamable from
>   `last_acked_seq + 1`) saw a gap on every later handshake and re-sent the whole database
>   (127k rows per restart on the rig) until it happened to write a new row. The as-of is now
>   `max(MAX(seq), last_acked_seq)`: every delta pruned below `last_acked_seq` is already folded
>   into the rows the snapshot carries, so that is the floor it may claim. Sender-side only,
>   wire-compatible, no schema change; pinned by
>   `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot`.
> - `0.3.3` (2026-08-20) — two latent apply-path hazards from the ScadaBridge adversarial
>   architecture review (P4-7). Neither had a live trigger in any current consumer; both are the
>   kind that surface as silent divergence, not as an error. Wire-compatible: no proto change, no
>   `lib_schema_version` bump, no on-disk schema change.
>   - **A replicated table may no longer be the parent of a row-mutating `ON DELETE` foreign key.**
>     The apply path writes remote rows with `INSERT OR REPLACE`, and REPLACE resolves a PK conflict
>     by DELETING the existing row first. Connections run `PRAGMA foreign_keys=ON`, so that implicit
>     delete fires the child's `ON DELETE CASCADE` / `SET NULL` / `SET DEFAULT` — while the capture
>     guard is raised. The child rows change on the applying node ONLY, produce no oplog entry, and
>     the pair diverges with nothing able to detect it (LWW compares only rows it was told about).
>     `RegisterReplicated` now inspects every foreign key in the database and refuses such a table
>     with a `LocalDbRegistrationException` naming the offending child column and action.
>     `RESTRICT` / `NO ACTION` remain fine — they abort the statement instead of mutating rows
>     behind capture's back, which the apply path already handles as per-entry poison. Checked at
>     registration; SQLite cannot add a foreign key to an existing table, so a child table created
>     later is caught at the next startup's registration rather than at the DDL.
>   - **An entry's row write and its ledger write are now atomic with each other.** They are two
>     statements, and SQLite aborts only the failing one — so a statement-level fault between them
>     (the case the per-entry poison catch exists for) committed a half-applied entry: a tombstone
>     whose table row was gone while `__localdb_row_version` still called it live, which the snapshot
>     streamer can only warn about and skip, so the peer never learned the row died; or an upsert
>     whose new row carried the OLD version and therefore lost the next comparison it should have
>     won. The pair is wrapped in a per-entry `SAVEPOINT` that is rolled back before dead-lettering,
>     so a poison entry leaves both writes or neither and the rest of the batch is untouched.
> - `0.3.2` (2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review
>   (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, no `lib_schema_version`
>   bump, no on-disk schema change — the keepalive is transport-level and the rest is local policy.
>   - **A live session now ends itself when a cap prune eats deltas it has not sent.** The pump reads
>     `seq > sentThruSeq`, which resumes SILENTLY above a pruned hole; the peer acks the later seqs
>     and the pair is durably divergent for the whole life of that session. `needs_snapshot` (set in
>     the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable
>     LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares
>     the oplog horizon (`MIN(seq)`, one b-tree seek) against its sent watermark and throws
>     `LocalDbOplogGapException` when deltas it still owes are gone. The horizon, not the flag,
>     because the flag is also raised for reasons that do not invalidate the running session.
>   - **The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins.**
>     Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash
>     or partition of the dialling node left the passive node's handler parked in `MoveNext` on a
>     half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted
>     peer's reconnects with `AlreadyExists`, indefinitely on an idle pair. The initiator now dials
>     with `KeepAlivePingDelay` / `KeepAlivePingTimeout` (new options, 60s/20s, `Always` policy so
>     idle connections ping too), and a fresh authenticated `Sync` cancels the incumbent and waits
>     for it to unwind before starting — the guard serializes sessions, it was never a fence, and an
>     authenticated same-key dialler is by definition the peer. An incumbent that will not yield
>     within the takeover timeout still gets `AlreadyExists`, because two live sessions over one
>     database is worse than a refused dial.
> - `0.3.1` (2026-08-18) — a faulting session flushes its control lane before cancelling its writer,
>   so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to
>   reconnect-loop against a generic disconnect.
> - `0.3.0` (2026-08-15) — **the snapshot-resync flag is per-table.** On-disk bookkeeping schema
>   goes to **v3** (new `__localdb_snapshot_state`, upgraded in place on open); wire-compatible with
>   `0.1.x`/`0.2.x` peers, and `lib_schema_version` deliberately stays `1` because it is compared
>   fail-closed for equality.
>   - **`RegisterReplicated(table, baselineExistingRows: true)` now flags a resync for THAT TABLE.**
>     Through `0.2.x` the flag was one bit for the whole database, so baselining one table
>     re-streamed every registered table in full — and on a pair enabling replication late, both
>     nodes did that to each other at once. A snapshot now streams only the flagged tables.
>   - **The database-wide flag survives, unchanged, for the cases that genuinely need it**: an oplog
>     cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of
>     every table is sound. What moved is only where a *baseline* records its debt.
>   - **Additive wire extension, negotiated by capability, not by version.** `Handshake` gains
>     `supports_partial_snapshot`; `SnapshotBegin` gains a `tables` scope where **empty means "all
>     tables"** — which is precisely what a pre-`0.3.0` sender emits and what a pre-`0.3.0` receiver
>     assumes. A peer that does not advertise the capability is served a **full** snapshot, because a
>     scoped one would make it advance `last_applied_remote_seq` past deltas it never received.
>   - **A partial snapshot moves no watermark on either side.** The receiver keeps
>     `last_applied_remote_seq` (it is one number for the whole peer oplog, and the untouched tables'
>     deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The
>     overlap re-delivers rows the snapshot already carried, which LWW absorbs.
>   - **Snapshot debt now clears on acknowledgement, not on enqueue** (new `SnapshotAck`, exchanged
>     only between `0.3.0` peers). A crash part-way through a snapshot re-streams the same scope on
>     reconnect instead of losing it. Against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue
>     behaviour is kept — there is no ack to wait for.
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
>   `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
>   - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
>     open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was
>     written only by graceful disposal — while deregistration deletes the table's oplog AND ledger
>     rows. A crash after a deregister boot could therefore recover the clock below stamps the peer
>     already holds, and LWW discards those as older. The current clock is now flushed inside the
>     deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never
>     regress the anchor).
>   - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped
>     at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a
>     mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now
>     stops draining the gRPC stream, which closes the peer's flow-control window.
>   - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the
>     handshake. That is what makes bounding safe: previously neither side drained its inbox while
>     sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was
>     already safe (snapshot rows and deltas both merge through LWW, the watermark write is
>     MAX-guarded) — only the ordering changed.
>   - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining
>     one table re-streams every registered table, in both directions on a late-enabled pair. Trim
>     high-volume tables (event logs) before turning replication on late.
> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
>   All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
>   the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
>   - **`DeregisterReplicated(table)`** — drops a table's capture triggers and prunes its oplog and
>     row-version rows. A node that does not replicate should not be paying for capture on every
>     write, and until now there was no way to stop once a file had been registered. Idempotent, and
>     usable on triggers left behind by an earlier build.
>   - **`RegisterReplicated(table, baselineExistingRows: true)`** — closes the late-opt-in hole: rows
>     that predate registration were invisible to the peer forever, because the snapshot streamer
>     pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at
>     an LWW-safe floor (HLC `0`, this node's id) and flags a snapshot resync so they actually
>     travel. Default is still off — the documented CDC contract is unchanged unless you ask.
>   - **O(1) backlog depth** — `MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)` replaces a
>     `COUNT(*)` over the whole unacked backlog that ran once a minute AND on every metrics scrape
>     (measured 58 VM steps vs 100,020 on a 50k backlog).
>   - **Byte-budget batching** — `MaxBatchBytes` (default 2 MB) bounds delta and snapshot messages by
>     summed serialized size, with `MaxBatchSize` demoted to a secondary row cap. Consumers no longer
>     have to tune the row count against their widest row to stay under the gRPC message limit.
>     Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
>   - **`LwwApplier` prepares once per statement** instead of three SQL parses per applied row.
>   - **Dropped `__localdb_oplog_hlc`** — an index no query ever used, costing a b-tree insert inside
>     every captured consumer write. Existing files lose it on the next open.
> - `0.1.3` (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN
>   writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq
>   space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's
>   watermark, the rebuilt node started its pump above it, and its first N writes were never sent.
>   Now the watermark is reset when the peer's node id changes, and a peer claiming to have applied
>   more of our stream than we ever produced is treated as evidence that *we* were rebuilt (pump
>   restarts from the beginning; LWW makes re-sending harmless). Found on the OtOpcUa docker-dev rig
>   — it only became reachable once `0.1.2` made back-fill work.
> - `0.1.2` (2026-07-21) — fixes snapshot resync for a **wiped peer**: a converged pair prunes its
>   oplog to empty on ack, and the handshake read an empty oplog as "no gap possible", so a peer
>   that came back with an empty database and a zero watermark was never snapshotted. The gap is
>   now measured against `last_acked_seq` when the oplog is empty.
> - `0.1.1` (2026-07-20) — creates the parent directory of `LocalDb:Path` (a missing directory was
>   a hard boot failure, `SQLite Error 14`).

## Quickstart

### Local-only (no replication)

```csharp
builder.Services.AddZbLocalDb(builder.Configuration, db =>
{
    // onReady: create your own tables, then opt them into (future) replication.
    db.ExecuteAsync(
        "CREATE TABLE IF NOT EXISTS widget (id INTEGER PRIMARY KEY, name TEXT, qty INTEGER)")
      .GetAwaiter().GetResult();
    db.RegisterReplicated("widget");   // PK required; installs capture triggers
    // The application assigns widget ids. A key SQLite assigns per node (AUTOINCREMENT, or a NULL
    // rowid) collides across a pair: AUTOINCREMENT is refused, prefer TEXT GUIDs. See Limitations.
});

// Consume ILocalDb anywhere:
await db.ExecuteAsync("INSERT INTO widget (id, name, qty) VALUES (@id, @n, @q)",
    new { id = 1, n = "bolt", q = 42 });
var rows = await db.QueryAsync("SELECT name, qty FROM widget",
    r => (Name: r.GetString(0), Qty: r.GetInt32(1)));
```

`AddZbLocalDb` is first-registration-wins: a second call is a no-op. `RegisterReplicated`
installs capture triggers even without the replication package present — you can register
tables now and turn on sync later.

### Replicated (2 nodes)

Every node registers the passive side (so either can accept a stream) and the initiator
(idle unless a `PeerAddress` is set). Configure exactly **one** side with the peer's
address; the other stays passive.

```csharp
// Both nodes:
builder.Services.AddZbLocalDb(builder.Configuration, db => { /* create + RegisterReplicated */ });
builder.Services.AddGrpc();
builder.Services.AddZbLocalDbReplication(builder.Configuration);   // passive host + initiator + maintenance

var app = builder.Build();
app.MapZbLocalDbSync();   // passive gRPC endpoint — host must have AddGrpc()
app.Run();
```

```jsonc
// Node A — the INITIATOR: dials node B
"LocalDb": {
  "Path": "nodeA.db",
  "Replication": { "PeerAddress": "https://nodeB:5001", "ApiKey": "…" }
}
// Node B — PASSIVE: no PeerAddress, its SyncBackgroundService idles and only serves the endpoint
"LocalDb": { "Path": "nodeB.db", "Replication": { } }
```

Both directions flow on the single bidirectional stream the initiator opens; the passive
node still pushes its own deltas back. Only one side needs `PeerAddress`.

## Configuration

### `LocalDb` section (`LocalDbOptions`)

| Key | Default | Meaning |
|---|---|---|
| `Path` | `""` (required) | Filesystem path to the SQLite database file. |
| `BusyTimeoutMs` | `5000` | SQLite `busy_timeout` pragma, in milliseconds. |
| `Synchronous` | `"NORMAL"` | SQLite `synchronous` pragma (`OFF` / `NORMAL` / `FULL` / `EXTRA`, case-insensitive). |

### `LocalDb:Replication` section (`ReplicationOptions`)

| Key | Default | Meaning |
|---|---|---|
| `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). |
| `ApiKey` | `null` | Shared pair key — set the **same** value on both nodes. The initiator sends it as `Authorization: Bearer <key>`; since 0.3.6 the passive endpoint **requires** it and refuses any other stream with `Unauthenticated`. Blank ⇒ no header sent and no check (any caller is accepted; a startup warning is logged). |
| `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. |
| `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
| `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
| `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
| `TombstoneRetention` | `7 days` | How long a deleted-row tombstone is kept **after the peer has received it** (`0.3.7`: pruned only once past this AND delivered — delta acked or snapshot discharged; the clock restarts at delivery). An undelivered tombstone is kept for the whole outage. |
| `TombstoneHardCap` | `null` (off) | Optional absolute ceiling on keeping an **undelivered** tombstone (`0.3.7`). Bounds ledger growth when the peer is gone for good; a peer that returns after more than this can resurrect deleted rows, so rebuild it instead. Must be ≥ `TombstoneRetention`. |
| `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
| `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. |
| `MaxSnapshotDeferredRows` | `100_000` | Ceiling on snapshot rows held in memory while their FK parent has not arrived (`0.3.6`). Past it the oldest are dead-lettered (and counted). |
| `MaxHlcDriftAhead` | `5 min` | Max tolerated HLC drift where the peer's clock runs ahead of ours. |
| `FailClosedOnDrift` | `false` | When true, an over-drift entry is dead-lettered instead of applied. |

All options are validated at startup (`ZB.MOM.WW.Configuration` validators).

## How it works

- **Capture** — three AFTER triggers per registered table write each change into one
  shared, HLC-stamped `__localdb_oplog` plus a `__localdb_row_version` ledger, in the
  consumer's own transaction. A DB-scoped `__localdb_applying` guard makes replicated
  applies skip re-capture.
- **Convergence** — last-writer-wins: compare incoming HLC, then `node_id` ordinal as the
  tie-break; the strict-greater winner is applied, the loser discarded (but still acked).
- **Streaming** — one long-lived bidirectional gRPC stream. The initiator dials; both
  sides push oplog batches above the peer's watermark and ack the applied watermark back,
  which drives pruning. Outbound uses a single writer fed by two lanes (an unbounded
  control lane for handshake/acks that always preempts a bounded data lane for
  delta/snapshot batches), so receiving never blocks on the ability to ack.
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
  full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
  the snapshot's as-of seq. When instead a single table was *baselined*, only that table is
  streamed and **no watermark moves on either side** — a partial snapshot speaks for nothing
  outside its own tables, so the deltas it overlaps are still pumped and merged by LWW.
  **Each node decides from its OWN state** whether it owes the peer a snapshot, and a peer
  that does not advertise `supports_partial_snapshot` is sent a full one. The snapshot is
  streamed *after* the handshake, alongside a running receive loop, so a node sending one
  still applies what the peer sends it — a mutual snapshot exchanges under flow control
  instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
  cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
  and prunes expired tombstones every `MaintenanceInterval` — only those the peer is known to
  hold (`0.3.7`; see Ops notes).

## Semantics worth knowing

- **PK-column updates** are captured as delete-of-old-PK + insert-of-new — a tombstone on
  the old key plus a fresh insert (full delete+insert support).
- **Upserts** (`INSERT … ON CONFLICT DO UPDATE`) are fully supported (the row-version
  trigger uses an explicit `ON CONFLICT` clause so it is not rewritten by the outer
  statement's conflict algorithm).
- **Clock-drift guard** — warn-once per session when a peer entry's HLC physical time is
  more than `MaxHlcDriftAhead` ahead. With `FailClosedOnDrift`, each over-drift entry is
  dead-lettered; otherwise it is applied (the HLC merge absorbs the skew). The **session
  never dies from drift** — no livelock on reconnect. Since `0.3.6` the guard covers
  **snapshot rows** too (warn once per snapshot; dead-lettered under `FailClosedOnDrift`, and
  their HLC is neither observed nor persisted as `last_seen_hlc`). Up to `0.3.5` a snapshot
  applied every row unchecked, so one future-stamped row could push the receiver's clock
  ahead permanently.
- **Writes never block on the peer**; local commit is independent of replication.
- **All timestamps are UTC** (HLC physical component is UTC Unix milliseconds).

## Limitations (honest list)

- **Exactly two nodes.** No mesh / N-node topologies.
- **LWW only.** No per-table pluggable conflict resolvers.
- **Replicated primary keys must be unique across both nodes.** LWW keys on the primary key,
  so two different rows inserted on the two nodes under the same key are read as two versions
  of one row, and one is lost on **both** nodes with no dead-letter and no metric. A key each
  node assigns on its own cannot replicate: `RegisterReplicated` **refuses an `AUTOINCREMENT`
  key** (`0.3.6`), and a plain `INTEGER PRIMARY KEY` is accepted only because the application
  may assign it from a node-scoped range — left `NULL` on insert it loses rows the same way.
  Use a TEXT GUID minted by the writer (what ScadaBridge's `site_events` and OtOpcUa's alarm
  store-and-forward did when they moved onto LocalDb).
- **BLOB columns are rejected at registration** (a column declared `BLOB` fails
  `RegisterReplicated`, because `json_object` cannot capture it). A *typeless* column can
  still hold a BLOB value at runtime, which then surfaces as a write-time error, not a
  registration error.
- **A replicated table may not be the parent of a row-mutating `ON DELETE` foreign key**
  (`CASCADE` / `SET NULL` / `SET DEFAULT` from any other table, or from itself). Applying a
  remote tombstone deletes the row (and up to `0.3.5` applying any remote row used
  `INSERT OR REPLACE`, whose implicit delete did the same), which fires that action with change
  capture suppressed — the child rows would change on the applying node only and never
  replicate, diverging the pair undetectably. `RegisterReplicated` refuses such a table
  (`0.3.3`); use `ON DELETE NO ACTION`/`RESTRICT` and delete children explicitly, or leave
  the parent unreplicated. The check runs at registration and reads the whole database's
  foreign keys, so a child table created *after* registration is caught at the next startup,
  not at the moment of the `CREATE TABLE`.
- **Foreign keys between replicated tables (`NO ACTION`/`RESTRICT`) survive a snapshot as of
  `0.3.6`.** The sender streams referenced tables before the tables that reference them, and
  the receiver defers a snapshot row that fails only its FK check, retrying it as more rows
  land; one whose parent never arrives is dead-lettered at `SnapshotComplete`. Before `0.3.6`
  a child that arrived before its parent (name order, or a self-reference whose parent's key
  sorts later as `pk_json` text) was dead-lettered for good. **Mixed pairs:** a pre-`0.3.6`
  receiver still loses self-referencing rows that arrive child-first, and a pre-`0.3.6`
  sender's name order relies on the `0.3.6` receiver's deferral — so upgrade both nodes before
  adding FK-linked tables. Deferred rows are held in memory until the snapshot completes, up
  to `MaxSnapshotDeferredRows` (default 100,000); past that the oldest are dead-lettered rather
  than buffered without bound — an older, name-ordered sender whose child table sorts first can
  otherwise defer a whole table.
- **Secondary `UNIQUE` keys must be unique across both nodes.** Remote rows are applied with
  `INSERT … ON CONFLICT(<pk>) DO UPDATE` (`0.3.6`), so only the row with the same primary key
  is touched. If the same natural key is created under different primary keys on the two nodes
  (e.g. during a partition), each node keeps its own row and dead-letters the peer's — visible in
  `__localdb_dead_letter`, but the pair holds different rows until the conflict is resolved by
  hand. Up to `0.3.5` `INSERT OR REPLACE` silently deleted the local row instead and the pair
  diverged undetectably. Prefer GUID or node-scoped natural keys.
- **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
  and are not snapshotted; only changes after registration are replicated. **Opt out with
  `RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
  ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync
  **for that table** so they reach the peer. The floor is below every genuine stamp, so a baselined row loses
  every comparison against a real remote write and wins only where the peer has no version
  of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
  the node-id tie-break — convergent, but arbitrary as to which content survives.
- **A snapshot still ships whole tables.** The flag is per-table as of `0.3.0`, so baselining
  one table streams that table and no other — but *that* table travels in full, and two
  cases still stream **everything**: an oplog cap prune / impossible peer ack (delta
  continuity is lost for the oplog as a whole), and any peer older than `0.3.0` (which
  cannot be told a snapshot is partial). So **purge or trim high-volume tables such as event
  logs before baselining them**, and expect one slow first sync. Bounded either way (batch
  budget out, bounded inbox in).
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
  SQLite↔SQLite peers this library targets).
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
- **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so
  peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far
  less in practice — messages are only that large when the sender has that much backlog).
  Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being
  drained and the peer is slowed by flow control.

## Observability

Metrics publish under the single meter **`ZB.MOM.WW.LocalDb.Replication`** (all instruments
null-guarded — replication runs identically whether or not metrics are collected):

| Instrument | Kind | Notes |
|---|---|---|
| `localdb.oplog.depth` | ObservableGauge | Unacked oplog backlog (polled cheaply at scrape time). |
| `localdb.sync.lag.seconds` | ObservableGauge | Seconds since the last successful sync (no measurement when never synced). |
| `localdb.sync.applied` | Counter | Rows applied from inbound **delta** batches. |
| `localdb.sync.conflicts_discarded` | Counter | Inbound rows that lost LWW. |
| `localdb.sync.dead_lettered` | Counter | Rows routed to the dead-letter table — poison / over-drift rows from deltas **and** (since `0.3.6`) snapshots, including FK orphans and rows spilled past `MaxSnapshotDeferredRows`. |
| `localdb.sync.reconnects` | Counter | Reconnect attempts (initiator). |
| `localdb.sync.snapshots` | Counter | Snapshots, tagged `direction=sent`/`received` (one increment per snapshot). |

**Counter split to note:** rows merged from a **snapshot** are *not* counted under
`localdb.sync.applied`; a completed snapshot instead increments `localdb.sync.snapshots`
once (with its `direction` tag). Only delta-batch rows increment `localdb.sync.applied`.

`ISyncStatus` (registered singleton) surfaces live state for dashboards / health:
`Connected` (true while **any** session is active — a node can run both roles at once),
`PeerNodeId`, `LastSyncUtc`, `OplogBacklog` (**nullable** — null means unknown because no
provider is wired or the poll threw, so a DB failure never reads as "0 backlog"), and
`ConnectionAttempts`.

## Ops notes

- **Tombstones and long outages** (`0.3.7`, scadaproj#16). A tombstone is what lets LWW reject an
  older write of a deleted row, so it is kept until the peer has it: while the database or its
  table owes the peer a snapshot (the snapshot is built from the ledger, so a pruned tombstone is a
  delete it can no longer carry), and while a local delete is still in the unacked oplog. Delivery
  (a delta ack covering the delete, or the snapshot debt being discharged) restamps
  `tombstone_utc`, and `TombstoneRetention` runs from there — the returning peer's own stale writes
  get a full retention to drain. The cost: **while the peer is down, every delete's tombstone stays
  in `__localdb_row_version`** (high-churn tables such as a store-and-forward queue grow the ledger
  for the whole outage). Set `TombstoneHardCap` to bound that; past it the pre-`0.3.7` resurrection
  risk returns, so treat a peer offline longer than the cap as one to rebuild, not reconnect. Not
  covered by any setting: a peer **restored from an old backup** of its own file presents stale rows
  under its old identity and watermarks — rebuild it empty instead.

- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
  **database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
  full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
  baselining writes.
- **Upgrading a pair to `0.3.0`** needs no coordination: the two nodes negotiate scoped snapshots
  by capability, so a `0.3.0` node paired with a `0.2.x` one simply keeps sending full snapshots
  until both are upgraded. Rolling the file forward is one-way — a `0.2.x` build reopening a v3
  file leaves `__localdb_snapshot_state` in place and ignores it, so any per-table debt recorded
  by `0.3.0` is invisible to it (roll forward again, or re-baseline).
- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
  its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
  registered-table digests fail-closed, so a node that drops a table its peer still
  replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
  neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
  accumulated while capture was off stay invisible to the peer — but trim high-volume
  that table first (see Limitations: baselining streams that one table in full, not every
  table — only an oplog cap prune or a pre-`0.3.0` peer streams everything). The HLC's
  durable anchor is flushed in the same transaction, so a crash after a deregister boot
  cannot bring the clock back below stamps peers already hold.
- **On-disk schema upgrades** are applied in place when the file is opened, in one
  transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
  local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
  keeps syncing.
- The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
  entries), over-drift rejects (when `FailClosedOnDrift`), snapshot rows whose FK parent never
  arrived, and secondary-`UNIQUE` clashes. Every row written there, on the delta and the
  snapshot path, is counted in `localdb.sync.dead_lettered` (snapshot-path rows were not
  counted before `0.3.6`). Inspect it to diagnose rejects.
- **Peer auth** (0.3.6+): with `ApiKey` set, the initiator sends `Authorization: Bearer <key>`
  and the passive `LocalDbSyncService` verifies it (constant-time) **before** the handshake and
  before it displaces the current stream — a caller without the key gets `Unauthenticated`, sees
  no table names or schema digests, and cannot evict the real peer. With `ApiKey` blank on a
  node, that node's endpoint accepts any caller (exactly the pre-0.3.6 behaviour) and logs a
  warning at startup; protect it by network isolation or a host interceptor. Up to 0.3.5 the
  library never checked the key, so ScadaBridge and OtOpcUa each added a fail-closed
  `LocalDbSyncAuthInterceptor` over the same key — those keep working unchanged and are now
  redundant with the library check (they still add fail-closed-when-blank, which the library
  deliberately does not, so a key-less pair keeps syncing).
  - **Rollout:** the wire is unchanged (the initiator has always sent the header when the key
    is set). Set the key on both nodes; a node with the key refuses a peer without it, so a
    pair with mismatched keys stops syncing until both agree. A pair where neither node has a
    key is unaffected.

## Testing

259 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
generation, LWW decision table, watermark/pruning, digest computation, schema-version
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, a peer offline
past `TombstoneRetention` not resurrecting a newer delete, mutual
snapshot over a capacity-1 transport, single-table baseline streaming only that table,
mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
Family conventions apply (`TreatWarningsAsErrors`).

Build / test from this directory:

```bash
dotnet build ZB.MOM.WW.LocalDb.slnx
dotnet test  ZB.MOM.WW.LocalDb.slnx
```

See the design doc: [`docs/plans/2026-07-17-localdb-design.md`](../docs/plans/2026-07-17-localdb-design.md).