SnapshotStreamer.SendAsync stamped as_of_seq = MAX(seq), which reads 0 once ack-pruning has emptied a converged sender's oplog. A receiver whose watermark genuinely is 0 (a rebuilt node — ScadaBridge's site-a-a since its 2026-08-15 rebuild) was healed by the snapshot but never lifted by it (the receiver's watermark update is correctly MAX-guarded), so the 0.1.2 empty-oplog horizon rule saw 0 < last_acked_seq as a gap on every subsequent handshake and re-streamed the whole database on every reconnect until the sender wrote again. Observed live: the same 127k-row full snapshot on two consecutive restarts of the pair. Fix: as_of_seq = max(MAX(seq), last_acked_seq). Every delta pruned below last_acked_seq is already folded into the rows the snapshot carries, so that is the floor it may claim; the receiver's MAX guard then lifts the watermark and the next handshake finds no gap. Sender-side only; wire-compatible; no schema change. Test WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot fails on the unfixed code exactly as the rig does (watermark 0, expected 3) and passes with the fix: the second episode exchanges only a delta, and a capturing logger on the snapshot streamer proves no snapshot was sent. Suite 216/216 on 8 of 9 runs; one unnamed single-test failure on the first run did not recur in eight further runs and could not be attributed.
29 KiB
ZB.MOM.WW.LocalDb
An embedded SQLite local cache with optional bidirectional async 2-node replication over gRPC, for the ZB.MOM.WW SCADA / OT family. It gives an application a fast, crash-safe local store that can optionally keep a second node in sync asynchronously — local writes never block on the peer, and a node that never enables replication is simply a fast local DB.
Change capture uses SQLite triggers writing a single HLC-stamped oplog inside the consumer's own write transaction (crash-consistent, non-bypassable). Convergence is last-writer-wins over a hybrid logical clock. All timestamps are UTC.
Packages (all 0.3.4)
| Package | Contents |
|---|---|
ZB.MOM.WW.LocalDb |
Embedded DB host: ILocalDb SQL surface (ExecuteAsync / QueryAsync / transactions), RegisterReplicated(table[, baselineExistingRows]) (PK + BLOB + foreign-key validation, trigger install, oplog schema), DeregisterReplicated(table) (trigger removal + oplog/ledger prune), HybridLogicalClock, AddZbLocalDb() DI. |
ZB.MOM.WW.LocalDb.Contracts |
localdb_sync.v1 proto + generated gRPC types (packable; a future peer could be non-.NET). |
ZB.MOM.WW.LocalDb.Replication |
gRPC sync engine: passive LocalDbSyncService, initiating SyncBackgroundService, MaintenanceBackgroundService, LWW apply, watermark acks + pruning, snapshot resync, ISyncStatus, LocalDbMetrics, AddZbLocalDbReplication() / MapZbLocalDbSync(). |
Status: published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).
0.3.4(2026-08-20) — a converged pair no longer re-streams a full snapshot on every reconnect once the sender's oplog has been ack-pruned to empty. Found on the ScadaBridge docker rig (its register row 106): a rebuilt node (watermark 0) was healed by a snapshot stampedas_of_seq = MAX(seq)— which is 0 on an empty oplog — so its MAX-guarded watermark stayed 0, and the healthy side's empty-oplog horizon rule (0.1.2: streamable fromlast_acked_seq + 1) saw a gap on every later handshake and re-sent the whole database (127k rows per restart on the rig) until it happened to write a new row. The as-of is nowmax(MAX(seq), last_acked_seq): every delta pruned belowlast_acked_seqis already folded into the rows the snapshot carries, so that is the floor it may claim. Sender-side only, wire-compatible, no schema change; pinned byWipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot.0.3.3(2026-08-20) — two latent apply-path hazards from the ScadaBridge adversarial architecture review (P4-7). Neither had a live trigger in any current consumer; both are the kind that surface as silent divergence, not as an error. Wire-compatible: no proto change, nolib_schema_versionbump, no on-disk schema change.
- A replicated table may no longer be the parent of a row-mutating
ON DELETEforeign key. The apply path writes remote rows withINSERT OR REPLACE, and REPLACE resolves a PK conflict by DELETING the existing row first. Connections runPRAGMA foreign_keys=ON, so that implicit delete fires the child'sON DELETE CASCADE/SET NULL/SET DEFAULT— while the capture guard is raised. The child rows change on the applying node ONLY, produce no oplog entry, and the pair diverges with nothing able to detect it (LWW compares only rows it was told about).RegisterReplicatednow inspects every foreign key in the database and refuses such a table with aLocalDbRegistrationExceptionnaming the offending child column and action.RESTRICT/NO ACTIONremain fine — they abort the statement instead of mutating rows behind capture's back, which the apply path already handles as per-entry poison. Checked at registration; SQLite cannot add a foreign key to an existing table, so a child table created later is caught at the next startup's registration rather than at the DDL.- An entry's row write and its ledger write are now atomic with each other. They are two statements, and SQLite aborts only the failing one — so a statement-level fault between them (the case the per-entry poison catch exists for) committed a half-applied entry: a tombstone whose table row was gone while
__localdb_row_versionstill called it live, which the snapshot streamer can only warn about and skip, so the peer never learned the row died; or an upsert whose new row carried the OLD version and therefore lost the next comparison it should have won. The pair is wrapped in a per-entrySAVEPOINTthat is rolled back before dead-lettering, so a poison entry leaves both writes or neither and the rest of the batch is untouched.0.3.2(2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, nolib_schema_versionbump, no on-disk schema change — the keepalive is transport-level and the rest is local policy.
- A live session now ends itself when a cap prune eats deltas it has not sent. The pump reads
seq > sentThruSeq, which resumes SILENTLY above a pruned hole; the peer acks the later seqs and the pair is durably divergent for the whole life of that session.needs_snapshot(set in the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares the oplog horizon (MIN(seq), one b-tree seek) against its sent watermark and throwsLocalDbOplogGapExceptionwhen deltas it still owes are gone. The horizon, not the flag, because the flag is also raised for reasons that do not invalidate the running session.- The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins. Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash or partition of the dialling node left the passive node's handler parked in
MoveNexton a half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted peer's reconnects withAlreadyExists, indefinitely on an idle pair. The initiator now dials withKeepAlivePingDelay/KeepAlivePingTimeout(new options, 60s/20s,Alwayspolicy so idle connections ping too), and a fresh authenticatedSynccancels the incumbent and waits for it to unwind before starting — the guard serializes sessions, it was never a fence, and an authenticated same-key dialler is by definition the peer. An incumbent that will not yield within the takeover timeout still getsAlreadyExists, because two live sessions over one database is worse than a refused dial.0.3.1(2026-08-18) — a faulting session flushes its control lane before cancelling its writer, so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to reconnect-loop against a generic disconnect.0.3.0(2026-08-15) — the snapshot-resync flag is per-table. On-disk bookkeeping schema goes to v3 (new__localdb_snapshot_state, upgraded in place on open); wire-compatible with0.1.x/0.2.xpeers, andlib_schema_versiondeliberately stays1because it is compared fail-closed for equality.
RegisterReplicated(table, baselineExistingRows: true)now flags a resync for THAT TABLE. Through0.2.xthe flag was one bit for the whole database, so baselining one table re-streamed every registered table in full — and on a pair enabling replication late, both nodes did that to each other at once. A snapshot now streams only the flagged tables.- The database-wide flag survives, unchanged, for the cases that genuinely need it: an oplog cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of every table is sound. What moved is only where a baseline records its debt.
- Additive wire extension, negotiated by capability, not by version.
Handshakegainssupports_partial_snapshot;SnapshotBegingains atablesscope where empty means "all tables" — which is precisely what a pre-0.3.0sender emits and what a pre-0.3.0receiver assumes. A peer that does not advertise the capability is served a full snapshot, because a scoped one would make it advancelast_applied_remote_seqpast deltas it never received.- A partial snapshot moves no watermark on either side. The receiver keeps
last_applied_remote_seq(it is one number for the whole peer oplog, and the untouched tables' deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The overlap re-delivers rows the snapshot already carried, which LWW absorbs.- Snapshot debt now clears on acknowledgement, not on enqueue (new
SnapshotAck, exchanged only between0.3.0peers). A crash part-way through a snapshot re-streams the same scope on reconnect instead of losing it. Against a pre-0.3.0peer the0.2.xclear-on-enqueue behaviour is kept — there is no ack to wait for.0.2.1(2026-08-14) — two adversarial-review findings against0.2.0. Wire-compatible with0.1.x/0.2.0peers; no proto change, nolib_schema_versionbump, no on-disk schema change.
DeregisterReplicatedno longer leaves the HLC without a crash anchor. Clock recovery on open isMAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc), andhlc_high_waterwas written only by graceful disposal — while deregistration deletes the table's oplog AND ledger rows. A crash after a deregister boot could therefore recover the clock below stamps the peer already holds, and LWW discards those as older. The current clock is now flushed inside the deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never regress the anchor).- The sync inbox is bounded (64 messages) instead of unbounded, so inbound memory is capped at roughly
64 x MaxBatchBytesrather than "whatever the peer sends" — the bad case being a mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now stops draining the gRPC stream, which closes the peer's flow-control window.- A snapshot this node owes is streamed alongside a running receive loop, not from inside the handshake. That is what makes bounding safe: previously neither side drained its inbox while sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was already safe (snapshot rows and deltas both merge through LWW, the watermark write is MAX-guarded) — only the ordering changed.
- Documented the cost of late baselining: the snapshot flag is per-database, so baselining one table re-streams every registered table, in both directions on a late-enabled pair. Trim high-volume tables (event logs) before turning replication on late.
0.2.0(2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review. All of it is wire-compatible with0.1.xpeers (no proto change, nolib_schema_versionbump); the on-disk bookkeeping schema goes to v2 and is upgraded in place on open.
DeregisterReplicated(table)— drops a table's capture triggers and prunes its oplog and row-version rows. A node that does not replicate should not be paying for capture on every write, and until now there was no way to stop once a file had been registered. Idempotent, and usable on triggers left behind by an earlier build.RegisterReplicated(table, baselineExistingRows: true)— closes the late-opt-in hole: rows that predate registration were invisible to the peer forever, because the snapshot streamer pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at an LWW-safe floor (HLC0, this node's id) and flags a snapshot resync so they actually travel. Default is still off — the documented CDC contract is unchanged unless you ask.- O(1) backlog depth —
MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)replaces aCOUNT(*)over the whole unacked backlog that ran once a minute AND on every metrics scrape (measured 58 VM steps vs 100,020 on a 50k backlog).- Byte-budget batching —
MaxBatchBytes(default 2 MB) bounds delta and snapshot messages by summed serialized size, withMaxBatchSizedemoted to a secondary row cap. Consumers no longer have to tune the row count against their widest row to stay under the gRPC message limit. Inbound delta batches and acks already queued are coalesced into one transaction / one ack.LwwApplierprepares once per statement instead of three SQL parses per applied row.- Dropped
__localdb_oplog_hlc— an index no query ever used, costing a b-tree insert inside every captured consumer write. Existing files lose it on the next open.0.1.3(2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN writes were silently dropped.last_applied_remote_seqis a watermark in the peer's seq space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's watermark, the rebuilt node started its pump above it, and its first N writes were never sent. Now the watermark is reset when the peer's node id changes, and a peer claiming to have applied more of our stream than we ever produced is treated as evidence that we were rebuilt (pump restarts from the beginning; LWW makes re-sending harmless). Found on the OtOpcUa docker-dev rig — it only became reachable once0.1.2made back-fill work.0.1.2(2026-07-21) — fixes snapshot resync for a wiped peer: a converged pair prunes its oplog to empty on ack, and the handshake read an empty oplog as "no gap possible", so a peer that came back with an empty database and a zero watermark was never snapshotted. The gap is now measured againstlast_acked_seqwhen the oplog is empty.0.1.1(2026-07-20) — creates the parent directory ofLocalDb:Path(a missing directory was a hard boot failure,SQLite Error 14).
Quickstart
Local-only (no replication)
builder.Services.AddZbLocalDb(builder.Configuration, db =>
{
// onReady: create your own tables, then opt them into (future) replication.
db.ExecuteAsync(
"CREATE TABLE IF NOT EXISTS widget (id INTEGER PRIMARY KEY, name TEXT, qty INTEGER)")
.GetAwaiter().GetResult();
db.RegisterReplicated("widget"); // PK required; installs capture triggers
});
// Consume ILocalDb anywhere:
await db.ExecuteAsync("INSERT INTO widget (id, name, qty) VALUES (@id, @n, @q)",
new { id = 1, n = "bolt", q = 42 });
var rows = await db.QueryAsync("SELECT name, qty FROM widget",
r => (Name: r.GetString(0), Qty: r.GetInt32(1)));
AddZbLocalDb is first-registration-wins: a second call is a no-op. RegisterReplicated
installs capture triggers even without the replication package present — you can register
tables now and turn on sync later.
Replicated (2 nodes)
Every node registers the passive side (so either can accept a stream) and the initiator
(idle unless a PeerAddress is set). Configure exactly one side with the peer's
address; the other stays passive.
// Both nodes:
builder.Services.AddZbLocalDb(builder.Configuration, db => { /* create + RegisterReplicated */ });
builder.Services.AddGrpc();
builder.Services.AddZbLocalDbReplication(builder.Configuration); // passive host + initiator + maintenance
var app = builder.Build();
app.MapZbLocalDbSync(); // passive gRPC endpoint — host must have AddGrpc()
app.Run();
// Node A — the INITIATOR: dials node B
"LocalDb": {
"Path": "nodeA.db",
"Replication": { "PeerAddress": "https://nodeB:5001", "ApiKey": "…" }
}
// Node B — PASSIVE: no PeerAddress, its SyncBackgroundService idles and only serves the endpoint
"LocalDb": { "Path": "nodeB.db", "Replication": { } }
Both directions flow on the single bidirectional stream the initiator opens; the passive
node still pushes its own deltas back. Only one side needs PeerAddress.
Configuration
LocalDb section (LocalDbOptions)
| Key | Default | Meaning |
|---|---|---|
Path |
"" (required) |
Filesystem path to the SQLite database file. |
BusyTimeoutMs |
5000 |
SQLite busy_timeout pragma, in milliseconds. |
Synchronous |
"NORMAL" |
SQLite synchronous pragma (OFF / NORMAL / FULL). |
LocalDb:Replication section (ReplicationOptions)
| Key | Default | Meaning |
|---|---|---|
PeerAddress |
"" |
gRPC address to dial. Empty ⇒ passive (initiator idles). |
ApiKey |
null |
Bearer token the initiator presents on its channel; null ⇒ no key header. |
FlushInterval |
250 ms |
How often the change pump flushes pending oplog deltas. |
MaxBatchBytes |
2 MB |
Primary batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
MaxBatchSize |
500 |
Secondary row-count cap per outbound batch (also the snapshot DB page size). |
MaxOplogRows |
1_000_000 |
Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
MaxOplogAge |
7 days |
Age ceiling; an unpruned entry older than this flags a snapshot resync. |
TombstoneRetention |
7 days |
How long deleted-row tombstones are kept before pruning. |
ReconnectBackoffMax |
60 s |
Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
MaintenanceInterval |
60 s |
How often maintenance enforces caps + prunes expired tombstones. |
MaxHlcDriftAhead |
5 min |
Max tolerated HLC drift where the peer's clock runs ahead of ours. |
FailClosedOnDrift |
false |
When true, an over-drift entry is dead-lettered instead of applied. |
All options are validated at startup (ZB.MOM.WW.Configuration validators).
How it works
- Capture — three AFTER triggers per registered table write each change into one
shared, HLC-stamped
__localdb_oplogplus a__localdb_row_versionledger, in the consumer's own transaction. A DB-scoped__localdb_applyingguard makes replicated applies skip re-capture. - Convergence — last-writer-wins: compare incoming HLC, then
node_idordinal as the tie-break; the strict-greater winner is applied, the loser discarded (but still acked). - Streaming — one long-lived bidirectional gRPC stream. The initiator dials; both sides push oplog batches above the peer's watermark and ack the applied watermark back, which drives pruning. Outbound uses a single writer fed by two lanes (an unbounded control lane for handshake/acks that always preempts a bounded data lane for delta/snapshot batches), so receiving never blocks on the ability to ack.
- Resync — when a peer's watermark falls below the pruned oplog horizon, it is sent a
full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
the snapshot's as-of seq. When instead a single table was baselined, only that table is
streamed and no watermark moves on either side — a partial snapshot speaks for nothing
outside its own tables, so the deltas it overlaps are still pumped and merged by LWW.
Each node decides from its OWN state whether it owes the peer a snapshot, and a peer
that does not advertise
supports_partial_snapshotis sent a full one. The snapshot is streamed after the handshake, alongside a running receive loop, so a node sending one still applies what the peer sends it — a mutual snapshot exchanges under flow control instead of being buffered whole in memory. Between0.3.0peers the sender's flags are cleared by the receiver'sSnapshotAck, so a crash mid-snapshot re-streams it. - Maintenance — a background service on both roles enforces the oplog row/age caps
and prunes expired tombstones every
MaintenanceInterval.
Semantics worth knowing
- PK-column updates are captured as delete-of-old-PK + insert-of-new — a tombstone on the old key plus a fresh insert (full delete+insert support).
- Upserts (
INSERT … ON CONFLICT DO UPDATE) are fully supported (the row-version trigger uses an explicitON CONFLICTclause so it is not rewritten by the outer statement's conflict algorithm). - Clock-drift guard — warn-once per session when a peer entry's HLC physical time is
more than
MaxHlcDriftAheadahead. WithFailClosedOnDrift, each over-drift entry is dead-lettered; otherwise it is applied (the HLC merge absorbs the skew). The session never dies from drift — no livelock on reconnect. - Writes never block on the peer; local commit is independent of replication.
- All timestamps are UTC (HLC physical component is UTC Unix milliseconds).
Limitations (honest list)
- Exactly two nodes. No mesh / N-node topologies.
- LWW only. No per-table pluggable conflict resolvers.
- BLOB columns are rejected at registration (a column declared
BLOBfailsRegisterReplicated, becausejson_objectcannot capture it). A typeless column can still hold a BLOB value at runtime, which then surfaces as a write-time error, not a registration error. - A replicated table may not be the parent of a row-mutating
ON DELETEforeign key (CASCADE/SET NULL/SET DEFAULTfrom any other table, or from itself). Applying a remote row usesINSERT OR REPLACE, whose implicit delete fires that action with change capture suppressed — the child rows would change on the applying node only and never replicate, diverging the pair undetectably.RegisterReplicatedrefuses such a table (0.3.3); useON DELETE NO ACTION/RESTRICTand delete children explicitly, or leave the parent unreplicated. The check runs at registration and reads the whole database's foreign keys, so a child table created after registration is caught at the next startup, not at the moment of theCREATE TABLE. - CDC semantics — rows that existed before
RegisterReplicatedare not captured and are not snapshotted; only changes after registration are replicated. Opt out withRegisterReplicated(table, baselineExistingRows: true), which seeds the row-version ledger for existing rows at the baseline floor (HLC0) and flags a snapshot resync for that table so they reach the peer. The floor is below every genuine stamp, so a baselined row loses every comparison against a real remote write and wins only where the peer has no version of that key at all. Two nodes that baseline the same key both hold HLC0and resolve by the node-id tie-break — convergent, but arbitrary as to which content survives. - A snapshot still ships whole tables. The flag is per-table as of
0.3.0, so baselining one table streams that table and no other — but that table travels in full, and two cases still stream everything: an oplog cap prune / impossible peer ack (delta continuity is lost for the oplog as a whole), and any peer older than0.3.0(which cannot be told a snapshot is partial). So purge or trim high-volume tables such as event logs before baselining them, and expect one slow first sync. Bounded either way (batch budget out, bounded inbox in). - REAL ±Infinity is carried in SQLite's JSON dialect on the wire (fine for the SQLite↔SQLite peers this library targets).
AddZbLocalDb/AddZbLocalDbReplicationare first-registration-wins (idempotent).- Inbound buffering is bounded, not zero. The duplex inbox holds up to 64 messages, so
peak inbound memory is roughly
64 x MaxBatchBytes(~128 MB at the 2 MB default, far less in practice — messages are only that large when the sender has that much backlog). LowerMaxBatchBytesto lower the ceiling. Beyond the bound the stream stops being drained and the peer is slowed by flow control.
Observability
Metrics publish under the single meter ZB.MOM.WW.LocalDb.Replication (all instruments
null-guarded — replication runs identically whether or not metrics are collected):
| Instrument | Kind | Notes |
|---|---|---|
localdb.oplog.depth |
ObservableGauge | Unacked oplog backlog (polled cheaply at scrape time). |
localdb.sync.lag.seconds |
ObservableGauge | Seconds since the last successful sync (no measurement when never synced). |
localdb.sync.applied |
Counter | Rows applied from inbound delta batches. |
localdb.sync.conflicts_discarded |
Counter | Inbound rows that lost LWW. |
localdb.sync.dead_lettered |
Counter | Poison / over-drift rows routed to the dead-letter table. |
localdb.sync.reconnects |
Counter | Reconnect attempts (initiator). |
localdb.sync.snapshots |
Counter | Snapshots, tagged direction=sent/received (one increment per snapshot). |
Counter split to note: rows merged from a snapshot are not counted under
localdb.sync.applied; a completed snapshot instead increments localdb.sync.snapshots
once (with its direction tag). Only delta-batch rows increment localdb.sync.applied.
ISyncStatus (registered singleton) surfaces live state for dashboards / health:
Connected (true while any session is active — a node can run both roles at once),
PeerNodeId, LastSyncUtc, OplogBacklog (nullable — null means unknown because no
provider is wired or the poll threw, so a DB failure never reads as "0 backlog"), and
ConnectionAttempts.
Ops notes
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
database-wide
needs_snapshot(disk safety over delta continuity); the peer then gets a full snapshot. Per-table debt lives separately in__localdb_snapshot_stateand is what baselining writes. - Upgrading a pair to
0.3.0needs no coordination: the two nodes negotiate scoped snapshots by capability, so a0.3.0node paired with a0.2.xone simply keeps sending full snapshots until both are upgraded. Rolling the file forward is one-way — a0.2.xbuild reopening a v3 file leaves__localdb_snapshot_statein place and ignores it, so any per-table debt recorded by0.3.0is invisible to it (roll forward again, or re-baseline). - Turning a table off —
DeregisterReplicated(table)drops its capture triggers and prunes its oplog + ledger rows. It must be applied symmetrically: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on neither. Re-enabling later should passbaselineExistingRows: true, or the rows that accumulated while capture was off stay invisible to the peer — but trim high-volume tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's durable anchor is flushed in the same transaction, so a crash after a deregister boot cannot bring the clock back below stamps peers already hold. - On-disk schema upgrades are applied in place when the file is opened, in one
transaction, and every step is safe to re-run.
schema_versionin__localdb_metais local-file state only — it is NOT the wirelib_schema_version, so a pair mid-upgrade keeps syncing. - The
__localdb_dead_lettertable holds poison rows (bad JSON, null-row non-tombstone entries) and over-drift rejects (whenFailClosedOnDrift). Inspect it to diagnose rejects. - Peer auth:
ApiKeyis sent by the initiator as a staticAuthorization: Bearer …header on its channel. The library's passive service does not itself verify the key — the passive host enforces whatever gRPC auth the host application configures (e.g.ZB.MOM.WW.Auth). Configure host-side authorization to actually gate inbound streams.
Testing
199 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
generation, LWW decision table, watermark/pruning, digest computation, schema-version
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
snapshot over a capacity-1 transport, single-table baseline streaming only that table,
mixed-version interop where a peer without supports_partial_snapshot gets a full one,
snapshot debt surviving until the peer's SnapshotAck), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
Family conventions apply (TreatWarningsAsErrors).
Build / test from this directory:
dotnet build ZB.MOM.WW.LocalDb.slnx
dotnet test ZB.MOM.WW.LocalDb.slnx
See the design doc: docs/plans/2026-07-17-localdb-design.md.