fix(localdb): count snapshot dead-letters and cap held-back snapshot rows

- SnapshotApplier now takes LocalDbMetrics (wired by SyncSessionFactory) and
  adds every row it dead-letters - poison, drift reject, FK orphan, retry
  failure, cap spill - to localdb.sync.dead_lettered. Before, only the delta
  path recorded the metric.
- New ReplicationOptions.MaxSnapshotDeferredRows (default 100,000, validated
  > 0) bounds the rows held for a missing FK parent; past it the oldest get
  one more apply with deferral off and are dead-lettered if still orphaned.
  An older name-ordered sender could otherwise defer a whole child table.

Unit + 2-node (SyncSessionFactory) tests fail pre-fix. No wire change.

(scadaproj#50)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:12:41 -05:00
parent fd46999946
commit b2adddbc8e
7 changed files with 376 additions and 7 deletions
+10 -4
View File
@@ -243,6 +243,7 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
| `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
| `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
| `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. |
| `MaxSnapshotDeferredRows` | `100_000` | Ceiling on snapshot rows held in memory while their FK parent has not arrived (`0.3.6`). Past it the oldest are dead-lettered (and counted). |
| `MaxHlcDriftAhead` | `5 min` | Max tolerated HLC drift where the peer's clock runs ahead of ours. |
| `FailClosedOnDrift` | `false` | When true, an over-drift entry is dead-lettered instead of applied. |
@@ -327,7 +328,10 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
sorts later as `pk_json` text) was dead-lettered for good. **Mixed pairs:** a pre-`0.3.6`
receiver still loses self-referencing rows that arrive child-first, and a pre-`0.3.6`
sender's name order relies on the `0.3.6` receiver's deferral — so upgrade both nodes before
adding FK-linked tables. Deferred rows are held in memory until the snapshot completes.
adding FK-linked tables. Deferred rows are held in memory until the snapshot completes, up
to `MaxSnapshotDeferredRows` (default 100,000); past that the oldest are dead-lettered rather
than buffered without bound — an older, name-ordered sender whose child table sorts first can
otherwise defer a whole table.
- **Secondary `UNIQUE` keys must be unique across both nodes.** Remote rows are applied with
`INSERT … ON CONFLICT(<pk>) DO UPDATE` (`0.3.6`), so only the row with the same primary key
is touched. If the same natural key is created under different primary keys on the two nodes
@@ -370,7 +374,7 @@ null-guarded — replication runs identically whether or not metrics are collect
| `localdb.sync.lag.seconds` | ObservableGauge | Seconds since the last successful sync (no measurement when never synced). |
| `localdb.sync.applied` | Counter | Rows applied from inbound **delta** batches. |
| `localdb.sync.conflicts_discarded` | Counter | Inbound rows that lost LWW. |
| `localdb.sync.dead_lettered` | Counter | Poison / over-drift rows routed to the dead-letter table. |
| `localdb.sync.dead_lettered` | Counter | Rows routed to the dead-letter table — poison / over-drift rows from deltas **and** (since `0.3.6`) snapshots, including FK orphans and rows spilled past `MaxSnapshotDeferredRows`. |
| `localdb.sync.reconnects` | Counter | Reconnect attempts (initiator). |
| `localdb.sync.snapshots` | Counter | Snapshots, tagged `direction=sent`/`received` (one increment per snapshot). |
@@ -409,8 +413,10 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
keeps syncing.
- The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
rejects.
entries), over-drift rejects (when `FailClosedOnDrift`), snapshot rows whose FK parent never
arrived, and secondary-`UNIQUE` clashes. Every row written there, on the delta and the
snapshot path, is counted in `localdb.sync.dead_lettered` (snapshot-path rows were not
counted before `0.3.6`). Inspect it to diagnose rejects.
- **Peer auth** (0.3.6+): with `ApiKey` set, the initiator sends `Authorization: Bearer <key>`
and the passive `LocalDbSyncService` verifies it (constant-time) **before** the handshake and
before it displaces the current stream — a caller without the key gets `Unauthenticated`, sees