fix(localdb): count snapshot dead-letters and cap held-back snapshot rows
- SnapshotApplier now takes LocalDbMetrics (wired by SyncSessionFactory) and adds every row it dead-letters - poison, drift reject, FK orphan, retry failure, cap spill - to localdb.sync.dead_lettered. Before, only the delta path recorded the metric. - New ReplicationOptions.MaxSnapshotDeferredRows (default 100,000, validated > 0) bounds the rows held for a missing FK parent; past it the oldest get one more apply with deferral off and are dead-lettered if still orphaned. An older name-ordered sender could otherwise defer a whole child table. Unit + 2-node (SyncSessionFactory) tests fail pre-fix. No wire change. (scadaproj#50) Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -243,6 +243,7 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
|
||||
| `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
|
||||
| `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
|
||||
| `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. |
|
||||
| `MaxSnapshotDeferredRows` | `100_000` | Ceiling on snapshot rows held in memory while their FK parent has not arrived (`0.3.6`). Past it the oldest are dead-lettered (and counted). |
|
||||
| `MaxHlcDriftAhead` | `5 min` | Max tolerated HLC drift where the peer's clock runs ahead of ours. |
|
||||
| `FailClosedOnDrift` | `false` | When true, an over-drift entry is dead-lettered instead of applied. |
|
||||
|
||||
@@ -327,7 +328,10 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
|
||||
sorts later as `pk_json` text) was dead-lettered for good. **Mixed pairs:** a pre-`0.3.6`
|
||||
receiver still loses self-referencing rows that arrive child-first, and a pre-`0.3.6`
|
||||
sender's name order relies on the `0.3.6` receiver's deferral — so upgrade both nodes before
|
||||
adding FK-linked tables. Deferred rows are held in memory until the snapshot completes.
|
||||
adding FK-linked tables. Deferred rows are held in memory until the snapshot completes, up
|
||||
to `MaxSnapshotDeferredRows` (default 100,000); past that the oldest are dead-lettered rather
|
||||
than buffered without bound — an older, name-ordered sender whose child table sorts first can
|
||||
otherwise defer a whole table.
|
||||
- **Secondary `UNIQUE` keys must be unique across both nodes.** Remote rows are applied with
|
||||
`INSERT … ON CONFLICT(<pk>) DO UPDATE` (`0.3.6`), so only the row with the same primary key
|
||||
is touched. If the same natural key is created under different primary keys on the two nodes
|
||||
@@ -370,7 +374,7 @@ null-guarded — replication runs identically whether or not metrics are collect
|
||||
| `localdb.sync.lag.seconds` | ObservableGauge | Seconds since the last successful sync (no measurement when never synced). |
|
||||
| `localdb.sync.applied` | Counter | Rows applied from inbound **delta** batches. |
|
||||
| `localdb.sync.conflicts_discarded` | Counter | Inbound rows that lost LWW. |
|
||||
| `localdb.sync.dead_lettered` | Counter | Poison / over-drift rows routed to the dead-letter table. |
|
||||
| `localdb.sync.dead_lettered` | Counter | Rows routed to the dead-letter table — poison / over-drift rows from deltas **and** (since `0.3.6`) snapshots, including FK orphans and rows spilled past `MaxSnapshotDeferredRows`. |
|
||||
| `localdb.sync.reconnects` | Counter | Reconnect attempts (initiator). |
|
||||
| `localdb.sync.snapshots` | Counter | Snapshots, tagged `direction=sent`/`received` (one increment per snapshot). |
|
||||
|
||||
@@ -409,8 +413,10 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
|
||||
local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
|
||||
keeps syncing.
|
||||
- The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
|
||||
entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
|
||||
rejects.
|
||||
entries), over-drift rejects (when `FailClosedOnDrift`), snapshot rows whose FK parent never
|
||||
arrived, and secondary-`UNIQUE` clashes. Every row written there, on the delta and the
|
||||
snapshot path, is counted in `localdb.sync.dead_lettered` (snapshot-path rows were not
|
||||
counted before `0.3.6`). Inspect it to diagnose rejects.
|
||||
- **Peer auth** (0.3.6+): with `ApiKey` set, the initiator sends `Authorization: Bearer <key>`
|
||||
and the passive `LocalDbSyncService` verifies it (constant-time) **before** the handshake and
|
||||
before it displaces the current stream — a caller without the key gets `Unauthenticated`, sees
|
||||
|
||||
Reference in New Issue
Block a user