fix(localdb): registration follows replication config — baseline on enable, deregister self-heal on disable (pair-symmetric)
LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.
Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.
configured => RegisterReplicated(table, baselineExistingRows: true), per
table, in the unchanged load-bearing DDL->register order
unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
drops the capture triggers an earlier build left behind and
prunes that table's oplog/row-version rows. One Information
line, only when it actually cleaned something.
PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").
Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.
Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
This commit is contained in:
@@ -418,6 +418,18 @@ replication stays inert until the sync port + peer are set.
|
||||
| `LocalDb:Replication:MaxBatchSize` | `500` (library) | **Secondary** row-count cap on a batch — *and* the DB read-page size. `16` on the replicating pair. Since LocalDb 0.2.0 the wire message is bounded by `MaxBatchBytes`, not by this, so it is no longer the hand-tuned byte proxy the 4 MB cap once forced. It still matters because `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer` materialise a whole read page **before** the byte-budget split runs — `16` bounds that transient allocation at ≈2.7 MB for ≈171 KB artifact chunk rows. |
|
||||
| `LocalDb:Replication:MaxBatchBytes` | `2 MB` (library) | **Primary** cap on one outbound delta/snapshot message: the summed serialized size of its entries, sized under gRPC's 4 MB receive limit. Left unset across the fleet. A single row larger than the budget is sent **alone** rather than stalling the stream. Added in LocalDb 0.2.0; needs no wire change, so a node on this version interoperates with any earlier peer in both directions. |
|
||||
|
||||
**These two keys also decide whether the node captures at all.** `LocalDbSetup.ReplicationIsConfigured` is
|
||||
`PeerAddress` **OR** `ApiKey` non-blank — the OR because only the *dialling* half sets `PeerAddress`, so
|
||||
keying on it alone would strip change capture from every passive node and leave the pair converging in one
|
||||
direction only. `SyncListenPort` is deliberately **not** part of it (a port with no `ApiKey` is fail-closed
|
||||
and syncs with nobody). A configured node registers all four tables with `baselineExistingRows: true`, so
|
||||
turning replication on converges the rows **already in the file** and not merely subsequent writes; an
|
||||
unconfigured node **deregisters** them at boot, dropping capture triggers an earlier build installed. ⚠️ The
|
||||
sync handshake compares registered-table digests **fail-closed**, so flipping either key is a
|
||||
both-nodes-together change: set it on both and restart both. On LocalDb 0.2.1 baselining flags a
|
||||
**per-database** snapshot resync (every registered table, in full, both directions), so trim a long-running
|
||||
`alarm_sf_events` buffer before enabling replication late.
|
||||
|
||||
Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
|
||||
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
|
||||
for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
|
||||
|
||||
@@ -82,8 +82,28 @@ Replication has two knobs, both under `LocalDb`:
|
||||
> wire protocol is unchanged (0.1.x peers still sync), and the on-disk bookkeeping schema v1 → v2
|
||||
> is upgraded **in place on open**, so an existing node's database file needs no operator action.
|
||||
> Two behaviours changed that an operator should know: batching is now byte-budgeted (the two rows
|
||||
> above), and `DeregisterReplicated` exists — this repo never calls it, so the deregistration
|
||||
> symmetry rule and the HLC-anchor fix 0.2.1 shipped for it do not apply to an OtOpcUa node.
|
||||
> above), and **both of 0.2.x's lifecycle APIs are now used** (2026-08-18) — see the box below.
|
||||
|
||||
> **⚠️ Registration follows this config, and flipping it is a BOTH-NODES change.** Since
|
||||
> 2026-08-18 `LocalDbSetup.OnReady` installs the change-capture triggers only when the node is
|
||||
> configured to replicate — the predicate is **`PeerAddress` OR `ApiKey` non-blank** (the OR because
|
||||
> the passive node has no `PeerAddress`; `SyncListenPort` is deliberately not part of it, since a
|
||||
> port with no key is fail-closed and syncs with nobody). Consequences at enablement time:
|
||||
>
|
||||
> - A configured node registers with **`baselineExistingRows: true`**, so the rows **already in the
|
||||
> file** are seeded into the row-version ledger at the LWW floor (HLC `0`) and actually travel.
|
||||
> Before this, enabling replication on a node that had been running converged on *subsequent
|
||||
> writes only*. ⚠️ On 0.2.1 the snapshot flag is **per-database**: baselining streams *every*
|
||||
> registered table in full, in both directions at once. **Trim a long-standing `alarm_sf_events`
|
||||
> backlog before enabling replication late**, and expect one slow first sync.
|
||||
> - An unconfigured node **deregisters** at boot — the capture triggers a pre-2026-08-18 build left
|
||||
> behind are dropped and that table's oplog/row-version rows pruned. It logs one Information line
|
||||
> naming the table count, and stays silent when there was nothing to clean.
|
||||
> - The sync handshake compares **registered-table digests, fail-closed**, so a node that
|
||||
> deregisters a table its peer still replicates stops syncing with a schema mismatch. **Set the
|
||||
> keys on both nodes and restart both** — never one. (0.2.1's HLC-anchor fix now matters here:
|
||||
> the clock is flushed inside the deregistration transaction, so a crash right after a
|
||||
> deregister boot cannot bring the clock below stamps the peer already holds.)
|
||||
|
||||
Example (site-a-1 initiator, site-a-2 passive):
|
||||
|
||||
|
||||
Reference in New Issue
Block a user