diff --git a/CLAUDE.md b/CLAUDE.md index 6cc357ba..75b1380f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -578,6 +578,36 @@ same event accepted on both nodes — which happens in every boot window, since docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue. +**The library pin is `0.2.1`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and +`.Replication`), bumped from `0.1.3` on 2026-08-15. It was a **pure pin bump — no code change**: +`RegisterReplicated(string)` is unchanged, everything 0.2.x added is additive, the wire protocol is +untouched (0.1.x peers still sync), and the on-disk bookkeeping schema v1 → v2 upgrades in place on +open, so no rig or volume action was needed. Three things about it are worth knowing: + +- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching + **byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized + under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row + cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which + materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps + `16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the + message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" + was corrected with the bump; do not restore that rationale. +- **`DeregisterReplicated` exists and this repo deliberately does not call it.** `LocalDbSetup.OnReady` + registers all four tables **unconditionally**, on every driver node, replicating or not — a + replication-configured node and a default-OFF one (central, site-b) carry identical CDC triggers. + ScadaBridge went the other way (conditional registration + a boot-time deregister self-heal), so + do not read that pattern across as settled here: it would be a behaviour change, and + `LocalDbSetupTests` pins the registered-table set and the DDL→register ordering exactly. The + cost of the current shape is three unused triggers per table on an unreplicated node; the + benefit is that turning replication on is a config change that needs no re-registration pass. +- **Consequently, `baselineExistingRows` is not used either.** Rows written on a node *before* + replication is enabled are not in `__localdb_row_version`, so enabling replication on a + long-running default-OFF node (central, site-b) converges on subsequent **writes**, not on the + rows already in the file. Passing `RegisterReplicated(table, baselineExistingRows: true)` is the + library's answer, but it flags a snapshot resync and is only correct when gated on replication + actually being configured — which the unconditional registration above cannot express. Enable + replication on a fresh pair, or accept the gap and let the next deploy re-cache. + See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config cache), `docs/AlarmHistorian.md`, and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`. diff --git a/Directory.Packages.props b/Directory.Packages.props index 340158e1..60deb106 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -175,8 +175,8 @@ package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the CVE-2025-6965-fixed native binary without needing the surgical pin below. --> - - + + diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 91218454..3ab758c0 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -660,8 +660,10 @@ services: # MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch # silently stops the pair converging. NEVER reuse this key outside this local dev rig. LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key" - # Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so - # 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom. + # Since LocalDb 0.2.0 the WIRE message is bounded by MaxBatchBytes (2 MB default, left + # unset), not by this. MaxBatchSize is now the secondary row cap AND the DB read-page size, + # and the read page is materialised in memory before the byte-budget split runs — so 16 is + # kept to bound that transient allocation (16 × ≈171 KB artifact chunk rows ≈ 2.7 MB). LocalDb__Replication__MaxBatchSize: "16" ports: - "4842:4840" @@ -728,7 +730,8 @@ services: # site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001 # but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's # single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key - # is mandatory — the interceptor fail-closes on a mismatch). + # is mandatory — the interceptor fail-closes on a mismatch; MaxBatchSize is now a read-page + # bound rather than a byte proxy — see the a-1 comment). # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the # buffer is a live, replicated table rather than an empty one — the live gate needs a real # queue to watch converge, and needs to see that only the Primary drains it. diff --git a/docs/Configuration.md b/docs/Configuration.md index 57c9f5e5..86fa1aeb 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -364,7 +364,8 @@ replication stays inert until the sync port + peer are set. | `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). | | `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://:`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. | | `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. | -| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). | +| `LocalDb:Replication:MaxBatchSize` | `500` (library) | **Secondary** row-count cap on a batch — *and* the DB read-page size. `16` on the replicating pair. Since LocalDb 0.2.0 the wire message is bounded by `MaxBatchBytes`, not by this, so it is no longer the hand-tuned byte proxy the 4 MB cap once forced. It still matters because `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer` materialise a whole read page **before** the byte-budget split runs — `16` bounds that transient allocation at ≈2.7 MB for ≈171 KB artifact chunk rows. | +| `LocalDb:Replication:MaxBatchBytes` | `2 MB` (library) | **Primary** cap on one outbound delta/snapshot message: the summed serialized size of its entries, sized under gRPC's 4 MB receive limit. Left unset across the fleet. A single row larger than the budget is sent **alone** rather than stalling the stream. Added in LocalDb 0.2.0; needs no wire change, so a node on this version interoperates with any earlier peer in both directions. | Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See [`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md) diff --git a/docs/operations/2026-07-20-localdb-pair-replication.md b/docs/operations/2026-07-20-localdb-pair-replication.md index 8b7fc525..2afe8ca6 100644 --- a/docs/operations/2026-07-20-localdb-pair-replication.md +++ b/docs/operations/2026-07-20-localdb-pair-replication.md @@ -74,7 +74,16 @@ Replication has two knobs, both under `LocalDb`: | `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. | | `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. | | `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. | -| `LocalDb:Replication:MaxBatchSize` | rows per replication batch | `16` for the artifact cache. Batching is **row-count-only** against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap. | +| `LocalDb:Replication:MaxBatchSize` | secondary row cap **and** the DB read-page size | `16` for the artifact cache. **Since LocalDb 0.2.0 this no longer bounds the wire message** — `MaxBatchBytes` does — so it is not the byte proxy the note here used to describe. Keep it at `16` anyway: `OplogStore`/`SnapshotStreamer` materialise a whole read page in memory *before* the byte-budget split runs, and 16 × ≈171 KB artifact chunk rows ≈ 2.7 MB bounds that transient allocation. Raising it no longer risks tripping the gRPC cap, only the allocation. | +| `LocalDb:Replication:MaxBatchBytes` | primary cap on one outbound message (bytes) | Library default **2 MB**, sized under gRPC's 4 MB receive limit. Left unset on this rig. A single row over budget is sent alone rather than stalling the stream. | + +> **Library version.** This repo pins **`ZB.MOM.WW.LocalDb` / `.Replication` 0.2.1** +> (`Directory.Packages.props`). The 0.1.3 → 0.2.1 move needed no config or code change here: the +> wire protocol is unchanged (0.1.x peers still sync), and the on-disk bookkeeping schema v1 → v2 +> is upgraded **in place on open**, so an existing node's database file needs no operator action. +> Two behaviours changed that an operator should know: batching is now byte-budgeted (the two rows +> above), and `DeregisterReplicated` exists — this repo never calls it, so the deregistration +> symmetry rule and the HLC-anchor fix 0.2.1 shipped for it do not apply to an OtOpcUa node. Example (site-a-1 initiator, site-a-2 passive): diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json b/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json index 87b177bd..a1392094 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json @@ -64,7 +64,8 @@ "SyncListenPort": 0, "Replication": { "_comment": "Empty = passive/off. Set PeerAddress on exactly ONE node of the pair (the stream is bidirectional, so the other side still pushes its own deltas back). ApiKey must be BYTE-IDENTICAL on both nodes — the host interceptor fail-closes, so a typo silently stops convergence rather than degrading to unauthenticated. Supply it via ${secret:...} in production, never committed.", - "_MaxBatchSizeComment": "Row count, NOT bytes, against gRPC's 4 MB default message cap. Artifact chunk rows are ~171 KB base64, so keep this at 16 on the pair (16 x 171 KB ~= 2.7 MB worst case)." + "_MaxBatchSizeComment": "Secondary ROW-COUNT cap, and the DB read-page size, since LocalDb 0.2.0. The wire message is bounded by MaxBatchBytes (2 MB default, sized under gRPC's 4 MB cap), so this is no longer the hand-tuned byte proxy it once was. It still matters: OplogStore/SnapshotStreamer materialise a whole read page in memory BEFORE the byte-budget split runs, so keep it at 16 on the pair to bound that transient allocation (artifact chunk rows are ~171 KB base64, so 16 x 171 KB ~= 2.7 MB).", + "_MaxBatchBytesComment": "Primary cap on one outbound delta/snapshot message: summed serialized entry size, default 2 MB. Leave unset unless the gRPC message limit is retuned. A single row over budget is sent alone rather than stalling the stream." } }, "Deployment": {