diff --git a/.claude/rules/cluster-topology.md b/.claude/rules/cluster-topology.md index f38a6f0c7..9d7ce0b72 100644 --- a/.claude/rules/cluster-topology.md +++ b/.claude/rules/cluster-topology.md @@ -309,8 +309,12 @@ same event accepted on both nodes — which happens in every boot window, since docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue. -**The library pin is `0.3.4`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and -`.Replication`), bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15). +**The library pin is `0.3.7`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and +`.Replication`; bumped from `0.3.4` on 2026-11-12, scadaproj#53 — the library's sync service now +enforces `LocalDb:Replication:ApiKey` itself (0.3.6, scadaproj#36; `LocalDbSyncAuthInterceptor` kept +for its refuse-when-blank rule), `RegisterReplicated` refuses `AUTOINCREMENT` keys (none here), and +a tombstone is held until the peer has it (0.3.7, scadaproj#16 — no resurrection after a long outage, +ledger grows while the peer is down; optional `TombstoneHardCap`)). Earlier: bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15). The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3 upgrades in place on open, so no rig or volume action was needed. `RegisterReplicated(string)` is unchanged and everything 0.2.x/0.3.x added is additive, **except** that `RegisterReplicated` (0.3.3) @@ -358,7 +362,7 @@ things are worth knowing: used to replicate only writes made after the restart. It seeds the ledger at the LWW floor (HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote write of the same key and wins only where the peer has no version at all. It is idempotent, so - it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.4`) the snapshot + it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.7`) the snapshot flag is **per-table**, not per-database — baselining one table no longer streams every registered table in both directions. (Prior to `0.3.0`, on the `0.2.1` pin this repo carried until 2026-09-03, the flag was per-database and baselining anything streamed *every* diff --git a/Directory.Packages.props b/Directory.Packages.props index 3f91892ed..11ebc01be 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -224,8 +224,8 @@ package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the CVE-2025-6965-fixed native binary without needing the surgical pin below. --> - - + + diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 989362637..d22fe517d 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -729,7 +729,9 @@ services: # DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It # MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch # silently stops the pair converging. NEVER reuse this key outside this local dev rig. - LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key" + # Both site-a nodes read the SAME variable, so an override can never split the pair; since + # LocalDb 0.3.6 the library's sync service checks this key too (scadaproj#36). + LocalDb__Replication__ApiKey: "${OTOPCUA_SITE_A_LOCALDB_SYNC_KEY:-dev-site-a-localdb-sync-key}" # Since LocalDb 0.2.0 the WIRE message is bounded by MaxBatchBytes (2 MB default, left # unset), not by this. MaxBatchSize is now the secondary row cap AND the DB read-page size, # and the read page is materialised in memory before the byte-budget split runs — so 16 is @@ -838,7 +840,7 @@ services: ServerHistorian__UseTls: "false" LocalDb__Path: "/app/data/otopcua-localdb.db" LocalDb__SyncListenPort: "9001" - LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key" + LocalDb__Replication__ApiKey: "${OTOPCUA_SITE_A_LOCALDB_SYNC_KEY:-dev-site-a-localdb-sync-key}" LocalDb__Replication__MaxBatchSize: "16" ports: - "4843:4840" diff --git a/docs/Configuration.md b/docs/Configuration.md index d80c4ca15..5c512b885 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -494,7 +494,8 @@ replication stays inert until the sync port + peer are set. | `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. | | `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). | | `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://:`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. | -| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. | +| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Enforced by the host interceptor and, since LocalDb 0.3.6, by the library's sync service too. Supply via `${secret:...}` / env — never a cleartext literal in production. | +| `LocalDb:Replication:TombstoneHardCap` | unset (off) | LocalDb 0.3.7. A tombstone is kept until it is past `TombstoneRetention` (7 days) **and** the peer holds it, so the ledger grows while a peer is down. This optional cap (≥ `TombstoneRetention`) bounds an undelivered tombstone's life; past it a returning node can resurrect deleted rows again. Leave unset unless disk is the constraint. | | `LocalDb:Replication:MaxBatchSize` | `500` (library) | **Secondary** row-count cap on a batch — *and* the DB read-page size. `16` on the replicating pair. Since LocalDb 0.2.0 the wire message is bounded by `MaxBatchBytes`, not by this, so it is no longer the hand-tuned byte proxy the 4 MB cap once forced. It still matters because `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer` materialise a whole read page **before** the byte-budget split runs — `16` bounds that transient allocation at ≈2.7 MB for ≈171 KB artifact chunk rows. | | `LocalDb:Replication:MaxBatchBytes` | `2 MB` (library) | **Primary** cap on one outbound delta/snapshot message: the summed serialized size of its entries, sized under gRPC's 4 MB receive limit. Left unset across the fleet. A single row larger than the budget is sent **alone** rather than stalling the stream. Added in LocalDb 0.2.0; needs no wire change, so a node on this version interoperates with any earlier peer in both directions. | @@ -510,7 +511,7 @@ turning replication on converges the rows **already in the file** and not merely unconfigured node **deregisters** them at boot, dropping capture triggers an earlier build installed. ⚠️ The sync handshake compares registered-table digests **fail-closed**, so flipping either key is a both-nodes-together change: set it on both and restart both. As of LocalDb 0.3.0 (this repo pins -0.3.4) baselining flags a **per-table** snapshot resync — enabling replication on one table no +0.3.7) baselining flags a **per-table** snapshot resync — enabling replication on one table no longer forces a full-database resync of every registered table in both directions. Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See diff --git a/docs/operations/2026-07-20-localdb-pair-replication.md b/docs/operations/2026-07-20-localdb-pair-replication.md index 5db75b758..c5b639fb2 100644 --- a/docs/operations/2026-07-20-localdb-pair-replication.md +++ b/docs/operations/2026-07-20-localdb-pair-replication.md @@ -77,9 +77,10 @@ Replication has two knobs, both under `LocalDb`: | `LocalDb:Replication:MaxBatchSize` | secondary row cap **and** the DB read-page size | `16` for the artifact cache. **Since LocalDb 0.2.0 this no longer bounds the wire message** — `MaxBatchBytes` does — so it is not the byte proxy the note here used to describe. Keep it at `16` anyway: `OplogStore`/`SnapshotStreamer` materialise a whole read page in memory *before* the byte-budget split runs, and 16 × ≈171 KB artifact chunk rows ≈ 2.7 MB bounds that transient allocation. Raising it no longer risks tripping the gRPC cap, only the allocation. | | `LocalDb:Replication:MaxBatchBytes` | primary cap on one outbound message (bytes) | Library default **2 MB**, sized under gRPC's 4 MB receive limit. Left unset on this rig. A single row over budget is sent alone rather than stalling the stream. | -> **Library version.** This repo pins **`ZB.MOM.WW.LocalDb` / `.Replication` 0.3.4** -> (`Directory.Packages.props`; bumped from `0.2.1` on 2026-09-03, itself bumped from `0.1.3` on -> 2026-08-15). Every move in that chain needed no config or code change here: the wire protocol is +> **Library version.** This repo pins **`ZB.MOM.WW.LocalDb` / `.Replication` 0.3.7** +> (`Directory.Packages.props`; bumped from `0.3.4` on 2026-11-12 under scadaproj#53, from `0.2.1` +> on 2026-09-03, and from `0.1.3` on 2026-08-15). 0.3.5–0.3.7 are wire-compatible too; what they +> change for an operator is the ApiKey rule and the tombstone rule below. Every move in that chain needed no config or code change here: the wire protocol is > unchanged end to end (0.1.x/0.2.x/0.3.x peers all still sync), and the on-disk bookkeeping schema > upgrades in place on open (now v3) with no operator action on an existing node's database file. > Behaviours worth knowing at the current pin: batching is byte-budgeted (the two rows above); @@ -133,8 +134,10 @@ site-a-2: LocalDb__SyncListenPort=9001 ### The ApiKey rule (fail-closed) -The replication library's passive endpoint verifies **no** authentication — the host -`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**: +Two gates check the same key. Since LocalDb 0.3.6 (scadaproj#36) the library's passive +`LocalDbSyncService` enforces `LocalDb:Replication:ApiKey` itself (Bearer, constant-time, +`Unauthenticated` on a mismatch). The host `LocalDbSyncAuthInterceptor` runs first and is kept for +its stricter rule — it is **fail-closed**: - **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never "no auth required". @@ -148,11 +151,16 @@ The replication library's passive endpoint verifies **no** authentication — th while its peer is down; the outage is not a data-loss event — the surviving node accumulates writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its peer, so avoid leaving a pair split for extended periods. -- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and - replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window. - If a node is offline **longer than the tombstone-retention window**, a delete that happened during - its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could - briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window. +- **Tombstones are held until the peer has them (LocalDb 0.3.7, scadaproj#16).** Retention prunes + to the newest 2 deployments and replicates the prune as tombstones. Through 0.3.6 a tombstone was + dropped after `LocalDb:Replication:TombstoneRetention` (7 days) whether or not the peer had it, so + a node offline longer than that could bring a pruned deployment back on rejoin. As of 0.3.7 a + tombstone is pruned only when it is past retention **and** delivered (retention restarts at + delivery), so a rejoin after any length of outage is safe. **The cost is ledger growth while the + peer is down:** the survivor keeps every undelivered tombstone (deployment prunes, drained + `alarm_sf_events` rows) until the partner returns. Optional `LocalDb:Replication:TombstoneHardCap` + (default unset = off, must be ≥ `TombstoneRetention`) bounds it, and past the cap the old risk + returns. A node restored from an old backup of its own file is still not covered. - **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume, a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the cached configuration back to it, with no new deploy and no central SQL. Two things are worth diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs index a8b524049..99ceafa23 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs @@ -9,10 +9,12 @@ using ZB.MOM.WW.LocalDb.Replication; namespace ZB.MOM.WW.OtOpcUa.Host.Configuration; /// -/// Gates the LocalDb passive sync endpoint. The replication library deliberately leaves inbound -/// authentication to the host — its LocalDbSyncService verifies nothing — so without this -/// interceptor anything that can reach a driver node's sync port could stream arbitrary rows -/// into that node's deployment-artifact cache. +/// Gates the LocalDb passive sync endpoint. Through LocalDb 0.3.5 the replication library left +/// inbound authentication to the host — its LocalDbSyncService verified nothing — so +/// without this interceptor anything that could reach a driver node's sync port could stream +/// arbitrary rows into that node's deployment-artifact cache. Since 0.3.6 (scadaproj#36) the +/// library checks the same key and header itself; this interceptor is kept for the one rule the +/// library does not apply — refusing every stream when no key is configured. /// /// ///