From ccfcc7c3d27b029800b141d6648a2bcc125c1947 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 14:07:58 -0500 Subject: [PATCH] docs(index): LocalDb 0.3.7 published; ScadaBridge + OtOpcUa pin it (#53) Published ZB.MOM.WW.LocalDb, .Contracts and .Replication 0.3.7 to the Gitea feed (carrying the 0.3.5/0.3.6 changes); SHA256 recorded. Pins: ScadaBridge main 12fa7aff (ScadaBridge#130), OtOpcUa master e3031654 (lmxopcua#889). Not deployed. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- CLAUDE.md | 2 +- ZB.MOM.WW.LocalDb/CLAUDE.md | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d5fb7bb..a86983d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -167,7 +167,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | | Secrets (encrypted store + `${secret:}` resolution) | **2026-11-09: source at `0.6.6`, UNPUBLISHED** — Sqlite 10.0.13, concurrency-safe SQLite init (#25), order-independent SqlServer/Akka registration + contentRootPath (#30/#31), `${secret:}` survives reload (#32), Akka actor starts with the host (#33), CLI value via stdin/prompt (#35), 13 P3 review fixes (#45). Built; family version **`0.6.3`** (published 2026-09-28; apps still pin `0.6.2`). Pins: **all four apps `0.6.2`** (verified against each repo's checked-in pins AND its origin default branch 2026-09-03 — the earlier "OtOpcUa unpushed / mxgw 0.6.0 / ScadaBridge 0.5.1" notes were stale). **Target `0.6.1`+, never `0.6.0`, and use the 4-arg `AddZbSecrets(..., contentRootPath)` overload at every pre-host `${secret:}` expander call site — the pin alone does not close the hole.** `Secrets:SqlitePath` must be absolute and outside the content root (fail-closed, no opt-out; the 2026-08-09 mxgw outage). Production topologies decided (scadaproj#3/#4 closed): OtOpcUa = Akka pair-local (`Replicator.AkkaDotNet`; it never used the gRPC hub); **ScadaBridge retired the gRPC secrets hub for itself (2026-09-28, ScadaBridge `main` @ `23a139a4`, follow-ups `2650f794`) — sites now receive secrets via Deploy Artifacts instead of the hub, central still on ONE shared SQL store (`Secrets:Store:Provider=SqlServer`, `0.5.1` minimum)**; both default-off elsewhere. **`0.6.3` (hub sweep outcome logging + public `ISecretsHubSyncStatus`) is PUBLISHED to the feed (2026-09-28) but pinned by no app** — ScadaBridge, its original motivation, retired the hub for itself; no hub consumer remains (OtOpcUa is Akka pair-local), so nothing can usefully pin it — **`ZB.MOM.WW.Secrets.Replicator.Grpc` is DEPRECATED (scadaproj#8, 2026-11-09):** out of the Secrets solution/pack, `IsPackable=false`, entry points `[Obsolete]`; `0.6.3` is its last published version (feed unlisting left to the owner). Full narrative (`0.1.x`→`0.6.3`, incidents, live gates, container caveat, `NU1102` note): [`ZB.MOM.WW.Secrets/CLAUDE.md`](ZB.MOM.WW.Secrets/CLAUDE.md). | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | -| LocalDb (embedded cache + 2-node sync) | **2026-11-09: source at `0.3.7`, UNPUBLISHED** — Sqlite 10.0.13; sync endpoint enforces `ApiKey` (#36; set the same key on both nodes); FK-ordered snapshots, PK upserts (no INSERT OR REPLACE), drift check on snapshots (#37–#39); `AUTOINCREMENT` tables refused at `RegisterReplicated`. **#16 fixed in `0.3.7`:** a tombstone is pruned only once past `TombstoneRetention` AND held by the peer (retention restarts at delivery), so a peer offline past retention no longer resurrects deleted rows; optional `TombstoneHardCap` (default off) bounds ledger growth during an outage. **Adopted** by ScadaBridge (Phases 1+2 merged; pins **`0.3.4`**) and OtOpcUa (Phases 1+2 merged; pins **`0.3.4`** since 2026-09-03, `b8b4c18f`) — no version skew remains (every release is wire-compatible, so one would be supported). Replication is **default-OFF**; enabled on the docker rigs' site-a pairs and on the **QA pair vq03/vq04** (ScadaBridge site pair + OtOpcUa pair — OtOpcUa's sync only actually CONNECTED after the 2026-09-10 `b8b4c18f` upgrade; the old build answered the sync POST with the login 302); DEV vd03 is single-node. Operational constraints: a pair stops/starts together (no rolling upgrade), a node offline > `TombstoneRetention` (7 d) can resurrect deleted rows. Full release narrative (`0.1.1`→`0.3.4`) + gates: [`ZB.MOM.WW.LocalDb/CLAUDE.md`](ZB.MOM.WW.LocalDb/CLAUDE.md). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + the two adoption designs ([ScadaBridge](docs/plans/2026-07-19-scadabridge-localdb-design.md), [OtOpcUa](docs/plans/2026-07-20-otopcua-localdb-design.md)) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | +| LocalDb (embedded cache + 2-node sync) | **`0.3.7` PUBLISHED 2026-11-12 (scadaproj#53) and pinned by both consumers** (ScadaBridge `12fa7aff`, OtOpcUa `e3031654`; not deployed) — Sqlite 10.0.13; sync endpoint enforces `ApiKey` (#36; set the same key on both nodes); FK-ordered snapshots, PK upserts (no INSERT OR REPLACE), drift check on snapshots (#37–#39); `AUTOINCREMENT` tables refused at `RegisterReplicated`. **#16 fixed in `0.3.7`:** a tombstone is pruned only once past `TombstoneRetention` AND held by the peer (retention restarts at delivery), so a peer offline past retention no longer resurrects deleted rows; optional `TombstoneHardCap` (default off) bounds ledger growth during an outage. **Adopted** by ScadaBridge (Phases 1+2 merged; pins **`0.3.7`**) and OtOpcUa (Phases 1+2 merged; pins **`0.3.7`**; deployed builds still `0.3.4`) — no version skew remains (every release is wire-compatible, so one would be supported). Replication is **default-OFF**; enabled on the docker rigs' site-a pairs and on the **QA pair vq03/vq04** (ScadaBridge site pair + OtOpcUa pair — OtOpcUa's sync only actually CONNECTED after the 2026-09-10 `b8b4c18f` upgrade; the old build answered the sync POST with the login 302); DEV vd03 is single-node. Operational constraints: a pair stops/starts together (no rolling upgrade), the sync `ApiKey` must match on both nodes, and from `0.3.7` the tombstone ledger grows while a peer is down (no more resurrection past `TombstoneRetention`). Full release narrative (`0.1.1`→`0.3.7`) + gates: [`ZB.MOM.WW.LocalDb/CLAUDE.md`](ZB.MOM.WW.LocalDb/CLAUDE.md). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + the two adoption designs ([ScadaBridge](docs/plans/2026-07-19-scadabridge-localdb-design.md), [OtOpcUa](docs/plans/2026-07-20-otopcua-localdb-design.md)) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | Each hosted library directory carries its own `CLAUDE.md` with the full version narrative, consumer matrix and adoption record — it loads automatically when you work under that directory. Health, Telemetry and Configuration diff --git a/ZB.MOM.WW.LocalDb/CLAUDE.md b/ZB.MOM.WW.LocalDb/CLAUDE.md index 93b9a62..78e9e75 100644 --- a/ZB.MOM.WW.LocalDb/CLAUDE.md +++ b/ZB.MOM.WW.LocalDb/CLAUDE.md @@ -4,9 +4,11 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA **Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #16 (resurrection past `TombstoneRetention`) was fixed in `0.3.7`. The history below is a record, not a status tracker; file new follow-ups as issues. -**`0.3.5` (unpublished, scadaproj#26):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change. +**PUBLISHED `0.3.7` (2026-11-12, scadaproj#53)** — all three packages on the Gitea feed (built from scadaproj `30a12dd`, Release, 259/259), carrying the 0.3.5 + 0.3.6 changes below. SHA256: `ZB.MOM.WW.LocalDb.0.3.7.nupkg` `5ac1db336e9b71700cd253bfe986acc13d24d0286fcc38bb277317845c892321`, `.Contracts` `5e7a6fce04f415720f9a6aac7e21ca43d1e842f6a74282120ae8aae2a62eb824`, `.Replication` `f579f1218ccc279836841c06e2989e8d0d67009866548eeea6e162d4eb45b07d` (downloaded copies match). 0.3.5 and 0.3.6 were never published on their own. **Pins:** ScadaBridge `0.3.7` (`main` `12fa7aff`, ScadaBridge#130), OtOpcUa `0.3.7` (`master` `e3031654`, lmxopcua#889); not deployed (DEV vd03 / QA vq03+vq04 still run 0.3.4 builds). Deploy notes for the QA pairs: stop/start each pair together, `LocalDb:Replication:ApiKey` byte-identical on both nodes (now checked by the library too), and the tombstone ledger grows while a peer is down. -**`0.3.6` (unpublished, scadaproj#36–#39, #46, #50):** code-review fixes, all wire-compatible (no proto, `lib_schema_version` or on-disk schema change). +**`0.3.5` (scadaproj#26, published as part of 0.3.7):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change. + +**`0.3.6` (scadaproj#36–#39, #46, #50, published as part of 0.3.7):** code-review fixes, all wire-compatible (no proto, `lib_schema_version` or on-disk schema change). - **#36 LocalDb-001:** the passive `LocalDbSyncService` now enforces `LocalDb:Replication:ApiKey` itself (Bearer, constant-time, checked before the handshake and before newest-wins takeover; `Unauthenticated` otherwise). Blank key = unchanged behaviour plus a startup warning. ScadaBridge's and OtOpcUa's own `LocalDbSyncAuthInterceptor`s use the same key and header, so they keep working; they are now redundant except for their fail-closed-when-blank rule. - **#37 LocalDb-002:** snapshots deliver FK-linked rows in any order. `SnapshotStreamer` sends referenced tables first (Kahn order, name tie-break); `SnapshotApplier` defers rows that fail only SQLITE_CONSTRAINT_FOREIGNKEY (`LwwApplier.ApplyBatchAsync(..., deferForeignKeyViolations: true)`), retries them as rows land, and dead-letters true orphans at `SnapshotComplete`. Before, a child that arrived first was dead-lettered and acked over permanently. - **#38 LocalDb-003:** the applier upserts with `INSERT … ON CONFLICT() DO UPDATE` (`DO NOTHING` when every column is PK) instead of `INSERT OR REPLACE`. A secondary-`UNIQUE` clash is now a dead-letter instead of a silent delete of another row; as a side effect a remote update of a parent with `ON DELETE RESTRICT` children applies instead of being dead-lettered. @@ -16,7 +18,7 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA - **#50:** snapshot-path dead-letters (poison, drift, FK orphan, cap spill) now count in `localdb.sync.dead_lettered` (`SnapshotApplier` takes `LocalDbMetrics`, wired by `SyncSessionFactory`). New option **`LocalDb:Replication:MaxSnapshotDeferredRows`** (default 100,000, validated > 0) caps snapshot rows held for a missing FK parent; past it the oldest are re-applied once and dead-lettered if still orphaned. **Open note (cross-repo, #50):** once ScadaBridge and OtOpcUa pin 0.3.6, their own `LocalDbSyncAuthInterceptor`s duplicate the library check — they differ only in refusing every stream when no key is set. Each app decides whether to keep that stricter rule or delete its interceptor; nothing in this library changes for either choice. 249/249 tests. -**`0.3.7` (unpublished, scadaproj#16 — owner chose option A):** a tombstone is pruned only when it is past `TombstoneRetention` **and** the peer holds it. `OplogStore.PruneTombstonesAsync` holds every tombstone while database-wide (`__localdb_peer_state.needs_snapshot`) or per-table (`__localdb_snapshot_state`) snapshot debt is outstanding, and a LOCAL tombstone while its delete is still unacked (`hlc >=` the first unacked oplog row's HLC — ledger and oplog share the HLC, local HLCs rise with seq). Delivery restamps `tombstone_utc` (in `RecordPeerAckAsync` for tombstones in the newly acked range; in `ClearSnapshotDebtAsync` for the snapshot's scope), so retention runs from delivery and the returning peer's stale queued writes get a full retention to drain. Remote-origin tombstones age on retention alone. New optional **`LocalDb:Replication:TombstoneHardCap`** (default `null` = off, validated ≥ `TombstoneRetention`) bounds an undelivered tombstone's life — past it the old risk returns. **The snapshot path (issue's second note) is covered by the debt hold:** the cap prune sets `needs_snapshot` in the same transaction that drops undelivered oplog rows, so the ledger tombstone the snapshot is built from survives until that snapshot is acknowledged. Cost: ledger growth for the length of an outage. Not covered: a peer restored from an old backup of its own file. No proto, `lib_schema_version` or on-disk schema change. Regression test `TombstoneResurrectionTests.PeerOfflinePastRetention_DeleteStillWinsOnBothNodes` (fails on 0.3.6 with the row resurrected, `A=1`) + positive control without the prune + pruned-after-delivery control; 5 `OplogStoreTests` with a controlled clock; every guard clause mutation-killed. 259/259. +**`0.3.7` (published 2026-11-12, scadaproj#16 — owner chose option A):** a tombstone is pruned only when it is past `TombstoneRetention` **and** the peer holds it. `OplogStore.PruneTombstonesAsync` holds every tombstone while database-wide (`__localdb_peer_state.needs_snapshot`) or per-table (`__localdb_snapshot_state`) snapshot debt is outstanding, and a LOCAL tombstone while its delete is still unacked (`hlc >=` the first unacked oplog row's HLC — ledger and oplog share the HLC, local HLCs rise with seq). Delivery restamps `tombstone_utc` (in `RecordPeerAckAsync` for tombstones in the newly acked range; in `ClearSnapshotDebtAsync` for the snapshot's scope), so retention runs from delivery and the returning peer's stale queued writes get a full retention to drain. Remote-origin tombstones age on retention alone. New optional **`LocalDb:Replication:TombstoneHardCap`** (default `null` = off, validated ≥ `TombstoneRetention`) bounds an undelivered tombstone's life — past it the old risk returns. **The snapshot path (issue's second note) is covered by the debt hold:** the cap prune sets `needs_snapshot` in the same transaction that drops undelivered oplog rows, so the ledger tombstone the snapshot is built from survives until that snapshot is acknowledged. Cost: ledger growth for the length of an outage. Not covered: a peer restored from an old backup of its own file. No proto, `lib_schema_version` or on-disk schema change. Regression test `TombstoneResurrectionTests.PeerOfflinePastRetention_DeleteStillWinsOnBothNodes` (fails on 0.3.6 with the row resurrected, `A=1`) + positive control without the prune + pruned-after-delivery control; 5 `OplogStoreTests` with a controlled clock; every guard clause mutation-killed. 259/259. ## Index history (migrated from the scadaproj root CLAUDE.md)