dohertj2/scadaproj
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(index): LocalDb 0.3.7 published; ScadaBridge + OtOpcUa pin it (#53)

Browse Source 
Published ZB.MOM.WW.LocalDb, .Contracts and .Replication 0.3.7 to the
Gitea feed (carrying the 0.3.5/0.3.6 changes); SHA256 recorded. Pins:
ScadaBridge main 12fa7aff (ScadaBridge#130), OtOpcUa master e3031654
(lmxopcua#889). Not deployed.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 14:07:58 -05:00
parent 470465dfbd
commit ccfcc7c3d2
2 changed files with 6 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+1 -1
View File
@@ -167,7 +167,7 @@ each project's **code-verified current state**, and the **gaps** between. See
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
| Secrets (encrypted store + `${secret:}` resolution) | **2026-11-09: source at `0.6.6`, UNPUBLISHED** — Sqlite 10.0.13, concurrency-safe SQLite init (#25), order-independent SqlServer/Akka registration + contentRootPath (#30/#31), `${secret:}` survives reload (#32), Akka actor starts with the host (#33), CLI value via stdin/prompt (#35), 13 P3 review fixes (#45). Built; family version **`0.6.3`** (published 2026-09-28; apps still pin `0.6.2`). Pins: **all four apps `0.6.2`** (verified against each repo's checked-in pins AND its origin default branch 2026-09-03 — the earlier "OtOpcUa unpushed / mxgw 0.6.0 / ScadaBridge 0.5.1" notes were stale). **Target `0.6.1`+, never `0.6.0`, and use the 4-arg `AddZbSecrets(..., contentRootPath)` overload at every pre-host `${secret:}` expander call site — the pin alone does not close the hole.** `Secrets:SqlitePath` must be absolute and outside the content root (fail-closed, no opt-out; the 2026-08-09 mxgw outage). Production topologies decided (scadaproj#3/#4 closed): OtOpcUa = Akka pair-local (`Replicator.AkkaDotNet`; it never used the gRPC hub); **ScadaBridge retired the gRPC secrets hub for itself (2026-09-28, ScadaBridge `main` @ `23a139a4`, follow-ups `2650f794`) — sites now receive secrets via Deploy Artifacts instead of the hub, central still on ONE shared SQL store (`Secrets:Store:Provider=SqlServer`, `0.5.1` minimum)**; both default-off elsewhere. **`0.6.3` (hub sweep outcome logging + public `ISecretsHubSyncStatus`) is PUBLISHED to the feed (2026-09-28) but pinned by no app** — ScadaBridge, its original motivation, retired the hub for itself; no hub consumer remains (OtOpcUa is Akka pair-local), so nothing can usefully pin it — **`ZB.MOM.WW.Secrets.Replicator.Grpc` is DEPRECATED (scadaproj#8, 2026-11-09):** out of the Secrets solution/pack, `IsPackable=false`, entry points `[Obsolete]`; `0.6.3` is its last published version (feed unlisting left to the owner). Full narrative (`0.1.x`→`0.6.3`, incidents, live gates, container caveat, `NU1102` note): [`ZB.MOM.WW.Secrets/CLAUDE.md`](ZB.MOM.WW.Secrets/CLAUDE.md). | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
| LocalDb (embedded cache + 2-node sync) | **2026-11-09: source at `0.3.7`, UNPUBLISHED** — Sqlite 10.0.13; sync endpoint enforces `ApiKey` (#36; set the same key on both nodes); FK-ordered snapshots, PK upserts (no INSERT OR REPLACE), drift check on snapshots (#37–#39); `AUTOINCREMENT` tables refused at `RegisterReplicated`. **#16 fixed in `0.3.7`:** a tombstone is pruned only once past `TombstoneRetention` AND held by the peer (retention restarts at delivery), so a peer offline past retention no longer resurrects deleted rows; optional `TombstoneHardCap` (default off) bounds ledger growth during an outage. **Adopted** by ScadaBridge (Phases 1+2 merged; pins **`0.3.4`**) and OtOpcUa (Phases 1+2 merged; pins **`0.3.4`** since 2026-09-03, `b8b4c18f`) — no version skew remains (every release is wire-compatible, so one would be supported). Replication is **default-OFF**; enabled on the docker rigs' site-a pairs and on the **QA pair vq03/vq04** (ScadaBridge site pair + OtOpcUa pair — OtOpcUa's sync only actually CONNECTED after the 2026-09-10 `b8b4c18f` upgrade; the old build answered the sync POST with the login 302); DEV vd03 is single-node. Operational constraints: a pair stops/starts together (no rolling upgrade), a node offline > `TombstoneRetention` (7 d) can resurrect deleted rows. Full release narrative (`0.1.1`→`0.3.4`) + gates: [`ZB.MOM.WW.LocalDb/CLAUDE.md`](ZB.MOM.WW.LocalDb/CLAUDE.md). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + the two adoption designs ([ScadaBridge](docs/plans/2026-07-19-scadabridge-localdb-design.md), [OtOpcUa](docs/plans/2026-07-20-otopcua-localdb-design.md)) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |
| LocalDb (embedded cache + 2-node sync) | **`0.3.7` PUBLISHED 2026-11-12 (scadaproj#53) and pinned by both consumers** (ScadaBridge `12fa7aff`, OtOpcUa `e3031654`; not deployed) — Sqlite 10.0.13; sync endpoint enforces `ApiKey` (#36; set the same key on both nodes); FK-ordered snapshots, PK upserts (no INSERT OR REPLACE), drift check on snapshots (#37–#39); `AUTOINCREMENT` tables refused at `RegisterReplicated`. **#16 fixed in `0.3.7`:** a tombstone is pruned only once past `TombstoneRetention` AND held by the peer (retention restarts at delivery), so a peer offline past retention no longer resurrects deleted rows; optional `TombstoneHardCap` (default off) bounds ledger growth during an outage. **Adopted** by ScadaBridge (Phases 1+2 merged; pins **`0.3.7`**) and OtOpcUa (Phases 1+2 merged; pins **`0.3.7`**; deployed builds still `0.3.4`) — no version skew remains (every release is wire-compatible, so one would be supported). Replication is **default-OFF**; enabled on the docker rigs' site-a pairs and on the **QA pair vq03/vq04** (ScadaBridge site pair + OtOpcUa pair — OtOpcUa's sync only actually CONNECTED after the 2026-09-10 `b8b4c18f` upgrade; the old build answered the sync POST with the login 302); DEV vd03 is single-node. Operational constraints: a pair stops/starts together (no rolling upgrade), the sync `ApiKey` must match on both nodes, and from `0.3.7` the tombstone ledger grows while a peer is down (no more resurrection past `TombstoneRetention`). Full release narrative (`0.1.1`→`0.3.7`) + gates: [`ZB.MOM.WW.LocalDb/CLAUDE.md`](ZB.MOM.WW.LocalDb/CLAUDE.md). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + the two adoption designs ([ScadaBridge](docs/plans/2026-07-19-scadabridge-localdb-design.md), [OtOpcUa](docs/plans/2026-07-20-otopcua-localdb-design.md)) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |
Each hosted library directory carries its own `CLAUDE.md` with the full version narrative, consumer matrix and
adoption record — it loads automatically when you work under that directory. Health, Telemetry and Configuration
ZB.MOM.WW.LocalDb/CLAUDE.md
+5 -3
View File
@@ -4,9 +4,11 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #16 (resurrection past `TombstoneRetention`) was fixed in `0.3.7`. The history below is a record, not a status tracker; file new follow-ups as issues.
**`0.3.5` (unpublished, scadaproj#26):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change.
**PUBLISHED `0.3.7` (2026-11-12, scadaproj#53)** — all three packages on the Gitea feed (built from scadaproj `30a12dd`, Release, 259/259), carrying the 0.3.5 + 0.3.6 changes below. SHA256: `ZB.MOM.WW.LocalDb.0.3.7.nupkg` `5ac1db336e9b71700cd253bfe986acc13d24d0286fcc38bb277317845c892321`, `.Contracts` `5e7a6fce04f415720f9a6aac7e21ca43d1e842f6a74282120ae8aae2a62eb824`, `.Replication` `f579f1218ccc279836841c06e2989e8d0d67009866548eeea6e162d4eb45b07d` (downloaded copies match). 0.3.5 and 0.3.6 were never published on their own. **Pins:** ScadaBridge `0.3.7` (`main` `12fa7aff`, ScadaBridge#130), OtOpcUa `0.3.7` (`master` `e3031654`, lmxopcua#889); not deployed (DEV vd03 / QA vq03+vq04 still run 0.3.4 builds). Deploy notes for the QA pairs: stop/start each pair together, `LocalDb:Replication:ApiKey` byte-identical on both nodes (now checked by the library too), and the tombstone ledger grows while a peer is down.
**`0.3.6` (unpublished, scadaproj#36–#39, #46, #50):** code-review fixes, all wire-compatible (no proto, `lib_schema_version` or on-disk schema change).
**`0.3.5` (scadaproj#26, published as part of 0.3.7):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change.
**`0.3.6` (scadaproj#36–#39, #46, #50, published as part of 0.3.7):** code-review fixes, all wire-compatible (no proto, `lib_schema_version` or on-disk schema change).
- **#36 LocalDb-001:** the passive `LocalDbSyncService` now enforces `LocalDb:Replication:ApiKey` itself (Bearer, constant-time, checked before the handshake and before newest-wins takeover; `Unauthenticated` otherwise). Blank key = unchanged behaviour plus a startup warning. ScadaBridge's and OtOpcUa's own `LocalDbSyncAuthInterceptor`s use the same key and header, so they keep working; they are now redundant except for their fail-closed-when-blank rule.
- **#37 LocalDb-002:** snapshots deliver FK-linked rows in any order. `SnapshotStreamer` sends referenced tables first (Kahn order, name tie-break); `SnapshotApplier` defers rows that fail only SQLITE_CONSTRAINT_FOREIGNKEY (`LwwApplier.ApplyBatchAsync(..., deferForeignKeyViolations: true)`), retries them as rows land, and dead-letters true orphans at `SnapshotComplete`. Before, a child that arrived first was dead-lettered and acked over permanently.
- **#38 LocalDb-003:** the applier upserts with `INSERT … ON CONFLICT(<pk>) DO UPDATE` (`DO NOTHING` when every column is PK) instead of `INSERT OR REPLACE`. A secondary-`UNIQUE` clash is now a dead-letter instead of a silent delete of another row; as a side effect a remote update of a parent with `ON DELETE RESTRICT` children applies instead of being dead-lettered.
@@ -16,7 +18,7 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA
- **#50:** snapshot-path dead-letters (poison, drift, FK orphan, cap spill) now count in `localdb.sync.dead_lettered` (`SnapshotApplier` takes `LocalDbMetrics`, wired by `SyncSessionFactory`). New option **`LocalDb:Replication:MaxSnapshotDeferredRows`** (default 100,000, validated > 0) caps snapshot rows held for a missing FK parent; past it the oldest are re-applied once and dead-lettered if still orphaned. **Open note (cross-repo, #50):** once ScadaBridge and OtOpcUa pin 0.3.6, their own `LocalDbSyncAuthInterceptor`s duplicate the library check — they differ only in refusing every stream when no key is set. Each app decides whether to keep that stricter rule or delete its interceptor; nothing in this library changes for either choice.
249/249 tests.
**`0.3.7` (unpublished, scadaproj#16 — owner chose option A):** a tombstone is pruned only when it is past `TombstoneRetention` **and** the peer holds it. `OplogStore.PruneTombstonesAsync` holds every tombstone while database-wide (`__localdb_peer_state.needs_snapshot`) or per-table (`__localdb_snapshot_state`) snapshot debt is outstanding, and a LOCAL tombstone while its delete is still unacked (`hlc >=` the first unacked oplog row's HLC — ledger and oplog share the HLC, local HLCs rise with seq). Delivery restamps `tombstone_utc` (in `RecordPeerAckAsync` for tombstones in the newly acked range; in `ClearSnapshotDebtAsync` for the snapshot's scope), so retention runs from delivery and the returning peer's stale queued writes get a full retention to drain. Remote-origin tombstones age on retention alone. New optional **`LocalDb:Replication:TombstoneHardCap`** (default `null` = off, validated ≥ `TombstoneRetention`) bounds an undelivered tombstone's life — past it the old risk returns. **The snapshot path (issue's second note) is covered by the debt hold:** the cap prune sets `needs_snapshot` in the same transaction that drops undelivered oplog rows, so the ledger tombstone the snapshot is built from survives until that snapshot is acknowledged. Cost: ledger growth for the length of an outage. Not covered: a peer restored from an old backup of its own file. No proto, `lib_schema_version` or on-disk schema change. Regression test `TombstoneResurrectionTests.PeerOfflinePastRetention_DeleteStillWinsOnBothNodes` (fails on 0.3.6 with the row resurrected, `A=1`) + positive control without the prune + pruned-after-delivery control; 5 `OplogStoreTests` with a controlled clock; every guard clause mutation-killed. 259/259.
**`0.3.7` (published 2026-11-12, scadaproj#16 — owner chose option A):** a tombstone is pruned only when it is past `TombstoneRetention` **and** the peer holds it. `OplogStore.PruneTombstonesAsync` holds every tombstone while database-wide (`__localdb_peer_state.needs_snapshot`) or per-table (`__localdb_snapshot_state`) snapshot debt is outstanding, and a LOCAL tombstone while its delete is still unacked (`hlc >=` the first unacked oplog row's HLC — ledger and oplog share the HLC, local HLCs rise with seq). Delivery restamps `tombstone_utc` (in `RecordPeerAckAsync` for tombstones in the newly acked range; in `ClearSnapshotDebtAsync` for the snapshot's scope), so retention runs from delivery and the returning peer's stale queued writes get a full retention to drain. Remote-origin tombstones age on retention alone. New optional **`LocalDb:Replication:TombstoneHardCap`** (default `null` = off, validated ≥ `TombstoneRetention`) bounds an undelivered tombstone's life — past it the old risk returns. **The snapshot path (issue's second note) is covered by the debt hold:** the cap prune sets `needs_snapshot` in the same transaction that drops undelivered oplog rows, so the ledger tombstone the snapshot is built from survives until that snapshot is acknowledged. Cost: ledger growth for the length of an outage. Not covered: a peer restored from an old backup of its own file. No proto, `lib_schema_version` or on-disk schema change. Regression test `TombstoneResurrectionTests.PeerOfflinePastRetention_DeleteStillWinsOnBothNodes` (fails on 0.3.6 with the row resurrected, `A=1`) + positive control without the prune + pruned-after-delivery control; 5 `OplogStoreTests` with a controlled clock; every guard clause mutation-killed. 259/259.
## Index history (migrated from the scadaproj root CLAUDE.md)