From fd180e576ff97932f4c0d50184f9bd22a7db7e80 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 6 Nov 2026 15:34:30 -0500 Subject: [PATCH] docs: point markdown backlogs at Gitea issues (status lives in Gitea) Adds the family backlog standard to the root index (open work = Gitea issues, shared label set, BTDB excluded) and converts components/*/GAPS, hosted-lib CLAUDE.md notes, code-reviews, infra/glauth and plan docs to banner + issue pointers (scadaproj #5-#19, ScadaBridge#79). --- CLAUDE.md | 17 +++++++++++++++++ ZB.MOM.WW.Audit/README.md | 6 +++--- ZB.MOM.WW.Auth/CLAUDE.md | 4 +++- ZB.MOM.WW.Configuration/CLAUDE.md | 6 +++--- ZB.MOM.WW.Health/CLAUDE.md | 4 +++- ZB.MOM.WW.LocalDb/CLAUDE.md | 4 +++- ZB.MOM.WW.Overview/CLAUDE.md | 4 +++- ZB.MOM.WW.Secrets/CLAUDE.md | 4 +++- ZB.MOM.WW.Telemetry/CLAUDE.md | 4 +++- ZB.MOM.WW.Telemetry/README.md | 2 +- ZB.MOM.WW.Theme/CLAUDE.md | 4 +++- code-reviews/README.md | 8 ++++++++ code-reviews/regen-readme.py | 8 ++++++++ components/README.md | 11 ++++++++--- components/audit/GAPS.md | 5 +++++ components/auth/GAPS.md | 5 +++++ components/configuration/GAPS.md | 5 +++++ components/health/GAPS.md | 11 ++++++++--- components/observability/GAPS.md | 11 ++++++++--- components/secrets/GAPS.md | 7 ++++++- components/ui-theme/GAPS.md | 7 ++++++- .../2026-07-22-overview-dashboard-design.md | 2 +- docs/plans/2026-08-13-vd03-nam-ldap-cutover.md | 9 +++++++-- infra/glauth/README.md | 4 +++- 24 files changed, 123 insertions(+), 29 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 00a1f15..d3073e9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -249,6 +249,23 @@ msbuild Database\Database.sqlproj /p:Configuration=Release # SSDT → bin\Rel > (`baseDN dc=zb,dc=local`, Transport=None). Source of truth and deploy runbook: > [`scadaproj/infra/glauth/`](infra/glauth/) (`config.toml` + `docker-compose.yml` + `README.md`). +## Backlog / open work (family standard, 2026-11-06) + +**Open work lives in Gitea issues, not markdown** — in each repo's own tracker on `gitea.dohertylan.com/dohertj2/` +(`lmxopcua`, `mxaccessgw`, `ScadaBridge`, `historiangw`, `scadaproj`). Check `tea issues list --repo dohertj2/` +before trusting any "open"/"pending" claim in a doc, including this index. + +- Markdown backlogs (`pending.md`, ScadaBridge's deferred-work register, archreview trackers, `components/*/GAPS.md`, + `*.tasks.json`) are **historical/design records**: each carries a banner and `→ #N` pointers to the issue that now owns + the item. Do not add status rows to them; file an issue and reference it as `#N` / `#N`. +- Every repo has the same labels: `type:{bug,feature,followup,review-finding,deploy-gate,debt,docs,test,infra}`, + `priority:p0..p3`, `blocked`, `cross-repo`, and `migrated` (items moved from markdown in the 2026-11-06 sweep). + OtOpcUa's `queue/*` labels belong to its automation — don't set them by hand. +- Family-wide work (shared-lib rollouts, adoption tracking) gets one `cross-repo` issue here in `scadaproj`; the per-repo + change gets its own issue in the repo that owns it, cross-linked. +- BTDB is excluded (corporate remote at `wonder-app-vd03:3000`; Claude artifacts stay local) — its backlog stays in its + gitignored local markdown. + ## Refreshing this index This file is meant to be re-scanned when `scadaproj` is opened in Claude Code: diff --git a/ZB.MOM.WW.Audit/README.md b/ZB.MOM.WW.Audit/README.md index 2e1bac5..afc5709 100644 --- a/ZB.MOM.WW.Audit/README.md +++ b/ZB.MOM.WW.Audit/README.md @@ -16,9 +16,9 @@ Canonical audit event model, best-effort writer seam, and redactor seam for the | Consumer | ZB.MOM.WW.Audit | |---|:---:| -| **OtOpcUa** | yes (adoption deferred) | -| **MxAccessGateway** | yes (adoption deferred) | -| **ScadaBridge** | yes (adoption deferred — "align, don't replace") | +| **OtOpcUa** | yes (adoption deferred; stale → #17) | +| **MxAccessGateway** | yes (adoption deferred; stale → #17) | +| **ScadaBridge** | yes (adoption deferred — "align, don't replace"; stale → #17) | Adoption is tracked in `components/audit/GAPS.md` in the outer `scadaproj` workspace. Each app brings its own transport (Akka broadcast / SQLite append / SQL ingest) and domain vocabulary (channels / kinds / event-types) — those stay per-project. The shared library provides the canonical record and the two seams that decouple "what to audit" from "how to store it". diff --git a/ZB.MOM.WW.Auth/CLAUDE.md b/ZB.MOM.WW.Auth/CLAUDE.md index 1f78d69..20cdac1 100644 --- a/ZB.MOM.WW.Auth/CLAUDE.md +++ b/ZB.MOM.WW.Auth/CLAUDE.md @@ -2,6 +2,8 @@ Shared login / identity / API-key library for the **ZB.MOM.WW SCADA family** (4 packages + `ApiKeys.SqlServer`; consumed by OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway). Build/test from this directory: `dotnet test` (the `ApiKeys.SqlServer` live suite is gated on `AUTH_SQLSERVER_CONNSTR`). Component spec + gaps: `../components/auth/`. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #5 (roll apps to `0.4.1`), #6 (VD03 NAM AD cutover), #18 (Secrets migrator jitter, ported from here). The history below is a record, not a status tracker; file new follow-ups as issues. + ## Index history (migrated from the scadaproj root CLAUDE.md) > Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.** @@ -36,6 +38,6 @@ MxAccessGateway & HistorianGateway → the original four; ScadaBridge → those **`0.1.4` (2026-07-09, archreview G-2):** optional `ApiKeyRecord.ExpiresUtc` (nullable = never-expires), `ApiKeyFailure.KeyExpired`, `ApiKeyVerifier` expiry enforcement (inclusive, injected clock), a `CreateKeyAsync(expiresUtc)` overload, and a SQLite **schema v3** additive `expires_utc` column (idempotent `ALTER`; donor v2 DBs upgrade in place, no key invalidated). **All four Auth pkgs published to the Gitea feed at `0.1.4`** (2026-07-09). **HistorianGateway adopted the expiry surface at `0.1.4`** — `apikey create --expires` + dashboard Expires column (Task-10 done). **`0.1.5` (2026-07-18)** is `0.1.4` plus the `SQLitePCLRaw.lib.e_sqlite3 2.1.12` transitive security pin (GHSA-2m69-gcr7-jv3q), no API change; all four apps now pin it (see consumer pins above). **`0.2.0` (2026-08-10) — LDAP backup-DC sticky failover** (plan `docs/plans/2026-08-10-ldap-backup-dc-failover.md`, design `…-design.md`; **4 pkgs published to the feed, restore-verified**; family `Directory.Build.props` now `0.2.0`): one additive `LdapOptions.FallbackServers` list (`"host"` or `"host:port"`, empty default = failover off, single-endpoint behaviour byte-unchanged) makes `LdapAuthService` walk an ordered endpoint list (primary + fallbacks) on **system-side failures only** — connect/TLS, service-account bind, or search. A user-side answer from a healthy directory (`Success`/`UserNotFound`/`AmbiguousUser`/`BadCredentials`/`GroupLookupFailed`) is **final and never retried elsewhere** (re-asking a second DC about a rejected password would multiply one bad-password event into several against the account lockout policy). Endpoint preference is **sticky** — the last endpoint that gave *any* authoritative answer (success **or** a user-side failure) is tried first next time — so a primary outage costs its connect timeout once, not per login; an exhausted walk leaves the preference untouched. Fail-closed contract preserved per-endpoint; boot validation rejects a malformed `FallbackServers` entry naming the field; optional `ILogger` emits a Warning per failed hop (endpoint + stage, no secrets) and an Information on sticky change; wired via `AddZbLdapAuth` (`GetService`, so logging is optional). Motivation: the ZB production directory is NAM AD with two DCs. Unit suite covers the walk/stickiness (14 tests) + env-gated live GLAuth failover test (`ZB_LDAP_IT=1`). **No `Abstractions` contract change beyond the new option; no `LdapAuthFailure` enum change.** App adoption (pin bump to `0.2.0` + authoring `FallbackServers`) is a follow-up. **`0.2.1` (2026-08-12) — AD continuation referrals no longer fail the search** (**4 pkgs published to the feed, restore-verified**; the assembly restored *from the feed* carries the fix and `0.2.0` does not — checked as a positive control, not by trusting the push). Active Directory answers **any search based at a domain root** with continuation references (to `CN=Configuration`, `DC=ForestDnsZones`, …). With referral following disabled — the default, and what we want, since chasing one would re-bind somewhere we never authenticated — Novell surfaces each by throwing `LdapReferralException` from `Next()`. That escaped `NovellLdapConnection.Search`, and `LdapAuthService` maps a failed search to `ServiceAccountBindFailed`, so **every login against `SearchBase = DC=nam,DC=zimmer,DC=com` failed system-side on both DCs while the directory, the credentials and the service account were all healthy** — ~546 ms, identical on each endpoint, so failover could not help either. Found by the new `tools/LdapProbe` (below) and live-proven fixed against the ZB NAM domain root. The search loop now skips referrals (`ReadEntries(ILdapSearchResults)`, extracted so it is unit-testable through the Novell seam without a live directory) and continues; a 1000-referral cap guards a client that throws without advancing, since that would spin forever inside a fail-closed auth path. **Non-referral faults still propagate** — swallowing those would turn "the directory is broken" into "no such user", which is the wrong answer to give about someone's credentials. 7 new tests (`NovellSearchReferralTests`), **6 verified red against the pre-fix code**, the seventh being the non-referral control that must stay green; suite 257 pass / 2 skip / 0 fail. **No `Abstractions` change; no config change.** **Consumer impact: an app on `0.1.5` cannot authenticate against an AD domain root at all**, so the pin bump is a functional fix, not housekeeping. **Closed 2026-08-13: all four apps adopted `0.2.1`** (see the table row for commits), and `FallbackServers` was live-gated the same day against the redundant local GLAuth pair (`infra/glauth-redundant/`). -**`0.3.0` (2026-09-02) — `ZB.MOM.WW.Auth.ApiKeys.SqlServer`, a SQL Server provider for the API-key store** (design `ScadaBridge/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; **5 pkgs published to the feed**, `ApiKeys.SqlServer` search-verified from the ScadaBridge checkout; family `Directory.Build.props` now `0.3.0`). Motivation: `Auth.ApiKeys`' SQLite store is a per-node file, so on a two-node active/standby host (ScadaBridge central) a key minted on one node exists only there and a failover 401s every inbound caller (ScadaBridge deferred-work row 108). The new package implements `IApiKeyStore`/`IApiKeyAdminStore`/`IApiKeyAuditStore` over `Microsoft.Data.SqlClient` in a `zbauth` schema (v1: `schema_version`, `api_keys`, `api_key_audit`; all timestamps `DATETIMEOFFSET(7)`, tick-exact) with a concurrent-boot-safe migrator (Serializable tx, retry on 1205/2714/1913/2627 with **jittered** backoff — the Secrets replicator's flat `100*attempt` backoff exhausted five retries about once per sixty 8-way boots in the port's soak test, so the same latent defect is an open follow-up there) and a `SqliteApiKeyStoreImporter` that copies an existing SQLite file into the schema insert-where-absent (idempotent, so both nodes of a pair can import their own file and the result is the union). Wiring: `services.AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)` registers the SQL Server stores + migrator **before** delegating to `AddZbApiKeyAuth`, whose `TryAdd*` registrations then yield (same displacement mechanism as Secrets 0.5.0). One `Abstractions` addition, additive: `IApiKeyStoreMigrator`, which `SqliteAuthStoreMigrator` now implements and `ApiKeyAdminCommands` accepts via a new ctor (old `SqliteAuthStoreMigrator` ctor retained — the three `0.2.1` consumers compile unchanged). Store semantics are pinned to the SQLite contract tests re-run live (mark-used skips revoked keys, delete only removes revoked keys, rotate clears last-used/revoked, list newest-first). 92 tests in the new project, 351 across the solution, 0 failed. A DBA-run copy of the DDL ships at `docs/sql/zbauth-schema.sql`. +**`0.3.0` (2026-09-02) — `ZB.MOM.WW.Auth.ApiKeys.SqlServer`, a SQL Server provider for the API-key store** (design `ScadaBridge/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; **5 pkgs published to the feed**, `ApiKeys.SqlServer` search-verified from the ScadaBridge checkout; family `Directory.Build.props` now `0.3.0`). Motivation: `Auth.ApiKeys`' SQLite store is a per-node file, so on a two-node active/standby host (ScadaBridge central) a key minted on one node exists only there and a failover 401s every inbound caller (ScadaBridge deferred-work row 108). The new package implements `IApiKeyStore`/`IApiKeyAdminStore`/`IApiKeyAuditStore` over `Microsoft.Data.SqlClient` in a `zbauth` schema (v1: `schema_version`, `api_keys`, `api_key_audit`; all timestamps `DATETIMEOFFSET(7)`, tick-exact) with a concurrent-boot-safe migrator (Serializable tx, retry on 1205/2714/1913/2627 with **jittered** backoff — the Secrets replicator's flat `100*attempt` backoff exhausted five retries about once per sixty 8-way boots in the port's soak test, so the same latent defect is an open follow-up there → #18) and a `SqliteApiKeyStoreImporter` that copies an existing SQLite file into the schema insert-where-absent (idempotent, so both nodes of a pair can import their own file and the result is the union). Wiring: `services.AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)` registers the SQL Server stores + migrator **before** delegating to `AddZbApiKeyAuth`, whose `TryAdd*` registrations then yield (same displacement mechanism as Secrets 0.5.0). One `Abstractions` addition, additive: `IApiKeyStoreMigrator`, which `SqliteAuthStoreMigrator` now implements and `ApiKeyAdminCommands` accepts via a new ctor (old `SqliteAuthStoreMigrator` ctor retained — the three `0.2.1` consumers compile unchanged). Store semantics are pinned to the SQLite contract tests re-run live (mark-used skips revoked keys, delete only removes revoked keys, rotate clears last-used/revoked, list newest-first). 92 tests in the new project, 351 across the solution, 0 failed. A DBA-run copy of the DDL ships at `docs/sql/zbauth-schema.sql`. **`tools/LdapProbe`** (`ldapprobe`, not packable, outside the Auth solution) is the diagnostic that found the above: it drives the **real** `LdapAuthService` so its verdict is what the four apps produce, then separately queries the directory for what the library cannot report — full group DNs, **nested** groups via `LDAP_MATCHING_RULE_IN_CHAIN`, the **primary group** (which AD omits from `memberOf`, so it can never be a role group), group type (security vs distribution), decoded `userAccountControl`, referral counts, and the TLS chain with key sizes. `--list-groups ''` enumerates every group in the directory matching a pattern, which a membership dump structurally cannot do. Findings recorded in [`ldap_setup.md`](ldap_setup.md), including the **`UG_WW_*` Wonderware group convention (39 groups)** that partly overturns that doc's earlier "no group→role convention exists" claim — it was true of Ignition, false of the domain. diff --git a/ZB.MOM.WW.Configuration/CLAUDE.md b/ZB.MOM.WW.Configuration/CLAUDE.md index a0008a6..469f819 100644 --- a/ZB.MOM.WW.Configuration/CLAUDE.md +++ b/ZB.MOM.WW.Configuration/CLAUDE.md @@ -4,7 +4,7 @@ Startup configuration-validation library for the **ZB.MOM.WW SCADA family** (OtO The library normalizes the three-project configuration-validation surface: a failure-accumulating `IValidateOptions` base, reusable rule primitives, a bind+validate+`ValidateOnStart` DI extension, and a pre-host `ConfigPreflight` aggregator for raw `IConfiguration` — so the plumbing is written once and domain rules stay per-project. -**Built at 0.1.0. Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01** (local default branches; not yet pushed to remotes). Adoption tracked in `~/Desktop/scadaproj/components/configuration/GAPS.md`. +**Built at 0.1.0. Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01** (local default branches; not yet pushed to remotes — stale → #17). Adoption tracked in `~/Desktop/scadaproj/components/configuration/GAPS.md`. --- @@ -66,7 +66,7 @@ ZB.MOM.WW.Configuration/ ## Status -Part of the **scadaproj component-normalization family** — this is the configuration + validation component. Built at **0.1.0**. **Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01** (local default branches; not yet pushed to remotes) — per-app result is tracked in: +Part of the **scadaproj component-normalization family** — this is the configuration + validation component. Built at **0.1.0**. **Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01** (local default branches; not yet pushed to remotes — stale → #17) — per-app result is tracked in: - `~/Desktop/scadaproj/components/configuration/GAPS.md` @@ -97,7 +97,7 @@ The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Configurat The implementation plan is at [`docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md). **Adopted across all three apps on 2026-06-01** (OtOpcUa, MxAccessGateway, ScadaBridge) on each repo's -local default branch (`main`/`master`) — merged, **not yet pushed** to remotes; the package was first +local default branch (`main`/`master`) — merged, **not yet pushed** to remotes (stale → #17); the package was first published to the Gitea feed. Behaviour-preserving onto `OptionsValidatorBase`/`AddValidatedOptions` for MxGateway + ScadaBridge (validator messages byte-identical), `StartupValidator` → `ConfigPreflight` for ScadaBridge, and net-new `Ldap`/`OpcUa` validators for OtOpcUa. Per-app result tracked in diff --git a/ZB.MOM.WW.Health/CLAUDE.md b/ZB.MOM.WW.Health/CLAUDE.md index f8c83c9..884f702 100644 --- a/ZB.MOM.WW.Health/CLAUDE.md +++ b/ZB.MOM.WW.Health/CLAUDE.md @@ -2,6 +2,8 @@ Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central health process or network hop; probes run in-process alongside the application. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #11 (re-verify per-app probe coverage vs the spec). The history below is a record, not a status tracker; file new follow-ups as issues. + The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic. **Built at 0.3.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`. @@ -99,6 +101,6 @@ orchestrator wiring, and ScadaBridge's distributed health-monitoring pipeline. The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) (.NET 10; 3 packages — `ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`; 64 tests (verified 2026-07-18); `dotnet pack` → 3 nupkgs @ 0.1.0). -**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified — see [`components/health/GAPS.md`](components/health/GAPS.md). +**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified (→ #11) — see [`components/health/GAPS.md`](components/health/GAPS.md). Build/test from `ZB.MOM.WW.Health/`: `dotnet test`. Consumer matrix: MxAccessGateway → core only; OtOpcUa & ScadaBridge → all three packages. diff --git a/ZB.MOM.WW.LocalDb/CLAUDE.md b/ZB.MOM.WW.LocalDb/CLAUDE.md index e624bf9..85318a4 100644 --- a/ZB.MOM.WW.LocalDb/CLAUDE.md +++ b/ZB.MOM.WW.LocalDb/CLAUDE.md @@ -2,6 +2,8 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA family** (3 packages; consumed by ScadaBridge and OtOpcUa). Build/test/pack from this directory: `dotnet build ZB.MOM.WW.LocalDb.slnx`, `dotnet test ZB.MOM.WW.LocalDb.slnx` (fully offline), `dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o artifacts`. Design: `../docs/plans/2026-07-17-localdb-design.md`; adoption designs `../docs/plans/2026-07-19-scadabridge-localdb-design.md` + `../docs/plans/2026-07-20-otopcua-localdb-design.md`. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #16 (resurrection guard past `TombstoneRetention`). The history below is a record, not a status tracker; file new follow-ups as issues. + ## Index history (migrated from the scadaproj root CLAUDE.md) > Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.** @@ -10,4 +12,4 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA | Component | Status | Goal | Design | Implementation | |---|---|---|---|---| -| LocalDb (embedded cache + 2-node sync) | **Adopted by ScadaBridge — Phases 1 AND 2 COMPLETE and MERGED** to `main` (2026-07-20, PR #23, merge `28ca04d7`, pushed to origin; both `feat/localdb-phase*` branches fully contained and kept only as history). Libs **`0.3.4`** on the Gitea feed (3 pkgs; `0.1.0`–`0.3.3` also present). **`0.3.4` (2026-08-20, scadaproj `75c1897` — ScadaBridge register row 106; sender-side, wire-compatible, no schema change):** `SnapshotStreamer` stamps `as_of_seq = max(MAX(seq), last_acked_seq)` instead of bare `MAX(seq)`, which read **0** once ack-pruning had emptied a converged sender's oplog — so a rebuilt peer (watermark 0) was healed by the snapshot but never lifted by it, and the `0.1.2` empty-oplog horizon rule re-streamed the whole database on EVERY reconnect until the sender wrote again (ScadaBridge's site-a-a: 127 k rows per restart, observed on two consecutive restarts). Pinned by `WipedPeer_SnapshotLiftsItsWatermark_…`; 216/216. **ScadaBridge pins `0.3.4`** (2026-08-20; live: first restart sent one last snapshot `as_of 149`, second restart sent none); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`) — note OtOpcUa's rebuilt-node path has the same loop until it re-pins. **`0.3.3` (2026-08-20, scadaproj `122940c` — ScadaBridge arch-review P4-7; wire-compatible, no proto/schema change)** closes two latent `LwwApplier` hazards: (1) **`RegisterReplicated` refuses a table that is the parent of a row-mutating `ON DELETE` FK** (`CASCADE`/`SET NULL`/`SET DEFAULT`; `RESTRICT`/`NO ACTION` allowed) via `LocalDbRegistrationException` naming `child.column ON DELETE ` — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired child actions while capture was suppressed, a silent divergence for any future FK'd table (none of ScadaBridge's ten or OtOpcUa's four has an FK today; registration-time check only, since SQLite can't add an FK to an existing table); (2) the per-entry apply (delete-or-upsert + ledger write) is **atomic under a `SAVEPOINT`** — a fault between the two statements left a deleted row with a live ledger that `SnapshotStreamer` warns-and-skips forever; poison-catch semantics preserved. 213/213 tests (+9, failure injected with a `RAISE(ABORT)` trigger on the ledger, no production seam). **ScadaBridge pins `0.3.3`** (2026-08-20); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`). **`0.3.2` (2026-08-19, scadaproj `f62b017` — ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no proto/schema change)**: (1) the sync pump **faults a live session when a cap-prune invalidates its position** (oplog min-seq horizon check before each read batch + on the batch head; new public `LocalDbOplogGapException` rides the 0.3.1 fault teardown) — previously a slow-but-alive peer silently streamed past the pruned gap and the pair diverged until the next reconnect; (2) **client h2 keepalive pings** via new validated `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (60s/20s, `Always` policy); (3) the server single-stream guard is **newest-wins with an awaited handover** (`TakeoverTimeout` 10s) — a fresh authenticated `Sync` displaces a dead incumbent instead of being refused `AlreadyExists` forever by a half-open link whose zombie side reports connected. 204/204 tests. **ScadaBridge pins `0.3.2`** (2026-08-19, adding Kestrel h2 keepalive on its site gRPC listener as the server half); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`). **`0.3.1` (2026-08-18, scadaproj `3e5a5fd`) flushes a faulting session's control lane before its writer is cancelled** — on a handshake-phase fault (i.e. a schema-mismatch rejection) `RunAsync`'s teardown cancelled the writer while the session's OWN Handshake could still be sitting in the lane (the enqueue→receive→validate→fault path has no yields while the writer's wakeup waits in the thread-pool queue; a pinning test lost the race 39/40), and mismatch detection is symmetric — the peer computes its fail-closed verdict FROM that Handshake — so the healthy peer misread a permanent schema mismatch as a generic disconnect and reconnect-looped at the fast backoff instead of the classified `FailedPrecondition`→LogError+max-backoff path. Teardown now completes both lanes so the writer drains what is already enqueued (2 s deferred cancel as the wedged-transport backstop); caller-initiated shutdown is not delayed, **no wire or config change** — an older peer simply receives the Handshake it was always owed. Closes ScadaBridge deferred-register row 41 (its flaky `Handshake_DigestMismatch_FailsClosed` reproduced 2/6 pre-fix under load, 6/6 green post-fix; the other named flake, `BatchBudgetTests.Pump_RowLargerThanTheBudget…`, was a harness defect — session faults hidden as opaque 15 s timeouts — fixed with a run-observing `NextAsync`). **ScadaBridge re-pinned through `0.3.1` to `0.3.2` (2026-08-19); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`).** **`0.3.0` (2026-08-15) makes the snapshot-resync flag PER-TABLE** (branch `localdb-per-table-snapshot`; the residual `0.2.1` explicitly deferred). On-disk bookkeeping schema goes to **v3** — a new `__localdb_snapshot_state` table, upgraded in place on open (the transition is a version stamp only: the shared DDL creates the table with `IF NOT EXISTS` on every open, and empty is the correct v3 state for a file that never baselined). **Wire-compatible with `0.1.x`/`0.2.x`, and `lib_schema_version` deliberately STAYS `1`** because `ValidateHandshake` compares it fail-closed for equality — bumping it would stop mixed pairs syncing outright, so the extension is negotiated by **capability**: `Handshake.supports_partial_snapshot` (new field 5) and `SnapshotBegin.tables` (new field 2, where **empty == all tables**, which is exactly what a pre-`0.3.0` sender emits and a pre-`0.3.0` receiver assumes). `RegisterReplicated(table, baselineExistingRows: true)` now flags **only that table**, so a snapshot streams only the flagged tables; the database-wide `__localdb_peer_state.needs_snapshot` survives unchanged for the cases that genuinely reach every table (oplog cap prune, impossible peer ack). **A peer that does not advertise the capability is served a FULL snapshot** — a scoped one would make it advance `last_applied_remote_seq` to the snapshot's as-of and silently discard every delta below it for the tables left out. For the same reason **a partial snapshot moves no watermark on either side**: the receiver keeps `last_applied_remote_seq` and the sender keeps `_sentThruSeq`, so the overlapping deltas are still pumped and merged by LWW. Crash safety: snapshot debt now clears on the receiver's new **`SnapshotAck`** (exchanged only between `0.3.0` peers) rather than on enqueue, so an ungraceful exit mid-snapshot re-streams the same scope on reconnect; against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue is kept, there being no ack to wait for. Composes with `0.2.1`'s bounded 64-message inbox and out-of-handshake snapshot (the ack rides the unbounded CONTROL lane, like `DeltaAck`). 199/199 tests, including a single-table-baseline scope test, a mixed-version interop test (old peer ⇒ full snapshot), an ack-gated-debt test and a v2→v3 upgrade test. **ScadaBridge pins `0.3.0`**; `Host/SiteLocalDbSetup.cs` needed no restructuring — it already calls `RegisterReplicated` once per table, which is what per-table flagging keys off, so adding an eleventh replicated table to an already-replicating site now streams that one table instead of all eleven. **`0.2.1` (2026-08-14) closes two adversarial-review findings against `0.2.0`** (same branch `localdb-perf-remediation`; wire-compatible, no proto/`lib_schema_version`/on-disk-schema change): **`DeregisterReplicated` now flushes the HLC's durable crash anchor inside its own transaction, before the prune** — clock recovery is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)` and `hlc_high_water` was written only by `Dispose`, so on a file whose tables had all been deregistered an ungraceful exit left the clock nothing to recover from and it could re-issue stamps below HLCs peers already held (LWW then discards them as older); and **the gRPC sync inbox is bounded at 64 messages with wait-mode backpressure** instead of unbounded, capping inbound memory at ~`64 × MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a **mutual full snapshot** (the snapshot flag is per-DATABASE and both sides of a late-enabled pair baseline unconditionally), previously buffered whole in memory on both nodes at once. Bounding required moving the snapshot a node owes **out of the handshake** to run alongside a live receive loop; the old ordering left neither side draining its inbox while sending and deadlocks under a bounded transport (pinned by a capacity-1 mutual-snapshot test). Per-table `needs_snapshot` was considered and **deferred** at `0.2.1` — it needs an on-disk schema change; the cost was instead documented (trim high-volume tables like event logs before enabling replication late). **That residual is closed by `0.3.0`, above.** 188/188 tests at `0.2.1`. **`0.2.0` (2026-08-14) is the performance + lifecycle release from the ScadaBridge architectural review (WP3.3, branch `localdb-perf-remediation`)** — wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump); the on-disk bookkeeping schema goes to **v2**, upgraded in place on open. Six changes: **`DeregisterReplicated(table)`** drops a table's capture triggers and prunes its oplog + row-version rows (idempotent, and deliberately usable on triggers left by an EARLIER build — there was previously no way to stop capturing once a file had been registered); **`RegisterReplicated(table, baselineExistingRows: true)`** closes the late-opt-in hole, seeding the row-version ledger for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flagging a snapshot resync, because the snapshot streamer pages from that ledger and CDC never wrote pre-registration rows there — a node that started replicating LATE never sent them, silently and forever (the two APIs are a designed pair: the ledger prune is safe only because re-registration can re-baseline; deregistration must still be **symmetric**, which the fail-closed registered-table digest handshake enforces loudly); **byte-budget batching** via `MaxBatchBytes` (default 2 MB) with `MaxBatchSize` demoted to a secondary row cap, plus inbound delta/ack coalescing; **O(1) backlog depth** (`MAX(seq) − MAX(last_acked_seq, MIN(seq) − 1)`) replacing a `COUNT(*)` over the whole backlog that ran per maintenance tick AND per metrics scrape; `LwwApplier` prepare-once-per-statement; and the unused `__localdb_oplog_hlc` index dropped. **`0.1.2` (2026-07-21) fixes snapshot resync for a WIPED peer**: a converged pair prunes every oplog row on ack, and the handshake read an empty oplog as "no gap possible" — so the steady state of a healthy pair was the one state from which a rebuilt node (lost volume, re-imaged host) could never be back-filled; it came back empty and stayed empty until the next deploy. The gap is now measured against `last_acked_seq` when the oplog is empty. Found by the OtOpcUa Phase 1 live gate (check 4) as a documented limitation, then closed. **`0.1.3` (2026-07-21) closes the other half, which only became REACHABLE once back-fill worked**: the rebuilt peer's OWN writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq space and a rebuilt peer numbers from 1, so the healthy node's stale watermark made the sending side skip its whole oplog — the pair looked converged right up until the moment only the rebuilt node applied a deploy. Fixed by resetting the inbound watermark on a peer-node-id change (NOT last_acked_seq — that is what 0.1.2 keys off) plus rejecting a peer claim beyond anything we ever produced. Live-gated on the OtOpcUa docker-dev rig: `OtOpcUa/docs/plans/2026-07-21-localdb-followups-live-gate.md`. **`0.1.1` fixes a latent boot defect**: the library never created the parent directory of `LocalDb:Path` and `SqliteLocalDb` opens the file eagerly, so a missing directory was a hard boot failure (`SQLite Error 14`) — routine given the relative `./data/…` defaults, and masked on docker only by the volume mount. ScadaBridge had worked around it app-side; the fix moved into the library and the shim is deleted. **Phase 1** (merged in PR #23): `OperationTracking` + `site_events` consolidated into one `LocalDb:Path` database, fail-closed bearer interceptor on the sync endpoint, `ISyncStatus` health bridge, one-time legacy migrator. **Phase 2** (merged in the same PR): adds `sf_messages` + 7 site config tables (10 replicated in total) and **deletes** ScadaBridge's bespoke `SiteReplicationActor`, its messages, StoreAndForward's `ReplicationService`, and `ReplaceAllAsync` — including the whole **notify-and-fetch** exchange, so a standby now makes NO config fetch on deploy (`SiteReconciliationActor`'s startup fetch survives; different path). `notification_lists`/`smtp_configurations` are created but deliberately NEVER registered — permanently empty, and a channel for them would carry plaintext SMTP passwords. **Live gate PASS, all 10 checks** (`ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md`): byte-identical config rows with identical HLC + origin node id, standby zero-fetch, 3-table cascade delete converging with tombstones on both nodes, site failover with zero duplicate ids, clean both-nodes-together restart. **Replication remains default-OFF**, enabled on the docker rig's site-a pair only (site-b/site-c unreplicated as the default-OFF pin); **no production deployment has it enabled.** **Operational constraints:** a site pair must be stopped/started TOGETHER (rolling upgrade unsupported since the `SfBufferSnapshot` compat handler was deleted), a node offline beyond `TombstoneRetention` (7 d) can resurrect deleted rows, and — before `0.2.0` — `MaxBatchSize` batched by ROW COUNT against a 4 MB gRPC cap, which is why the rig pinned it to **16** (the 500 default would allow ~35 MB). **That pin is retired**: ScadaBridge consumed `0.2.0` on 2026-08-14 (branch `arch-review-remediation`), removed `MaxBatchSize` from both `docker/site-a` node configs and left `MaxBatchBytes` at its 2 MB default. The same consumption made `Host/SiteLocalDbSetup.cs` self-healing in both directions: an unreplicated node (site-b/site-c) now **deregisters** all ten tables at boot, dropping stale triggers an earlier build installed, and a replicating node registers with **`baselineExistingRows: true`**, so turning replication ON for a site that had been running without it finally converges on the rows already in the file rather than only on writes made after the restart. **Also adopted by OtOpcUa — Phase 1 COMPLETE and MERGED** to `master` 2026-07-21 (merge `c957db52`), and **Phase 2 (alarm store-and-forward on LocalDb) EXECUTED, live-gated and MERGED** to `master` (merge `06b5d81b`; the earlier "designed but NOT executed" claim was stale — corrected 2026-08-07; residual: live-gate checks 3+4 were never re-run on the post-mesh-Phase-6 topology): a pair-local consolidated SQLite cache of the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention) that lets a driver node **boot from cache when central SQL Server is unreachable**, registered in the `hasDriver` branch only (admin-only nodes never register it), with the same default-OFF fail-closed replication. Its live gate passed 8/8 and caught four defects offline tests missed. **Consumers: ScadaBridge (Phases 1+2) and OtOpcUa (Phases 1+2)** — **both pin `0.3.4`** as of 2026-09-03 (ScadaBridge since 2026-08-20; OtOpcUa `b8b4c18f`, a pure pin bump with tests unchanged versus its baseline). Earlier: **ScadaBridge pinned `0.3.0`** (2026-08-15, residual R1; it consumed `0.2.0` on 2026-08-14 under arch-review WP3.3) and **OtOpcUa pinned `0.2.1`** (2026-08-15, residual R4, merged to its `master` @ `5a7aae14` — a pure pin bump, no code change; it also corrected its `MaxBatchSize = 16` rationale, kept at 16 as the DB read-page bound). The remaining 0.2.1-vs-0.3.0 skew is supported: every release is wire-compatible (`0.3.0` negotiates its scoped snapshots by capability and falls back to a full snapshot for an older peer) and the schema v2/v3 upgrades are local-file state. **OtOpcUa closed the ScadaBridge deferred-register row 39 gap on 2026-08-18** (branch `perf/followups-r3`, commit `3e62f161`): `LocalDbSetup.OnReady` no longer registers all four tables unconditionally — registration follows one predicate, `LocalDb:Replication:PeerAddress` **OR** `:ApiKey` non-blank (the OR because only the dialling half of a pair sets `PeerAddress`; `SyncListenPort` is deliberately excluded, a port with no key being fail-closed). A configured node registers with **`baselineExistingRows: true`**, so a late replication enable finally converges on the rows already in the file; an unconfigured node **deregisters** at boot, dropping capture triggers an earlier build installed and logging once only when it actually cleaned something. Same shape ScadaBridge landed under arch-review WP3.3, written against OtOpcUa's own seams. OtOpcUa re-pinned to `0.3.4` on 2026-09-03 (`b8b4c18f`), so its snapshot flag is per-table too and the earlier per-DATABASE caveat (a late baseline streamed every registered table in full) no longer applies. Flipping the predicate is a both-nodes-together change (fail-closed registered-table digest handshake). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + [`docs/plans/2026-07-19-scadabridge-localdb-design.md`](docs/plans/2026-07-19-scadabridge-localdb-design.md) + [`docs/plans/2026-07-20-otopcua-localdb-design.md`](docs/plans/2026-07-20-otopcua-localdb-design.md) (both adoption designs live here); ScadaBridge holds the Phase 1 plan + Phase 2 gate under its own `docs/plans/`, OtOpcUa its two phase plans + two live gates | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | +| LocalDb (embedded cache + 2-node sync) | **Adopted by ScadaBridge — Phases 1 AND 2 COMPLETE and MERGED** to `main` (2026-07-20, PR #23, merge `28ca04d7`, pushed to origin; both `feat/localdb-phase*` branches fully contained and kept only as history). Libs **`0.3.4`** on the Gitea feed (3 pkgs; `0.1.0`–`0.3.3` also present). **`0.3.4` (2026-08-20, scadaproj `75c1897` — ScadaBridge register row 106; sender-side, wire-compatible, no schema change):** `SnapshotStreamer` stamps `as_of_seq = max(MAX(seq), last_acked_seq)` instead of bare `MAX(seq)`, which read **0** once ack-pruning had emptied a converged sender's oplog — so a rebuilt peer (watermark 0) was healed by the snapshot but never lifted by it, and the `0.1.2` empty-oplog horizon rule re-streamed the whole database on EVERY reconnect until the sender wrote again (ScadaBridge's site-a-a: 127 k rows per restart, observed on two consecutive restarts). Pinned by `WipedPeer_SnapshotLiftsItsWatermark_…`; 216/216. **ScadaBridge pins `0.3.4`** (2026-08-20; live: first restart sent one last snapshot `as_of 149`, second restart sent none); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`) — note OtOpcUa's rebuilt-node path has the same loop until it re-pins. **`0.3.3` (2026-08-20, scadaproj `122940c` — ScadaBridge arch-review P4-7; wire-compatible, no proto/schema change)** closes two latent `LwwApplier` hazards: (1) **`RegisterReplicated` refuses a table that is the parent of a row-mutating `ON DELETE` FK** (`CASCADE`/`SET NULL`/`SET DEFAULT`; `RESTRICT`/`NO ACTION` allowed) via `LocalDbRegistrationException` naming `child.column ON DELETE ` — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired child actions while capture was suppressed, a silent divergence for any future FK'd table (none of ScadaBridge's ten or OtOpcUa's four has an FK today; registration-time check only, since SQLite can't add an FK to an existing table); (2) the per-entry apply (delete-or-upsert + ledger write) is **atomic under a `SAVEPOINT`** — a fault between the two statements left a deleted row with a live ledger that `SnapshotStreamer` warns-and-skips forever; poison-catch semantics preserved. 213/213 tests (+9, failure injected with a `RAISE(ABORT)` trigger on the ledger, no production seam). **ScadaBridge pins `0.3.3`** (2026-08-20); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`). **`0.3.2` (2026-08-19, scadaproj `f62b017` — ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no proto/schema change)**: (1) the sync pump **faults a live session when a cap-prune invalidates its position** (oplog min-seq horizon check before each read batch + on the batch head; new public `LocalDbOplogGapException` rides the 0.3.1 fault teardown) — previously a slow-but-alive peer silently streamed past the pruned gap and the pair diverged until the next reconnect; (2) **client h2 keepalive pings** via new validated `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (60s/20s, `Always` policy); (3) the server single-stream guard is **newest-wins with an awaited handover** (`TakeoverTimeout` 10s) — a fresh authenticated `Sync` displaces a dead incumbent instead of being refused `AlreadyExists` forever by a half-open link whose zombie side reports connected. 204/204 tests. **ScadaBridge pins `0.3.2`** (2026-08-19, adding Kestrel h2 keepalive on its site gRPC listener as the server half); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`). **`0.3.1` (2026-08-18, scadaproj `3e5a5fd`) flushes a faulting session's control lane before its writer is cancelled** — on a handshake-phase fault (i.e. a schema-mismatch rejection) `RunAsync`'s teardown cancelled the writer while the session's OWN Handshake could still be sitting in the lane (the enqueue→receive→validate→fault path has no yields while the writer's wakeup waits in the thread-pool queue; a pinning test lost the race 39/40), and mismatch detection is symmetric — the peer computes its fail-closed verdict FROM that Handshake — so the healthy peer misread a permanent schema mismatch as a generic disconnect and reconnect-looped at the fast backoff instead of the classified `FailedPrecondition`→LogError+max-backoff path. Teardown now completes both lanes so the writer drains what is already enqueued (2 s deferred cancel as the wedged-transport backstop); caller-initiated shutdown is not delayed, **no wire or config change** — an older peer simply receives the Handshake it was always owed. Closes ScadaBridge deferred-register row 41 (its flaky `Handshake_DigestMismatch_FailsClosed` reproduced 2/6 pre-fix under load, 6/6 green post-fix; the other named flake, `BatchBudgetTests.Pump_RowLargerThanTheBudget…`, was a harness defect — session faults hidden as opaque 15 s timeouts — fixed with a run-observing `NextAsync`). **ScadaBridge re-pinned through `0.3.1` to `0.3.2` (2026-08-19); OtOpcUa remained `0.2.1` (until its 2026-09-03 re-pin to `0.3.4`).** **`0.3.0` (2026-08-15) makes the snapshot-resync flag PER-TABLE** (branch `localdb-per-table-snapshot`; the residual `0.2.1` explicitly deferred). On-disk bookkeeping schema goes to **v3** — a new `__localdb_snapshot_state` table, upgraded in place on open (the transition is a version stamp only: the shared DDL creates the table with `IF NOT EXISTS` on every open, and empty is the correct v3 state for a file that never baselined). **Wire-compatible with `0.1.x`/`0.2.x`, and `lib_schema_version` deliberately STAYS `1`** because `ValidateHandshake` compares it fail-closed for equality — bumping it would stop mixed pairs syncing outright, so the extension is negotiated by **capability**: `Handshake.supports_partial_snapshot` (new field 5) and `SnapshotBegin.tables` (new field 2, where **empty == all tables**, which is exactly what a pre-`0.3.0` sender emits and a pre-`0.3.0` receiver assumes). `RegisterReplicated(table, baselineExistingRows: true)` now flags **only that table**, so a snapshot streams only the flagged tables; the database-wide `__localdb_peer_state.needs_snapshot` survives unchanged for the cases that genuinely reach every table (oplog cap prune, impossible peer ack). **A peer that does not advertise the capability is served a FULL snapshot** — a scoped one would make it advance `last_applied_remote_seq` to the snapshot's as-of and silently discard every delta below it for the tables left out. For the same reason **a partial snapshot moves no watermark on either side**: the receiver keeps `last_applied_remote_seq` and the sender keeps `_sentThruSeq`, so the overlapping deltas are still pumped and merged by LWW. Crash safety: snapshot debt now clears on the receiver's new **`SnapshotAck`** (exchanged only between `0.3.0` peers) rather than on enqueue, so an ungraceful exit mid-snapshot re-streams the same scope on reconnect; against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue is kept, there being no ack to wait for. Composes with `0.2.1`'s bounded 64-message inbox and out-of-handshake snapshot (the ack rides the unbounded CONTROL lane, like `DeltaAck`). 199/199 tests, including a single-table-baseline scope test, a mixed-version interop test (old peer ⇒ full snapshot), an ack-gated-debt test and a v2→v3 upgrade test. **ScadaBridge pins `0.3.0`**; `Host/SiteLocalDbSetup.cs` needed no restructuring — it already calls `RegisterReplicated` once per table, which is what per-table flagging keys off, so adding an eleventh replicated table to an already-replicating site now streams that one table instead of all eleven. **`0.2.1` (2026-08-14) closes two adversarial-review findings against `0.2.0`** (same branch `localdb-perf-remediation`; wire-compatible, no proto/`lib_schema_version`/on-disk-schema change): **`DeregisterReplicated` now flushes the HLC's durable crash anchor inside its own transaction, before the prune** — clock recovery is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)` and `hlc_high_water` was written only by `Dispose`, so on a file whose tables had all been deregistered an ungraceful exit left the clock nothing to recover from and it could re-issue stamps below HLCs peers already held (LWW then discards them as older); and **the gRPC sync inbox is bounded at 64 messages with wait-mode backpressure** instead of unbounded, capping inbound memory at ~`64 × MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a **mutual full snapshot** (the snapshot flag is per-DATABASE and both sides of a late-enabled pair baseline unconditionally), previously buffered whole in memory on both nodes at once. Bounding required moving the snapshot a node owes **out of the handshake** to run alongside a live receive loop; the old ordering left neither side draining its inbox while sending and deadlocks under a bounded transport (pinned by a capacity-1 mutual-snapshot test). Per-table `needs_snapshot` was considered and **deferred** at `0.2.1` — it needs an on-disk schema change; the cost was instead documented (trim high-volume tables like event logs before enabling replication late). **That residual is closed by `0.3.0`, above.** 188/188 tests at `0.2.1`. **`0.2.0` (2026-08-14) is the performance + lifecycle release from the ScadaBridge architectural review (WP3.3, branch `localdb-perf-remediation`)** — wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump); the on-disk bookkeeping schema goes to **v2**, upgraded in place on open. Six changes: **`DeregisterReplicated(table)`** drops a table's capture triggers and prunes its oplog + row-version rows (idempotent, and deliberately usable on triggers left by an EARLIER build — there was previously no way to stop capturing once a file had been registered); **`RegisterReplicated(table, baselineExistingRows: true)`** closes the late-opt-in hole, seeding the row-version ledger for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flagging a snapshot resync, because the snapshot streamer pages from that ledger and CDC never wrote pre-registration rows there — a node that started replicating LATE never sent them, silently and forever (the two APIs are a designed pair: the ledger prune is safe only because re-registration can re-baseline; deregistration must still be **symmetric**, which the fail-closed registered-table digest handshake enforces loudly); **byte-budget batching** via `MaxBatchBytes` (default 2 MB) with `MaxBatchSize` demoted to a secondary row cap, plus inbound delta/ack coalescing; **O(1) backlog depth** (`MAX(seq) − MAX(last_acked_seq, MIN(seq) − 1)`) replacing a `COUNT(*)` over the whole backlog that ran per maintenance tick AND per metrics scrape; `LwwApplier` prepare-once-per-statement; and the unused `__localdb_oplog_hlc` index dropped. **`0.1.2` (2026-07-21) fixes snapshot resync for a WIPED peer**: a converged pair prunes every oplog row on ack, and the handshake read an empty oplog as "no gap possible" — so the steady state of a healthy pair was the one state from which a rebuilt node (lost volume, re-imaged host) could never be back-filled; it came back empty and stayed empty until the next deploy. The gap is now measured against `last_acked_seq` when the oplog is empty. Found by the OtOpcUa Phase 1 live gate (check 4) as a documented limitation, then closed. **`0.1.3` (2026-07-21) closes the other half, which only became REACHABLE once back-fill worked**: the rebuilt peer's OWN writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq space and a rebuilt peer numbers from 1, so the healthy node's stale watermark made the sending side skip its whole oplog — the pair looked converged right up until the moment only the rebuilt node applied a deploy. Fixed by resetting the inbound watermark on a peer-node-id change (NOT last_acked_seq — that is what 0.1.2 keys off) plus rejecting a peer claim beyond anything we ever produced. Live-gated on the OtOpcUa docker-dev rig: `OtOpcUa/docs/plans/2026-07-21-localdb-followups-live-gate.md`. **`0.1.1` fixes a latent boot defect**: the library never created the parent directory of `LocalDb:Path` and `SqliteLocalDb` opens the file eagerly, so a missing directory was a hard boot failure (`SQLite Error 14`) — routine given the relative `./data/…` defaults, and masked on docker only by the volume mount. ScadaBridge had worked around it app-side; the fix moved into the library and the shim is deleted. **Phase 1** (merged in PR #23): `OperationTracking` + `site_events` consolidated into one `LocalDb:Path` database, fail-closed bearer interceptor on the sync endpoint, `ISyncStatus` health bridge, one-time legacy migrator. **Phase 2** (merged in the same PR): adds `sf_messages` + 7 site config tables (10 replicated in total) and **deletes** ScadaBridge's bespoke `SiteReplicationActor`, its messages, StoreAndForward's `ReplicationService`, and `ReplaceAllAsync` — including the whole **notify-and-fetch** exchange, so a standby now makes NO config fetch on deploy (`SiteReconciliationActor`'s startup fetch survives; different path). `notification_lists`/`smtp_configurations` are created but deliberately NEVER registered — permanently empty, and a channel for them would carry plaintext SMTP passwords. **Live gate PASS, all 10 checks** (`ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md`): byte-identical config rows with identical HLC + origin node id, standby zero-fetch, 3-table cascade delete converging with tombstones on both nodes, site failover with zero duplicate ids, clean both-nodes-together restart. **Replication remains default-OFF**, enabled on the docker rig's site-a pair only (site-b/site-c unreplicated as the default-OFF pin); **no production deployment has it enabled.** **Operational constraints:** a site pair must be stopped/started TOGETHER (rolling upgrade unsupported since the `SfBufferSnapshot` compat handler was deleted), a node offline beyond `TombstoneRetention` (7 d) can resurrect deleted rows (→ #16), and — before `0.2.0` — `MaxBatchSize` batched by ROW COUNT against a 4 MB gRPC cap, which is why the rig pinned it to **16** (the 500 default would allow ~35 MB). **That pin is retired**: ScadaBridge consumed `0.2.0` on 2026-08-14 (branch `arch-review-remediation`), removed `MaxBatchSize` from both `docker/site-a` node configs and left `MaxBatchBytes` at its 2 MB default. The same consumption made `Host/SiteLocalDbSetup.cs` self-healing in both directions: an unreplicated node (site-b/site-c) now **deregisters** all ten tables at boot, dropping stale triggers an earlier build installed, and a replicating node registers with **`baselineExistingRows: true`**, so turning replication ON for a site that had been running without it finally converges on the rows already in the file rather than only on writes made after the restart. **Also adopted by OtOpcUa — Phase 1 COMPLETE and MERGED** to `master` 2026-07-21 (merge `c957db52`), and **Phase 2 (alarm store-and-forward on LocalDb) EXECUTED, live-gated and MERGED** to `master` (merge `06b5d81b`; the earlier "designed but NOT executed" claim was stale — corrected 2026-08-07; residual: live-gate checks 3+4 were never re-run on the post-mesh-Phase-6 topology): a pair-local consolidated SQLite cache of the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention) that lets a driver node **boot from cache when central SQL Server is unreachable**, registered in the `hasDriver` branch only (admin-only nodes never register it), with the same default-OFF fail-closed replication. Its live gate passed 8/8 and caught four defects offline tests missed. **Consumers: ScadaBridge (Phases 1+2) and OtOpcUa (Phases 1+2)** — **both pin `0.3.4`** as of 2026-09-03 (ScadaBridge since 2026-08-20; OtOpcUa `b8b4c18f`, a pure pin bump with tests unchanged versus its baseline). Earlier: **ScadaBridge pinned `0.3.0`** (2026-08-15, residual R1; it consumed `0.2.0` on 2026-08-14 under arch-review WP3.3) and **OtOpcUa pinned `0.2.1`** (2026-08-15, residual R4, merged to its `master` @ `5a7aae14` — a pure pin bump, no code change; it also corrected its `MaxBatchSize = 16` rationale, kept at 16 as the DB read-page bound). The remaining 0.2.1-vs-0.3.0 skew is supported: every release is wire-compatible (`0.3.0` negotiates its scoped snapshots by capability and falls back to a full snapshot for an older peer) and the schema v2/v3 upgrades are local-file state. **OtOpcUa closed the ScadaBridge deferred-register row 39 gap on 2026-08-18** (branch `perf/followups-r3`, commit `3e62f161`): `LocalDbSetup.OnReady` no longer registers all four tables unconditionally — registration follows one predicate, `LocalDb:Replication:PeerAddress` **OR** `:ApiKey` non-blank (the OR because only the dialling half of a pair sets `PeerAddress`; `SyncListenPort` is deliberately excluded, a port with no key being fail-closed). A configured node registers with **`baselineExistingRows: true`**, so a late replication enable finally converges on the rows already in the file; an unconfigured node **deregisters** at boot, dropping capture triggers an earlier build installed and logging once only when it actually cleaned something. Same shape ScadaBridge landed under arch-review WP3.3, written against OtOpcUa's own seams. OtOpcUa re-pinned to `0.3.4` on 2026-09-03 (`b8b4c18f`), so its snapshot flag is per-table too and the earlier per-DATABASE caveat (a late baseline streamed every registered table in full) no longer applies. Flipping the predicate is a both-nodes-together change (fail-closed registered-table digest handshake). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + [`docs/plans/2026-07-19-scadabridge-localdb-design.md`](docs/plans/2026-07-19-scadabridge-localdb-design.md) + [`docs/plans/2026-07-20-otopcua-localdb-design.md`](docs/plans/2026-07-20-otopcua-localdb-design.md) (both adoption designs live here); ScadaBridge holds the Phase 1 plan + Phase 2 gate under its own `docs/plans/`, OtOpcUa its two phase plans + two live gates | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | diff --git a/ZB.MOM.WW.Overview/CLAUDE.md b/ZB.MOM.WW.Overview/CLAUDE.md index e425bfa..c809b1e 100644 --- a/ZB.MOM.WW.Overview/CLAUDE.md +++ b/ZB.MOM.WW.Overview/CLAUDE.md @@ -2,6 +2,8 @@ The hosted family dashboard (Blazor Server). Build/test from this directory: `dotnet build ZB.MOM.WW.Overview.slnx`, `dotnet test ZB.MOM.WW.Overview.slnx` (fully offline). +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #14 (re-run the two OtOpcUa-blocked Phase 4 acceptance checks). The history below is a record, not a status tracker; file new follow-ups as issues. + ## Index history (migrated from the scadaproj root CLAUDE.md) > Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.** @@ -15,7 +17,7 @@ the `ZB.MOM.WW.Health` `0.2.0`/`0.3.0` per-entry `data` work exists to feed. **N read-only, holds no credentials** — a requirement, not an omission; it references no Auth/Audit/ Secrets package. Design + impl plan: `docs/plans/2026-07-22-overview-dashboard-{design,impl-plan}.md` (ledger 19/20; the open task is Phase 4 live acceptance, recorded in `41013dc` as 7 of 9 design §9 -checks passing, 2 blocked on an OtOpcUa rig redeploy). +checks passing, 2 blocked on an OtOpcUa rig redeploy → #14). The `Applications` registry ships **empty** on purpose: a deployment supplying none fails startup naming the key rather than booting an empty dashboard. Per-deployment overlays are committed — diff --git a/ZB.MOM.WW.Secrets/CLAUDE.md b/ZB.MOM.WW.Secrets/CLAUDE.md index 9fa8a30..9dbc160 100644 --- a/ZB.MOM.WW.Secrets/CLAUDE.md +++ b/ZB.MOM.WW.Secrets/CLAUDE.md @@ -2,6 +2,8 @@ Shared encrypted secret store + `${secret:}` resolution for the **ZB.MOM.WW SCADA family** (5 packages + the `secret` CLI; replicators for SqlServer / AkkaDotNet / Grpc). Build/test from this directory: `dotnet test`. Component spec + gaps: `../components/secrets/`. Operator runbooks under `docs/operations/`. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #7 (ignitionoee router pins), #8 (Replicator.Grpc / `0.6.3` fate), #17 (stale status text), #18 (migrator retry jitter). The history below is a record, not a status tracker; file new follow-ups as issues. + ## Index history (migrated from the scadaproj root CLAUDE.md) > Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.** @@ -10,4 +12,4 @@ Shared encrypted secret store + `${secret:}` resolution for the **ZB.MOM.WW SCAD | Component | Status | Goal | Design | Implementation | |---|---|---|---|---| -| Secrets (encrypted store + `${secret:}` resolution) | Built (current lib version **`0.5.1`**, **published to the Gitea feed** — the `0.1.x`→`0.5.1` progression is narrated below; **ScadaBridge pins `0.5.1`** (it consumes the new Grpc replicator), the other three apps still pin the `0.2.3` libraries, and the `0.3.0` interactive-console delta ships in the CLI tool; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; **production topologies DECIDED 2026-08-07 ([scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with **one fleet-wide production KEK via appsettings/env, never committed**; ScadaBridge = a **new `ZB.MOM.WW.Secrets.Replicator.Grpc` central-hosted hub (pull-only — updates originate at central only, by design)**, because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. **That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07).** The `Grpc` replicator is **BUILT and PUBLISHED** — family `0.4.0` (`scadaproj main @ 321ab22`) then `0.4.1` (`c86cead`, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced **at the contract** (`secrets_hub.v1` has exactly two RPCs, `GetManifest`/`GetSecrets`, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a `GetSecrets` batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. **ScadaBridge is wired both ends and live-gated PASS 4/4** (`ScadaBridge main @ f6c3f7c5`, merged + pushed; gate `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`): the node role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub, mode key `Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}, hub mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (`lmxopcua master @ 1bd79c8e`, gate `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan `docs/plans/2026-08-07-secrets-central-shared-sql-store.md`) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' `ISecretStore` is **ONE shared SQL-Server store** (`AddZbSecretsSqlServerStore`; central now REQUIRES `Secrets:SqlServer:ConnectionString`, fail-closed — blank refused naming the key, `${secret:}` refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-`central-b` defect is unreproducible. The site client gained ordered **`FallbackEndpoints`** with sticky preference (family `0.5.0`, `main @ 7a2980e`→`e62b4a5`): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **`0.5.1`** (`main @ 31ca940`): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now `0.6.2`** (2026-08-11, `main @ c5e894d`; 0.6.0 `944595a`, 0.6.1 `cbfe296` — all three published and restore-verified). **`0.6.2` is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.** `Secrets:SqlitePath` is now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator` using two rules in the new public `SecretsStorePathRules`, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — `C:\Windows\System32` for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store into `src/` and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where `…\MxGateway\Server\gateway-auth.db` left with `Server\` and an empty store was created at the same path, so **no gRPC consumer could authenticate for two days, silently**). The rules **reject rather than relocate**, and are deliberately **not** exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. **BREAKING for configuration, not API:** `SecretsOptions.SqlitePath` now defaults to `SecretsOptions.DefaultSqlitePath` (absolute, `ZB.MOM.WW/secrets.db` under **`LocalApplicationData`**), so **an app configuring a relative path will not boot** — remove the key or set an absolute one. Per-user, not `CommonApplicationData`, on purpose: that is `/usr/share` on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. **The migration edge was closed by evidence before the default moved** — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in **service env**, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: `AddZbSecrets` validator registration is now idempotent — `TryAddEnumerable` de-duplicates on implementation *type*, which a factory descriptor does not carry, so it **threw** on the second call rather than skipping (mxgw calls `AddZbSecrets` twice; the Akka replicator calls it internally). 500 tests pass. **Rollout status:** pre-bump pins were **ScadaBridge `0.5.1`, the other three `0.2.3`** — so three of the four are four-minor jumps, not re-pins. **mxgw MERGED + PUSHED 0.6.0** (`mxaccessgw origin/main @ 9bc70d1`, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — **but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up.** HistorianGateway has adopted **0.6.2** (4-arg content-root call site included) and re-verified on the exact probe that found the defect (in-content-root path → aborts with **no `.db`/`-wal`/`-shm` left behind**); **merged to `main` + pushed 2026-08-13 (`e42ff31`)**. **OtOpcUa has adopted 0.6.2** (`b037cc7a`, superseding `835e35a8`; unpushed) with the heavy part discharged by **live evidence rather than changelog trust**: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves `Replicator.AkkaDotNet`, whose pair-local topology was live-gated on `0.2.3` — a re-gate may be the real cost there. **`0.6.1` closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target `0.6.1`, never `0.6.0`.** The validator resolves the content root from `IHostEnvironment` and skipped the under-content-root rule when it was absent — but an app that expands `${secret:}` references **before building its host** composes the subsystem in a throwaway `ServiceCollection` with no `IHostEnvironment` in it and then **migrates the store from that container**. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator **created the database**, and the boot failed a second later once the real host registered `IHostEnvironment`. Net effect: a rejected boot that still left an empty store with `-wal`/`-shm` **at the rejected path** — the exact artifact that made the MxGateway outage read as *the database is there, it's just empty*. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating *no `IHostEnvironment` registered* with *no content root exists*. Fix is an additive overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`**; the 3-arg overload is unchanged. **THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix.** The exposure is **pre-host-expander-shaped, not app-shaped**: an app with a correctly configured path shows *no symptom* and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. **mxgw and HistorianGateway are CONFIRMED affected** (mxgw `GatewayApplication.cs:82`, HG `Program.cs:90` — both `new ServiceCollection().AddZbSecrets(...)` followed by `SqliteSecretsStoreMigrator.MigrateAsync`); **OtOpcUa is CONFIRMED affected too** (`Program.cs`, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. **Containers: the rule deliberately over-rejects.** A store on a volume mounted INSIDE the content root (`/app/data/…` with `/app` the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and **no opt-out flag** — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an *improvement* rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so **the same path would pass on one host and fail on another** — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used `/data`); the file sits at the volume root either way, so seeded stores carry over. **`NU1102` after a bump is a cached-index artifact, not the feed** — and it **recurs on every publish**, consistently stranding `Replicator.AkkaDotNet` at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. `dotnet nuget locals http-cache --clear`. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a **do-not-simplify-back-to-the-3-arg-overload** comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. **Verify with positive controls, not a clean boot** — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the **old relative default is orphaned, not migrated, and nothing signals it** — deleting the key, rather than leaving a relative value, is what makes the new default apply. `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` `0.1.0` for the shared `OptionsValidatorBase`/`ValidationBuilder` vocabulary. **`Secrets.Ui` is byte-identical between 0.2.3 and 0.6.0** (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned `0.5.1` (`main` `68f812ea`→`4d7f09d5`→`e697477c`, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to **ONE run against central's shared SQL store** (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, **[ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)**, is **FIXED+CLOSED 2026-08-08 (`ScadaBridge main @ e9c412e5`)**: the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays `running`, `restart: unless-stopped` never fires) was root-caused to **dotnet running as container PID 1** — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so `abort()` could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an `AppDomain.UnhandledException` handler registered before anything can throw exits `134` via the `exit()` syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus `init: true` on all 8 compose nodes for crash paths that bypass the managed event (`Environment.FailFast`, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with `RestartCount` climbing under `unless-stopped`, trace intact, proven with and without init; rig redeployed healthy with `docker-init` as PID 1. The CoordinatedShutdown no-`Environment.Exit` source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's `run-by-clr-shutdown-hook`). **Operator note: production overlays running this image under their own compose should also add `init: true`.** Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; **all four apps are Bootstrap hosts** — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: `src/ZB.MOM.WW.MxGateway.Server/libman.json` pins `bootstrap@5.3.3` into `wwwroot/lib/bootstrap/` and `App.razor:7` links it ahead of ``. What mxgw's CLAUDE.md forbids is Blazor *component libraries* (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `