From 5da4ab3ebdd5984efde61c7df2f06dc6bb78a45e Mon Sep 17 00:00:00 2001
From: Joseph Doherty <dohejw01@gmail.com>
Date: Thu, 13 Aug 2026 08:58:09 -0400
Subject: [PATCH] =?UTF-8?q?docs(index):=20HistorianGateway=20UI-sweep=20br?=
 =?UTF-8?q?anch=20merged=20to=20main=20(e42ff31)=20=E2=80=94=20Theme=200.4?=
 =?UTF-8?q?.1=20+=20Health=200.3.0=20+=20Secrets=200.6.2=20+=20Auth=200.2.?=
 =?UTF-8?q?1=20now=20on=20its=20default?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

---
 CLAUDE.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index 408830f..64a63f3 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -141,14 +141,14 @@ each project's **code-verified current state**, and the **gaps** between. See
 
 | Component | Status | Goal | Design | Implementation |
 |---|---|---|---|---|
-| Auth (login / identity / authz) | Adopted (**feed carries `0.2.1`, and ALL FOUR apps pin `0.2.1` as of 2026-08-13**: OtOpcUa `master @ 08e33fc7`, ScadaBridge `main @ d28824d9`, MxAccessGateway `main @ 1cb14d2`, HistorianGateway `e42ff31` on its `dashboard-ui-sweep-theme-0.4.1` branch — each built + tested green, all PUSHED to origin 2026-08-13). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`; the app-level exposure is closed by the pin bumps. **`FallbackServers` is live-gated** via the local redundant GLAuth pair ([`infra/glauth-redundant/`](infra/glauth-redundant/), primary :3893 + backup :3894 on the dev Mac): HistorianGateway dashboard drill PASS 2026-08-13 (failover, stickiness, walk-back). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
-| UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.4.1`). **`0.4.0` (2026-08-11) upstreamed the Bootstrap `.btn`/`.btn-sm` sizing block** that all **five** consumers had independently written into their own stylesheets (Bootstrap hard-codes `--bs-btn-font-size: 1rem`, oversized against the kit's 0.9rem body) — sized by `--bs-btn-*` variable override **only**, never by redeclaring box properties, because `.btn-group`'s seam/radius machinery reads those same variables and a literal `font-size` at equal specificity silently flattens `.btn-sm` into a padding-only difference (a real defect found in mxgw's local copy). Same release added the `rail-btn-block` full-width rail-foot modifier; **`0.4.1` fixes it** — 0.4.0 shipped `width: auto`, which fills for an `<a>` but shrink-wraps a `<button>` (measured 114px short in the same rail), now `calc(100% - 1.2rem)` to clear `.rail-btn`'s side margins. **Consumers are five, not four** — the ignitionoee router joined. **All five consumers are on `0.4.1` and all five are PUSHED** (2026-08-11): ignitionoee `main @ 38aa5c1`, OtOpcUa `master @ e82881c1`, mxgw `main @ 0a9715d`. The two whose sweeps were not deploy-ready pushed to **branches**, deliberately leaving their default branch untouched: HistorianGateway `dashboard-ui-sweep-theme-0.4.1 @ df7b879` (`main` stays `e9c31da`) and ScadaBridge `ui/central-ui-cleanup-sweep @ 9e243493` (`main` was then `b6f383a2`). **ScadaBridge's caveat is now CLOSED and its sweep is merged, deployed and live** (2026-08-11): the branch landed on `main`, and the two follow-ups `d14e0ee4` + `3eb7df74` fixed exactly the four detail-modal surfaces the caveat named — the modal is now gated on **user intent (the held id)**, never on the row still resolving, so a background refresh that drops the row leaves it mounted and says why the summary is missing instead of unmounting the subtree and disposing Close's own handler mid-click. That disposal *was* the intermittent `GetRequiredEventBindingEntry`, and `NotificationReportDetailModalTests.Modal_StaysOpen_WhenItsRowLeavesThePage` now pins it. **Deployed to wonder-app-vd03: OtOpcUa `e82881c1`, and ScadaBridge `3eb7df74` (2026-08-11 — theme.css/layout.css/nav-state.js served from `_content/ZB.MOM.WW.Theme` and referenced by the pages; all six changed routes 200). HistorianGateway is still `e9c31da` = 0.3.1 — its 0.4.1 remains on the branch.** **On bump, TRIM the local `.btn` block, don't delete it** — four of the five were pure sizing, but mxgw's also carried `border-radius`/`font-weight`/`white-space` that a blind delete would have dropped. | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
+| Auth (login / identity / authz) | Adopted (**feed carries `0.2.1`, and ALL FOUR apps pin `0.2.1` as of 2026-08-13**: OtOpcUa `master @ 08e33fc7`, ScadaBridge `main @ d28824d9`, MxAccessGateway `main @ 1cb14d2`, HistorianGateway `e42ff31`, **merged to `main`** — each built + tested green, all PUSHED to origin 2026-08-13). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`; the app-level exposure is closed by the pin bumps. **`FallbackServers` is live-gated** via the local redundant GLAuth pair ([`infra/glauth-redundant/`](infra/glauth-redundant/), primary :3893 + backup :3894 on the dev Mac): HistorianGateway dashboard drill PASS 2026-08-13 (failover, stickiness, walk-back). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
+| UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.4.1`). **`0.4.0` (2026-08-11) upstreamed the Bootstrap `.btn`/`.btn-sm` sizing block** that all **five** consumers had independently written into their own stylesheets (Bootstrap hard-codes `--bs-btn-font-size: 1rem`, oversized against the kit's 0.9rem body) — sized by `--bs-btn-*` variable override **only**, never by redeclaring box properties, because `.btn-group`'s seam/radius machinery reads those same variables and a literal `font-size` at equal specificity silently flattens `.btn-sm` into a padding-only difference (a real defect found in mxgw's local copy). Same release added the `rail-btn-block` full-width rail-foot modifier; **`0.4.1` fixes it** — 0.4.0 shipped `width: auto`, which fills for an `<a>` but shrink-wraps a `<button>` (measured 114px short in the same rail), now `calc(100% - 1.2rem)` to clear `.rail-btn`'s side margins. **Consumers are five, not four** — the ignitionoee router joined. **All five consumers are on `0.4.1` and all five are PUSHED** (2026-08-11): ignitionoee `main @ 38aa5c1`, OtOpcUa `master @ e82881c1`, mxgw `main @ 0a9715d`. The two whose sweeps were not deploy-ready pushed to **branches**, deliberately leaving their default branch untouched: HistorianGateway `dashboard-ui-sweep-theme-0.4.1` — **MERGED to `main` (ff to `e42ff31`) + pushed 2026-08-13**, carrying Theme 0.4.1 + Health 0.3.0 + Secrets 0.6.2 + Auth 0.2.1 and ScadaBridge `ui/central-ui-cleanup-sweep @ 9e243493` (`main` was then `b6f383a2`). **ScadaBridge's caveat is now CLOSED and its sweep is merged, deployed and live** (2026-08-11): the branch landed on `main`, and the two follow-ups `d14e0ee4` + `3eb7df74` fixed exactly the four detail-modal surfaces the caveat named — the modal is now gated on **user intent (the held id)**, never on the row still resolving, so a background refresh that drops the row leaves it mounted and says why the summary is missing instead of unmounting the subtree and disposing Close's own handler mid-click. That disposal *was* the intermittent `GetRequiredEventBindingEntry`, and `NotificationReportDetailModalTests.Modal_StaysOpen_WhenItsRowLeavesThePage` now pins it. **Deployed to wonder-app-vd03: OtOpcUa `e82881c1`, and ScadaBridge `3eb7df74` (2026-08-11 — theme.css/layout.css/nav-state.js served from `_content/ZB.MOM.WW.Theme` and referenced by the pages; all six changed routes 200). HistorianGateway on wonder still runs `e9c31da` = Theme 0.3.1 — its 0.4.1 is now on `main` (merged 2026-08-13) but not yet deployed to the box.** **On bump, TRIM the local `.btn` block, don't delete it** — four of the five were pure sizing, but mxgw's also carried `border-radius`/`font-weight`/`white-space` that a blind delete would have dropped. | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
 | Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`–`0.3.0`, current lib version **`0.3.0`**; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. **Both app branches are MERGED and PUSHED** (verified 2026-08-07): OtOpcUa `88adec04` is on `origin/master`; ScadaBridge merged via `b3dc17a5` (2026-07-24) on `origin/main`, private-copy deletions included. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
 | Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
 | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
 | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
 | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
-| Secrets (encrypted store + `${secret:}` resolution) | Built (current lib version **`0.5.1`**, **published to the Gitea feed** — the `0.1.x`→`0.5.1` progression is narrated below; **ScadaBridge pins `0.5.1`** (it consumes the new Grpc replicator), the other three apps still pin the `0.2.3` libraries, and the `0.3.0` interactive-console delta ships in the CLI tool; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; **production topologies DECIDED 2026-08-07 ([scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with **one fleet-wide production KEK via appsettings/env, never committed**; ScadaBridge = a **new `ZB.MOM.WW.Secrets.Replicator.Grpc` central-hosted hub (pull-only — updates originate at central only, by design)**, because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. **That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07).** The `Grpc` replicator is **BUILT and PUBLISHED** — family `0.4.0` (`scadaproj main @ 321ab22`) then `0.4.1` (`c86cead`, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced **at the contract** (`secrets_hub.v1` has exactly two RPCs, `GetManifest`/`GetSecrets`, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a `GetSecrets` batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. **ScadaBridge is wired both ends and live-gated PASS 4/4** (`ScadaBridge main @ f6c3f7c5`, merged + pushed; gate `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`): the node role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub, mode key `Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}, hub mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (`lmxopcua master @ 1bd79c8e`, gate `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan `docs/plans/2026-08-07-secrets-central-shared-sql-store.md`) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' `ISecretStore` is **ONE shared SQL-Server store** (`AddZbSecretsSqlServerStore`; central now REQUIRES `Secrets:SqlServer:ConnectionString`, fail-closed — blank refused naming the key, `${secret:}` refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-`central-b` defect is unreproducible. The site client gained ordered **`FallbackEndpoints`** with sticky preference (family `0.5.0`, `main @ 7a2980e`→`e62b4a5`): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **`0.5.1`** (`main @ 31ca940`): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now `0.6.2`** (2026-08-11, `main @ c5e894d`; 0.6.0 `944595a`, 0.6.1 `cbfe296` — all three published and restore-verified). **`0.6.2` is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.** `Secrets:SqlitePath` is now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator` using two rules in the new public `SecretsStorePathRules`, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — `C:\Windows\System32` for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store into `src/` and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where `…\MxGateway\Server\gateway-auth.db` left with `Server\` and an empty store was created at the same path, so **no gRPC consumer could authenticate for two days, silently**). The rules **reject rather than relocate**, and are deliberately **not** exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. **BREAKING for configuration, not API:** `SecretsOptions.SqlitePath` now defaults to `SecretsOptions.DefaultSqlitePath` (absolute, `ZB.MOM.WW/secrets.db` under **`LocalApplicationData`**), so **an app configuring a relative path will not boot** — remove the key or set an absolute one. Per-user, not `CommonApplicationData`, on purpose: that is `/usr/share` on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. **The migration edge was closed by evidence before the default moved** — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in **service env**, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: `AddZbSecrets` validator registration is now idempotent — `TryAddEnumerable` de-duplicates on implementation *type*, which a factory descriptor does not carry, so it **threw** on the second call rather than skipping (mxgw calls `AddZbSecrets` twice; the Akka replicator calls it internally). 500 tests pass. **Rollout status:** pre-bump pins were **ScadaBridge `0.5.1`, the other three `0.2.3`** — so three of the four are four-minor jumps, not re-pins. **mxgw MERGED + PUSHED 0.6.0** (`mxaccessgw origin/main @ 9bc70d1`, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — **but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up.** HistorianGateway has adopted **0.6.1** and re-verified on the exact probe that found the defect (in-content-root path → aborts with **no `.db`/`-wal`/`-shm` left behind**); its branch is pending its user. **OtOpcUa has adopted 0.6.2** (`b037cc7a`, superseding `835e35a8`; unpushed) with the heavy part discharged by **live evidence rather than changelog trust**: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves `Replicator.AkkaDotNet`, whose pair-local topology was live-gated on `0.2.3` — a re-gate may be the real cost there. **`0.6.1` closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target `0.6.1`, never `0.6.0`.** The validator resolves the content root from `IHostEnvironment` and skipped the under-content-root rule when it was absent — but an app that expands `${secret:}` references **before building its host** composes the subsystem in a throwaway `ServiceCollection` with no `IHostEnvironment` in it and then **migrates the store from that container**. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator **created the database**, and the boot failed a second later once the real host registered `IHostEnvironment`. Net effect: a rejected boot that still left an empty store with `-wal`/`-shm` **at the rejected path** — the exact artifact that made the MxGateway outage read as *the database is there, it's just empty*. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating *no `IHostEnvironment` registered* with *no content root exists*. Fix is an additive overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`**; the 3-arg overload is unchanged. **THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix.** The exposure is **pre-host-expander-shaped, not app-shaped**: an app with a correctly configured path shows *no symptom* and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. **mxgw and HistorianGateway are CONFIRMED affected** (mxgw `GatewayApplication.cs:82`, HG `Program.cs:90` — both `new ServiceCollection().AddZbSecrets(...)` followed by `SqliteSecretsStoreMigrator.MigrateAsync`); **OtOpcUa is CONFIRMED affected too** (`Program.cs`, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. **Containers: the rule deliberately over-rejects.** A store on a volume mounted INSIDE the content root (`/app/data/…` with `/app` the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and **no opt-out flag** — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an *improvement* rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so **the same path would pass on one host and fail on another** — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used `/data`); the file sits at the volume root either way, so seeded stores carry over. **`NU1102` after a bump is a cached-index artifact, not the feed** — and it **recurs on every publish**, consistently stranding `Replicator.AkkaDotNet` at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. `dotnet nuget locals http-cache --clear`. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a **do-not-simplify-back-to-the-3-arg-overload** comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. **Verify with positive controls, not a clean boot** — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the **old relative default is orphaned, not migrated, and nothing signals it** — deleting the key, rather than leaving a relative value, is what makes the new default apply. `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` `0.1.0` for the shared `OptionsValidatorBase`/`ValidationBuilder` vocabulary. **`Secrets.Ui` is byte-identical between 0.2.3 and 0.6.0** (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned `0.5.1` (`main` `68f812ea`→`4d7f09d5`→`e697477c`, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to **ONE run against central's shared SQL store** (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, **[ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)**, is **FIXED+CLOSED 2026-08-08 (`ScadaBridge main @ e9c412e5`)**: the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays `running`, `restart: unless-stopped` never fires) was root-caused to **dotnet running as container PID 1** — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so `abort()` could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an `AppDomain.UnhandledException` handler registered before anything can throw exits `134` via the `exit()` syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus `init: true` on all 8 compose nodes for crash paths that bypass the managed event (`Environment.FailFast`, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with `RestartCount` climbing under `unless-stopped`, trace intact, proven with and without init; rig redeployed healthy with `docker-init` as PID 1. The CoordinatedShutdown no-`Environment.Exit` source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's `run-by-clr-shutdown-hook`). **Operator note: production overlays running this image under their own compose should also add `init: true`.** Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; **all four apps are Bootstrap hosts** — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: `src/ZB.MOM.WW.MxGateway.Server/libman.json` pins `bootstrap@5.3.3` into `wwwroot/lib/bootstrap/` and `App.razor:7` links it ahead of `<ThemeHead />`. What mxgw's CLAUDE.md forbids is Blazor *component libraries* (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `<style>` block** (Theme tokens w/ fallbacks; testids unchanged; bUnit pins). Scoped `.razor.css` was deliberately rejected — only ScadaBridge linked a scoped-CSS bundle **at the time (2026-07-19)**, so isolation CSS would silently not load on the other 3 hosts — still the right call, and note the premise has since changed: OtOpcUa's 2026-08-10 AdminUI sweep linked its own dead bundle (`App.razor:23`), so as of 2026-08-11 two of the four link one, while mxgw and HistorianGateway have zero `*.razor.css` and emit no bundle at all (the same silent-failure class). Live-verified on ScadaBridge (bumped to `0.2.3` @ `63510778`): modal visible `display=flex` with zero JS forcing, delete flows through the real path. **Rollout COMPLETE 2026-07-19 — all four apps on `0.2.3`, pushed:** OtOpcUa (`lmxopcua master @ be87ddeb`; AdminUI 665/665 + secrets integration 8/8; docker-dev rig rebuilt, modal gate PASS live), HistorianGateway (`main @ 7452eb4`, from `0.2.0`; plus `6bc005d` **fixing the pre-existing AngleSharp NU1902 warn-as-error that had made the whole HG test suite unbuildable** — test-only pin to 1.5.2, suite now runs: 854 pass / 1 pre-existing VPN-gated live failure, histsdk failures identical to the pre-bump baseline; docker container rebuilt, modal gate PASS live), and mxgw (`main @ 8f7ee49`, from `0.2.0`; 780 pass / 45 macOS-only NamedPipe-harness failures identical before and after the bump — zero delta). **mxgw's modal gate was originally SKIPPED on the false "Bootstrap-free so never affected" reasoning, and is now PASS (2026-08-11)** — run live against a local rig: modal root class is `zb-secrets-modal`, `document.querySelector('.modal')` is null so Bootstrap's `.modal{display:none}` has nothing to bite, computed `display:flex` / `position:fixed` / z-index 1081 over a full-viewport 1080 backdrop, and `elementFromPoint` at the confirm button's centre returns the button rather than the backdrop (visible ≠ hit-testable — that probe is the one that proves usable). Cancel disarmed cleanly with no delete performed. The exemption reached the right conclusion by luck: mxgw was a Bootstrap host and *was* affected, but had already taken the `0.2.3` fix independently of the sweep. **Lesson: an exemption from a family-wide gate must re-derive the exempting property from the repo at gate time** — see [`admin_ui_cleanup.md`](admin_ui_cleanup.md) **`0.3.0` (2026-07-19, merged to `main` @ `87ffe1b`) added a menu-driven Spectre.Console interactive operator console** to the `secret` CLI: bare `secret` with no verb on a real TTY (redirected I/O or any verb → unchanged headless behavior, no-args-redirected → usage/exit-2) opens a guided session that targets a deployment's store by reading the **target app's own appsettings.json** (base + env overlay + env vars → `Secrets` section, so it hits exactly the store the app reads; recents cached at `~/.zb-secrets/recent-targets.json`, names/paths only). Eight flows: list/set/reveal/delete plus the two headline ones — **Reference audit & seed** (classifies every `${secret:NAME}` ref Ok/PresentUnverified/Missing/Tombstoned/Undecryptable/InvalidName, then seeds the gaps — deployment setup) and **KEK doctor** (per-row Ok/WrongKek/Corrupt verdicts + guided rewrap-all or lost-KEK re-set — lockout recovery), plus ciphertext-only bundle export/import. Degraded (no-KEK) sessions open metadata-only and upgrade in place via masked paste/key-file; no login (local trust); key material never a literal arg. **5 packages published at `0.3.0` to the Gitea feed** (restore-verified); the four apps still pin the `0.2.3` libraries (the console ships in the CLI tool, not the app packages). Runbook `ZB.MOM.WW.Secrets/docs/operations/interactive-console.md` + quickstart `ZB.MOM.WW.Secrets/docs/interactive-console-quickstart.md`. | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
+| Secrets (encrypted store + `${secret:}` resolution) | Built (current lib version **`0.5.1`**, **published to the Gitea feed** — the `0.1.x`→`0.5.1` progression is narrated below; **ScadaBridge pins `0.5.1`** (it consumes the new Grpc replicator), the other three apps still pin the `0.2.3` libraries, and the `0.3.0` interactive-console delta ships in the CLI tool; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; **production topologies DECIDED 2026-08-07 ([scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with **one fleet-wide production KEK via appsettings/env, never committed**; ScadaBridge = a **new `ZB.MOM.WW.Secrets.Replicator.Grpc` central-hosted hub (pull-only — updates originate at central only, by design)**, because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. **That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07).** The `Grpc` replicator is **BUILT and PUBLISHED** — family `0.4.0` (`scadaproj main @ 321ab22`) then `0.4.1` (`c86cead`, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced **at the contract** (`secrets_hub.v1` has exactly two RPCs, `GetManifest`/`GetSecrets`, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a `GetSecrets` batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. **ScadaBridge is wired both ends and live-gated PASS 4/4** (`ScadaBridge main @ f6c3f7c5`, merged + pushed; gate `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`): the node role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub, mode key `Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}, hub mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (`lmxopcua master @ 1bd79c8e`, gate `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan `docs/plans/2026-08-07-secrets-central-shared-sql-store.md`) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' `ISecretStore` is **ONE shared SQL-Server store** (`AddZbSecretsSqlServerStore`; central now REQUIRES `Secrets:SqlServer:ConnectionString`, fail-closed — blank refused naming the key, `${secret:}` refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-`central-b` defect is unreproducible. The site client gained ordered **`FallbackEndpoints`** with sticky preference (family `0.5.0`, `main @ 7a2980e`→`e62b4a5`): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **`0.5.1`** (`main @ 31ca940`): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now `0.6.2`** (2026-08-11, `main @ c5e894d`; 0.6.0 `944595a`, 0.6.1 `cbfe296` — all three published and restore-verified). **`0.6.2` is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.** `Secrets:SqlitePath` is now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator` using two rules in the new public `SecretsStorePathRules`, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — `C:\Windows\System32` for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store into `src/` and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where `…\MxGateway\Server\gateway-auth.db` left with `Server\` and an empty store was created at the same path, so **no gRPC consumer could authenticate for two days, silently**). The rules **reject rather than relocate**, and are deliberately **not** exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. **BREAKING for configuration, not API:** `SecretsOptions.SqlitePath` now defaults to `SecretsOptions.DefaultSqlitePath` (absolute, `ZB.MOM.WW/secrets.db` under **`LocalApplicationData`**), so **an app configuring a relative path will not boot** — remove the key or set an absolute one. Per-user, not `CommonApplicationData`, on purpose: that is `/usr/share` on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. **The migration edge was closed by evidence before the default moved** — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in **service env**, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: `AddZbSecrets` validator registration is now idempotent — `TryAddEnumerable` de-duplicates on implementation *type*, which a factory descriptor does not carry, so it **threw** on the second call rather than skipping (mxgw calls `AddZbSecrets` twice; the Akka replicator calls it internally). 500 tests pass. **Rollout status:** pre-bump pins were **ScadaBridge `0.5.1`, the other three `0.2.3`** — so three of the four are four-minor jumps, not re-pins. **mxgw MERGED + PUSHED 0.6.0** (`mxaccessgw origin/main @ 9bc70d1`, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — **but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up.** HistorianGateway has adopted **0.6.2** (4-arg content-root call site included) and re-verified on the exact probe that found the defect (in-content-root path → aborts with **no `.db`/`-wal`/`-shm` left behind**); **merged to `main` + pushed 2026-08-13 (`e42ff31`)**. **OtOpcUa has adopted 0.6.2** (`b037cc7a`, superseding `835e35a8`; unpushed) with the heavy part discharged by **live evidence rather than changelog trust**: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves `Replicator.AkkaDotNet`, whose pair-local topology was live-gated on `0.2.3` — a re-gate may be the real cost there. **`0.6.1` closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target `0.6.1`, never `0.6.0`.** The validator resolves the content root from `IHostEnvironment` and skipped the under-content-root rule when it was absent — but an app that expands `${secret:}` references **before building its host** composes the subsystem in a throwaway `ServiceCollection` with no `IHostEnvironment` in it and then **migrates the store from that container**. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator **created the database**, and the boot failed a second later once the real host registered `IHostEnvironment`. Net effect: a rejected boot that still left an empty store with `-wal`/`-shm` **at the rejected path** — the exact artifact that made the MxGateway outage read as *the database is there, it's just empty*. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating *no `IHostEnvironment` registered* with *no content root exists*. Fix is an additive overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`**; the 3-arg overload is unchanged. **THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix.** The exposure is **pre-host-expander-shaped, not app-shaped**: an app with a correctly configured path shows *no symptom* and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. **mxgw and HistorianGateway are CONFIRMED affected** (mxgw `GatewayApplication.cs:82`, HG `Program.cs:90` — both `new ServiceCollection().AddZbSecrets(...)` followed by `SqliteSecretsStoreMigrator.MigrateAsync`); **OtOpcUa is CONFIRMED affected too** (`Program.cs`, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. **Containers: the rule deliberately over-rejects.** A store on a volume mounted INSIDE the content root (`/app/data/…` with `/app` the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and **no opt-out flag** — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an *improvement* rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so **the same path would pass on one host and fail on another** — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used `/data`); the file sits at the volume root either way, so seeded stores carry over. **`NU1102` after a bump is a cached-index artifact, not the feed** — and it **recurs on every publish**, consistently stranding `Replicator.AkkaDotNet` at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. `dotnet nuget locals http-cache --clear`. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a **do-not-simplify-back-to-the-3-arg-overload** comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. **Verify with positive controls, not a clean boot** — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the **old relative default is orphaned, not migrated, and nothing signals it** — deleting the key, rather than leaving a relative value, is what makes the new default apply. `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` `0.1.0` for the shared `OptionsValidatorBase`/`ValidationBuilder` vocabulary. **`Secrets.Ui` is byte-identical between 0.2.3 and 0.6.0** (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned `0.5.1` (`main` `68f812ea`→`4d7f09d5`→`e697477c`, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to **ONE run against central's shared SQL store** (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, **[ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)**, is **FIXED+CLOSED 2026-08-08 (`ScadaBridge main @ e9c412e5`)**: the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays `running`, `restart: unless-stopped` never fires) was root-caused to **dotnet running as container PID 1** — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so `abort()` could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an `AppDomain.UnhandledException` handler registered before anything can throw exits `134` via the `exit()` syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus `init: true` on all 8 compose nodes for crash paths that bypass the managed event (`Environment.FailFast`, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with `RestartCount` climbing under `unless-stopped`, trace intact, proven with and without init; rig redeployed healthy with `docker-init` as PID 1. The CoordinatedShutdown no-`Environment.Exit` source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's `run-by-clr-shutdown-hook`). **Operator note: production overlays running this image under their own compose should also add `init: true`.** Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; **all four apps are Bootstrap hosts** — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: `src/ZB.MOM.WW.MxGateway.Server/libman.json` pins `bootstrap@5.3.3` into `wwwroot/lib/bootstrap/` and `App.razor:7` links it ahead of `<ThemeHead />`. What mxgw's CLAUDE.md forbids is Blazor *component libraries* (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `<style>` block** (Theme tokens w/ fallbacks; testids unchanged; bUnit pins). Scoped `.razor.css` was deliberately rejected — only ScadaBridge linked a scoped-CSS bundle **at the time (2026-07-19)**, so isolation CSS would silently not load on the other 3 hosts — still the right call, and note the premise has since changed: OtOpcUa's 2026-08-10 AdminUI sweep linked its own dead bundle (`App.razor:23`), so as of 2026-08-11 two of the four link one, while mxgw and HistorianGateway have zero `*.razor.css` and emit no bundle at all (the same silent-failure class). Live-verified on ScadaBridge (bumped to `0.2.3` @ `63510778`): modal visible `display=flex` with zero JS forcing, delete flows through the real path. **Rollout COMPLETE 2026-07-19 — all four apps on `0.2.3`, pushed:** OtOpcUa (`lmxopcua master @ be87ddeb`; AdminUI 665/665 + secrets integration 8/8; docker-dev rig rebuilt, modal gate PASS live), HistorianGateway (`main @ 7452eb4`, from `0.2.0`; plus `6bc005d` **fixing the pre-existing AngleSharp NU1902 warn-as-error that had made the whole HG test suite unbuildable** — test-only pin to 1.5.2, suite now runs: 854 pass / 1 pre-existing VPN-gated live failure, histsdk failures identical to the pre-bump baseline; docker container rebuilt, modal gate PASS live), and mxgw (`main @ 8f7ee49`, from `0.2.0`; 780 pass / 45 macOS-only NamedPipe-harness failures identical before and after the bump — zero delta). **mxgw's modal gate was originally SKIPPED on the false "Bootstrap-free so never affected" reasoning, and is now PASS (2026-08-11)** — run live against a local rig: modal root class is `zb-secrets-modal`, `document.querySelector('.modal')` is null so Bootstrap's `.modal{display:none}` has nothing to bite, computed `display:flex` / `position:fixed` / z-index 1081 over a full-viewport 1080 backdrop, and `elementFromPoint` at the confirm button's centre returns the button rather than the backdrop (visible ≠ hit-testable — that probe is the one that proves usable). Cancel disarmed cleanly with no delete performed. The exemption reached the right conclusion by luck: mxgw was a Bootstrap host and *was* affected, but had already taken the `0.2.3` fix independently of the sweep. **Lesson: an exemption from a family-wide gate must re-derive the exempting property from the repo at gate time** — see [`admin_ui_cleanup.md`](admin_ui_cleanup.md) **`0.3.0` (2026-07-19, merged to `main` @ `87ffe1b`) added a menu-driven Spectre.Console interactive operator console** to the `secret` CLI: bare `secret` with no verb on a real TTY (redirected I/O or any verb → unchanged headless behavior, no-args-redirected → usage/exit-2) opens a guided session that targets a deployment's store by reading the **target app's own appsettings.json** (base + env overlay + env vars → `Secrets` section, so it hits exactly the store the app reads; recents cached at `~/.zb-secrets/recent-targets.json`, names/paths only). Eight flows: list/set/reveal/delete plus the two headline ones — **Reference audit & seed** (classifies every `${secret:NAME}` ref Ok/PresentUnverified/Missing/Tombstoned/Undecryptable/InvalidName, then seeds the gaps — deployment setup) and **KEK doctor** (per-row Ok/WrongKek/Corrupt verdicts + guided rewrap-all or lost-KEK re-set — lockout recovery), plus ciphertext-only bundle export/import. Degraded (no-KEK) sessions open metadata-only and upgrade in place via masked paste/key-file; no login (local trust); key material never a literal arg. **5 packages published at `0.3.0` to the Gitea feed** (restore-verified); the four apps still pin the `0.2.3` libraries (the console ships in the CLI tool, not the app packages). Runbook `ZB.MOM.WW.Secrets/docs/operations/interactive-console.md` + quickstart `ZB.MOM.WW.Secrets/docs/interactive-console-quickstart.md`. | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
 | LocalDb (embedded cache + 2-node sync) | **Adopted by ScadaBridge — Phases 1 AND 2 COMPLETE and MERGED** to `main` (2026-07-20, PR #23, merge `28ca04d7`, pushed to origin; both `feat/localdb-phase*` branches fully contained and kept only as history). Libs **`0.1.3`** on the Gitea feed (3 pkgs, restore-verified; `0.1.0`–`0.1.2` also present). **`0.1.2` (2026-07-21) fixes snapshot resync for a WIPED peer**: a converged pair prunes every oplog row on ack, and the handshake read an empty oplog as "no gap possible" — so the steady state of a healthy pair was the one state from which a rebuilt node (lost volume, re-imaged host) could never be back-filled; it came back empty and stayed empty until the next deploy. The gap is now measured against `last_acked_seq` when the oplog is empty. Found by the OtOpcUa Phase 1 live gate (check 4) as a documented limitation, then closed. **`0.1.3` (2026-07-21) closes the other half, which only became REACHABLE once back-fill worked**: the rebuilt peer's OWN writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq space and a rebuilt peer numbers from 1, so the healthy node's stale watermark made the sending side skip its whole oplog — the pair looked converged right up until the moment only the rebuilt node applied a deploy. Fixed by resetting the inbound watermark on a peer-node-id change (NOT last_acked_seq — that is what 0.1.2 keys off) plus rejecting a peer claim beyond anything we ever produced. Live-gated on the OtOpcUa docker-dev rig: `OtOpcUa/docs/plans/2026-07-21-localdb-followups-live-gate.md`. **`0.1.1` fixes a latent boot defect**: the library never created the parent directory of `LocalDb:Path` and `SqliteLocalDb` opens the file eagerly, so a missing directory was a hard boot failure (`SQLite Error 14`) — routine given the relative `./data/…` defaults, and masked on docker only by the volume mount. ScadaBridge had worked around it app-side; the fix moved into the library and the shim is deleted. **Phase 1** (merged in PR #23): `OperationTracking` + `site_events` consolidated into one `LocalDb:Path` database, fail-closed bearer interceptor on the sync endpoint, `ISyncStatus` health bridge, one-time legacy migrator. **Phase 2** (merged in the same PR): adds `sf_messages` + 7 site config tables (10 replicated in total) and **deletes** ScadaBridge's bespoke `SiteReplicationActor`, its messages, StoreAndForward's `ReplicationService`, and `ReplaceAllAsync` — including the whole **notify-and-fetch** exchange, so a standby now makes NO config fetch on deploy (`SiteReconciliationActor`'s startup fetch survives; different path). `notification_lists`/`smtp_configurations` are created but deliberately NEVER registered — permanently empty, and a channel for them would carry plaintext SMTP passwords. **Live gate PASS, all 10 checks** (`ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md`): byte-identical config rows with identical HLC + origin node id, standby zero-fetch, 3-table cascade delete converging with tombstones on both nodes, site failover with zero duplicate ids, clean both-nodes-together restart. **Replication remains default-OFF**, enabled on the docker rig's site-a pair only (site-b/site-c unreplicated as the default-OFF pin); **no production deployment has it enabled.** **Operational constraints:** a site pair must be stopped/started TOGETHER (rolling upgrade unsupported since the `SfBufferSnapshot` compat handler was deleted), a node offline beyond `TombstoneRetention` (7 d) can resurrect deleted rows, and `MaxBatchSize` batches by ROW COUNT against a 4 MB gRPC cap (rig pins **16**; the 500 default would allow ~35 MB). **Also adopted by OtOpcUa — Phase 1 COMPLETE and MERGED** to `master` 2026-07-21 (merge `c957db52`), and **Phase 2 (alarm store-and-forward on LocalDb) EXECUTED, live-gated and MERGED** to `master` (merge `06b5d81b`; the earlier "designed but NOT executed" claim was stale — corrected 2026-08-07; residual: live-gate checks 3+4 were never re-run on the post-mesh-Phase-6 topology): a pair-local consolidated SQLite cache of the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention) that lets a driver node **boot from cache when central SQL Server is unreachable**, registered in the `hasDriver` branch only (admin-only nodes never register it), with the same default-OFF fail-closed replication. Its live gate passed 8/8 and caught four defects offline tests missed. **Consumers: ScadaBridge (Phases 1+2) and OtOpcUa (Phases 1+2)**, both pinned to `0.1.3` — verified 2026-07-21 (OtOpcUa Phase 2 merge verified 2026-08-07). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + [`docs/plans/2026-07-19-scadabridge-localdb-design.md`](docs/plans/2026-07-19-scadabridge-localdb-design.md) + [`docs/plans/2026-07-20-otopcua-localdb-design.md`](docs/plans/2026-07-20-otopcua-localdb-design.md) (both adoption designs live here); ScadaBridge holds the Phase 1 plan + Phase 2 gate under its own `docs/plans/`, OtOpcUa its two phase plans + two live gates | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |
 
 The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a