scadaproj/ZB.MOM.WW.Secrets/CLAUDE.md

# ZB.MOM.WW.Secrets

Shared encrypted secret store + `${secret:}` resolution for the **ZB.MOM.WW SCADA family** (5 packages + the `secret` CLI; replicators for SqlServer / AkkaDotNet / Grpc). Build/test from this directory: `dotnet test`. Component spec + gaps: `../components/secrets/`. Operator runbooks under `docs/operations/`.

## Index history (migrated from the scadaproj root CLAUDE.md)

> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**

### Component-table row as it read in the index

| Component | Status | Goal | Design | Implementation |
|---|---|---|---|---|
| Secrets (encrypted store + `${secret:}` resolution) | Built (current lib version **`0.5.1`**, **published to the Gitea feed** — the `0.1.x`→`0.5.1` progression is narrated below; **ScadaBridge pins `0.5.1`** (it consumes the new Grpc replicator), the other three apps still pin the `0.2.3` libraries, and the `0.3.0` interactive-console delta ships in the CLI tool; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; **production topologies DECIDED 2026-08-07 ([scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with **one fleet-wide production KEK via appsettings/env, never committed**; ScadaBridge = a **new `ZB.MOM.WW.Secrets.Replicator.Grpc` central-hosted hub (pull-only — updates originate at central only, by design)**, because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. **That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07).** The `Grpc` replicator is **BUILT and PUBLISHED** — family `0.4.0` (`scadaproj main @ 321ab22`) then `0.4.1` (`c86cead`, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced **at the contract** (`secrets_hub.v1` has exactly two RPCs, `GetManifest`/`GetSecrets`, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a `GetSecrets` batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. **ScadaBridge is wired both ends and live-gated PASS 4/4** (`ScadaBridge main @ f6c3f7c5`, merged + pushed; gate `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`): the node role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub, mode key `Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}, hub mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (`lmxopcua master @ 1bd79c8e`, gate `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan `docs/plans/2026-08-07-secrets-central-shared-sql-store.md`) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' `ISecretStore` is **ONE shared SQL-Server store** (`AddZbSecretsSqlServerStore`; central now REQUIRES `Secrets:SqlServer:ConnectionString`, fail-closed — blank refused naming the key, `${secret:}` refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-`central-b` defect is unreproducible. The site client gained ordered **`FallbackEndpoints`** with sticky preference (family `0.5.0`, `main @ 7a2980e`→`e62b4a5`): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **`0.5.1`** (`main @ 31ca940`): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now `0.6.2`** (2026-08-11, `main @ c5e894d`; 0.6.0 `944595a`, 0.6.1 `cbfe296` — all three published and restore-verified). **`0.6.2` is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.** `Secrets:SqlitePath` is now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator` using two rules in the new public `SecretsStorePathRules`, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — `C:\Windows\System32` for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store into `src/` and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where `…\MxGateway\Server\gateway-auth.db` left with `Server\` and an empty store was created at the same path, so **no gRPC consumer could authenticate for two days, silently**). The rules **reject rather than relocate**, and are deliberately **not** exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. **BREAKING for configuration, not API:** `SecretsOptions.SqlitePath` now defaults to `SecretsOptions.DefaultSqlitePath` (absolute, `ZB.MOM.WW/secrets.db` under **`LocalApplicationData`**), so **an app configuring a relative path will not boot** — remove the key or set an absolute one. Per-user, not `CommonApplicationData`, on purpose: that is `/usr/share` on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. **The migration edge was closed by evidence before the default moved** — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in **service env**, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: `AddZbSecrets` validator registration is now idempotent — `TryAddEnumerable` de-duplicates on implementation *type*, which a factory descriptor does not carry, so it **threw** on the second call rather than skipping (mxgw calls `AddZbSecrets` twice; the Akka replicator calls it internally). 500 tests pass. **Rollout status:** pre-bump pins were **ScadaBridge `0.5.1`, the other three `0.2.3`** — so three of the four are four-minor jumps, not re-pins. **mxgw MERGED + PUSHED 0.6.0** (`mxaccessgw origin/main @ 9bc70d1`, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — **but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up.** HistorianGateway has adopted **0.6.2** (4-arg content-root call site included) and re-verified on the exact probe that found the defect (in-content-root path → aborts with **no `.db`/`-wal`/`-shm` left behind**); **merged to `main` + pushed 2026-08-13 (`e42ff31`)**. **OtOpcUa has adopted 0.6.2** (`b037cc7a`, superseding `835e35a8`; unpushed) with the heavy part discharged by **live evidence rather than changelog trust**: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves `Replicator.AkkaDotNet`, whose pair-local topology was live-gated on `0.2.3` — a re-gate may be the real cost there. **`0.6.1` closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target `0.6.1`, never `0.6.0`.** The validator resolves the content root from `IHostEnvironment` and skipped the under-content-root rule when it was absent — but an app that expands `${secret:}` references **before building its host** composes the subsystem in a throwaway `ServiceCollection` with no `IHostEnvironment` in it and then **migrates the store from that container**. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator **created the database**, and the boot failed a second later once the real host registered `IHostEnvironment`. Net effect: a rejected boot that still left an empty store with `-wal`/`-shm` **at the rejected path** — the exact artifact that made the MxGateway outage read as *the database is there, it's just empty*. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating *no `IHostEnvironment` registered* with *no content root exists*. Fix is an additive overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`**; the 3-arg overload is unchanged. **THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix.** The exposure is **pre-host-expander-shaped, not app-shaped**: an app with a correctly configured path shows *no symptom* and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. **mxgw and HistorianGateway are CONFIRMED affected** (mxgw `GatewayApplication.cs:82`, HG `Program.cs:90` — both `new ServiceCollection().AddZbSecrets(...)` followed by `SqliteSecretsStoreMigrator.MigrateAsync`); **OtOpcUa is CONFIRMED affected too** (`Program.cs`, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. **Containers: the rule deliberately over-rejects.** A store on a volume mounted INSIDE the content root (`/app/data/…` with `/app` the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and **no opt-out flag** — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an *improvement* rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so **the same path would pass on one host and fail on another** — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used `/data`); the file sits at the volume root either way, so seeded stores carry over. **`NU1102` after a bump is a cached-index artifact, not the feed** — and it **recurs on every publish**, consistently stranding `Replicator.AkkaDotNet` at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. `dotnet nuget locals http-cache --clear`. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a **do-not-simplify-back-to-the-3-arg-overload** comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. **Verify with positive controls, not a clean boot** — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the **old relative default is orphaned, not migrated, and nothing signals it** — deleting the key, rather than leaving a relative value, is what makes the new default apply. `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` `0.1.0` for the shared `OptionsValidatorBase`/`ValidationBuilder` vocabulary. **`Secrets.Ui` is byte-identical between 0.2.3 and 0.6.0** (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned `0.5.1` (`main` `68f812ea`→`4d7f09d5`→`e697477c`, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to **ONE run against central's shared SQL store** (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, **[ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)**, is **FIXED+CLOSED 2026-08-08 (`ScadaBridge main @ e9c412e5`)**: the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays `running`, `restart: unless-stopped` never fires) was root-caused to **dotnet running as container PID 1** — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so `abort()` could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an `AppDomain.UnhandledException` handler registered before anything can throw exits `134` via the `exit()` syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus `init: true` on all 8 compose nodes for crash paths that bypass the managed event (`Environment.FailFast`, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with `RestartCount` climbing under `unless-stopped`, trace intact, proven with and without init; rig redeployed healthy with `docker-init` as PID 1. The CoordinatedShutdown no-`Environment.Exit` source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's `run-by-clr-shutdown-hook`). **Operator note: production overlays running this image under their own compose should also add `init: true`.** Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; **all four apps are Bootstrap hosts** — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: `src/ZB.MOM.WW.MxGateway.Server/libman.json` pins `bootstrap@5.3.3` into `wwwroot/lib/bootstrap/` and `App.razor:7` links it ahead of `<ThemeHead />`. What mxgw's CLAUDE.md forbids is Blazor *component libraries* (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `<style>` block** (Theme tokens w/ fallbacks; testids unchanged; bUnit pins). Scoped `.razor.css` was deliberately rejected — only ScadaBridge linked a scoped-CSS bundle **at the time (2026-07-19)**, so isolation CSS would silently not load on the other 3 hosts — still the right call, and note the premise has since changed: OtOpcUa's 2026-08-10 AdminUI sweep linked its own dead bundle (`App.razor:23`), so as of 2026-08-11 two of the four link one, while mxgw and HistorianGateway have zero `*.razor.css` and emit no bundle at all (the same silent-failure class). Live-verified on ScadaBridge (bumped to `0.2.3` @ `63510778`): modal visible `display=flex` with zero JS forcing, delete flows through the real path. **Rollout COMPLETE 2026-07-19 — all four apps on `0.2.3`, pushed:** OtOpcUa (`lmxopcua master @ be87ddeb`; AdminUI 665/665 + secrets integration 8/8; docker-dev rig rebuilt, modal gate PASS live), HistorianGateway (`main @ 7452eb4`, from `0.2.0`; plus `6bc005d` **fixing the pre-existing AngleSharp NU1902 warn-as-error that had made the whole HG test suite unbuildable** — test-only pin to 1.5.2, suite now runs: 854 pass / 1 pre-existing VPN-gated live failure, histsdk failures identical to the pre-bump baseline; docker container rebuilt, modal gate PASS live), and mxgw (`main @ 8f7ee49`, from `0.2.0`; 780 pass / 45 macOS-only NamedPipe-harness failures identical before and after the bump — zero delta). **mxgw's modal gate was originally SKIPPED on the false "Bootstrap-free so never affected" reasoning, and is now PASS (2026-08-11)** — run live against a local rig: modal root class is `zb-secrets-modal`, `document.querySelector('.modal')` is null so Bootstrap's `.modal{display:none}` has nothing to bite, computed `display:flex` / `position:fixed` / z-index 1081 over a full-viewport 1080 backdrop, and `elementFromPoint` at the confirm button's centre returns the button rather than the backdrop (visible ≠ hit-testable — that probe is the one that proves usable). Cancel disarmed cleanly with no delete performed. The exemption reached the right conclusion by luck: mxgw was a Bootstrap host and *was* affected, but had already taken the `0.2.3` fix independently of the sweep. **Lesson: an exemption from a family-wide gate must re-derive the exempting property from the repo at gate time** — see [`admin_ui_cleanup.md`](admin_ui_cleanup.md) **`0.3.0` (2026-07-19, merged to `main` @ `87ffe1b`) added a menu-driven Spectre.Console interactive operator console** to the `secret` CLI: bare `secret` with no verb on a real TTY (redirected I/O or any verb → unchanged headless behavior, no-args-redirected → usage/exit-2) opens a guided session that targets a deployment's store by reading the **target app's own appsettings.json** (base + env overlay + env vars → `Secrets` section, so it hits exactly the store the app reads; recents cached at `~/.zb-secrets/recent-targets.json`, names/paths only). Eight flows: list/set/reveal/delete plus the two headline ones — **Reference audit & seed** (classifies every `${secret:NAME}` ref Ok/PresentUnverified/Missing/Tombstoned/Undecryptable/InvalidName, then seeds the gaps — deployment setup) and **KEK doctor** (per-row Ok/WrongKek/Corrupt verdicts + guided rewrap-all or lost-KEK re-set — lockout recovery), plus ciphertext-only bundle export/import. Degraded (no-KEK) sessions open metadata-only and upgrade in place via masked paste/key-file; no login (local trust); key material never a literal arg. **5 packages published at `0.3.0` to the Gitea feed** (restore-verified); the four apps still pin the `0.2.3` libraries (the console ships in the CLI tool, not the app packages). Runbook `ZB.MOM.WW.Secrets/docs/operations/interactive-console.md` + quickstart `ZB.MOM.WW.Secrets/docs/interactive-console-quickstart.md`. **`0.6.3` (2026-09-28, task 1 of the site-secrets-on-artifact-deploy plan) makes every gRPC hub follower sweep observable:** `GrpcSecretSyncService`'s background loop now records outcomes on a new internal `SecretsHubSyncStatus` (registered singleton, exposed publicly as `ISecretsHubSyncStatus` with `LastSweepUtc`/`LastSuccessUtc`/`ConsecutiveFailures`/`LastError` — message-only, never a secret value) and logs three new events: EventId 4 (Debug, every sweep that finds nothing to pull), EventId 5 (Information, the same in-sync outcome but throttled to at most once per the new `SecretsGrpcHubClientOptions.InSyncLogInterval`, default 1 h, validated positive), and EventId 6 (Information, the first sweep to succeed after one or more failures — "recovered after N failed sweep(s)"). EventIds 1–3 are unchanged. `AddZbSecretsGrpcHubClient` registers `SecretsHubSyncStatus` as a singleton for both the concrete type and `ISecretsHubSyncStatus`, resolving to the same instance the sync service writes to. **NOT YET PUBLISHED to the Gitea feed** — built and tested, merged to scadaproj `main` (`29d0c50`); ScadaBridge has not been re-pinned. **Superseded for ScadaBridge, not just deferred (2026-09-28):** the same site-secrets-via-artifact-deploy work (`docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`, ScadaBridge `main` @ `23a139a4`) retires ALL secrets replication in ScadaBridge — the pull-only gRPC hub this 0.6.3 logging targets, and the historical SQL-Server replication mode — so **ScadaBridge is no longer a hub consumer at all** and `0.6.3` will never be re-pinned there; a site now receives its secrets over Deploy Artifacts instead. **`0.6.3` is published to the Gitea feed (2026-09-28, all six packages; CHANGELOG entry added)** for the library's other hub consumers (OtOpcUa) — no app pins it yet, and non-hub consumers gain nothing from it over `0.6.2`. | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |