scadaproj/ZB.MOM.WW.Secrets/CLAUDE.md

ZB.MOM.WW.Secrets

Shared encrypted secret store + ${secret:} resolution for the ZB.MOM.WW SCADA family (5 packages + the secret CLI; replicators for SqlServer / AkkaDotNet / Grpc). Build/test from this directory: dotnet test. Component spec + gaps: ../components/secrets/. Operator runbooks under docs/operations/.

Index history (migrated from the scadaproj root CLAUDE.md)

Moved verbatim out of the scadaproj root CLAUDE.md on 2026-09-03 so it loads only when needed. This file is now the authoritative place for this history — append new entries here and keep the root index row short.

Component-table row as it read in the index

Component	Status	Goal	Design	Implementation
Secrets (encrypted store + ${secret:} resolution)	Built (current lib version 0.5.1, published to the Gitea feed — the 0.1.x→0.5.1 progression is narrated below; ScadaBridge pins 0.5.1 (it consumes the new Grpc replicator), the other three apps still pin the 0.2.3 libraries, and the 0.3.0 interactive-console delta ships in the CLI tool; HistorianGateway adopted + live-proven 2026-07-16; mxaccessgw adopted G-4/G-5/G-6 + merged to origin/main @ e088dfa 2026-07-16, box-verified fail-closed + encrypt-at-rest; ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to origin/main @ 128f1596 2026-07-16, G-3 live-proven vs the real production MxGateway gateway wonder-app-vd03:5120 (secret-ref ApiKey → Connected + browsed real Galaxy); OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to origin/master @ 872cf7e3 (lmxopcua) 2026-07-16 — the last app, so all four now adopted; Layer-B driver secrets (Galaxy API key + OpcUaClient Password/UserCertificatePassword) resolve fail-closed at driver session-open, AddZbSecrets registered unconditionally so driver-only nodes work; G-2 live-proven vs the real production MxGateway wonder-app-vd03:5120 — a secret:-ref Galaxy ApiKey resolved through OtOpcUa's real GalaxyDriverBrowser → dummy key MxGatewayAuthenticationException, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); G-8 KEK-rotation BUILT (lib bumped 0.1.2→0.1.3, 2026-07-17): Rewrap DEK primitive + CAS-guarded ApplyRewrapAsync + KekRotationService.RewrapAllAsync + secret rewrap-all CLI + operator runbook (ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). G-7 clustered replication BUILT 2026-07-18 (lib 0.2.0) — BOTH fork options, packaged as two new libs: ZB.MOM.WW.Secrets.Replicator.SqlServer (shared SQL-Server ISecretStore = the plan's Option A, plus a local-store-with-hub mode) and ZB.MOM.WW.Secrets.Replicator.AkkaDotNet (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained ISecretsStoreMigrator, SecretLastWriterWins (one shared LWW tie-break so stores cannot diverge), SecretReplicationReconciler, and ReplicatingSecretStore — the last of which fixes a real gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including 15 live tests vs a real SQL Server 2022 (the SQLite suite ported case-for-case) and a 9-test in-process 2-node Akka cluster (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (rewrap-all runs once per independent store). Operator runbook ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md. Corrected 2026-07-18: G-8 + the G-7 plan ARE committed and pushed on main (d82d345) — the "not yet committed/pushed" note was stale. 0.1.3 PUBLISHED to the feed 2026-07-18 (restore-verified), carrying a transitive security pin (SQLitePCLRaw.lib.e_sqlite3 → 2.1.12, advisory GHSA-2m69-gcr7-jv3q) that 0.1.2 lacked. All 4 apps bumped to 0.2.1 2026-07-18 (local feat/secrets-0.2.0 branches, NOT pushed). 0.2.1 fixes a critical defect: 0.2.0's Akka replicator was silently inert (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is wired but default-OFF and NOT live-validated — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a live SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. 0.2.0 PUBLISHED to the feed 2026-07-18 (5 packages, restore-verified from a scratch consumer; merged to main @ 15ef1f3) — no app has adopted a clustered topology yet. 0.2.2 (2026-07-18, scadaproj#1 FIXED): the Akka replicator's hosted-process DI deadlock root-caused — a circular singleton dependency hidden in factory lambdas (ISecretStore decorator → ISecretReplicator → actor provider → ISecretCacheInvalidator → DefaultSecretResolver → ISecretStore), which MS.DI's StackGuard turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT DistributedPubSub/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (DeferredSecretCacheInvalidator, resolves on first eviction); Akka live gate now 6/6 (was: hang), regression coverage HostedProcessResolutionTests (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. 0.2.2 PUBLISHED to the feed 2026-07-18 (5 packages, restore-verified from a scratch consumer). OtOpcUa bumped to 0.2.2 same day (lmxopcua master @ 2254ae3d): Akka gate re-run vs the FEED packages 6/6, plus a new provider-based The_startup_hook_actually_creates_the_replication_actor test of OtOpcUa's own wiring (8/8); ENABLED + live-proven on the OtOpcUa docker-dev rig same day (lmxopcua master @ c878fbbd, #482 CLOSED): all six rig nodes on Secrets:Replication:Enabled=true with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; production topologies DECIDED 2026-08-07 (scadaproj#3, design docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md): OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with one fleet-wide production KEK via appsettings/env, never committed; ScadaBridge = a new ZB.MOM.WW.Secrets.Replicator.Grpc central-hosted hub (pull-only — updates originate at central only, by design), because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07). The Grpc replicator is BUILT and PUBLISHED — family 0.4.0 (scadaproj main @ 321ab22) then 0.4.1 (c86cead, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced at the contract (secrets_hub.v1 has exactly two RPCs, GetManifest/GetSecrets, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a GetSecrets batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. ScadaBridge is wired both ends and live-gated PASS 4/4 (ScadaBridge main @ f6c3f7c5, merged + pushed; gate ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md): the node role is a method parameter (`SecretsNodeRole.Central	Site) so config cannot flip a site into hosting the hub, mode key Secrets:Replication:Mode ∈ {SqlServer(default, legacy-equivalent),Grpc}, hub mapped on central's existing h2c control-plane listener (CentralGrpcPort, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (lmxopcua master @ 1bd79c8e, gate OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan docs/plans/2026-08-07-secrets-central-shared-sql-store.md) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' ISecretStore is **ONE shared SQL-Server store** (AddZbSecretsSqlServerStore; central now REQUIRES Secrets:SqlServer:ConnectionString, fail-closed — blank refused naming the key, ${secret:} refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-central-b defect is unreproducible. The site client gained ordered **FallbackEndpoints** with sticky preference (family 0.5.0, main @ 7a2980e→e62b4a5): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **0.5.1** (main @ 31ca940): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now 0.6.2** (2026-08-11, main @ c5e894d; 0.6.0 944595a, 0.6.1 cbfe296 — all three published and restore-verified). **0.6.2is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.**Secrets:SqlitePath is now validated at startup (ValidateOnStart) by SecretsOptionsValidatorusing two rules in the new publicSecretsStorePathRules, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — C:\Windows\System32for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store intosrc/and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where…\MxGateway\Server\gateway-auth.dbleft withServer` and an empty store was created at the same path, so no gRPC consumer could authenticate for two days, silently). The rules reject rather than relocate, and are deliberately not exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. BREAKING for configuration, not API: SecretsOptions.SqlitePath now defaults to SecretsOptions.DefaultSqlitePath (absolute, ZB.MOM.WW/secrets.db under LocalApplicationData), so an app configuring a relative path will not boot — remove the key or set an absolute one. Per-user, not CommonApplicationData, on purpose: that is /usr/share on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. The migration edge was closed by evidence before the default moved — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in service env, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: AddZbSecrets validator registration is now idempotent — TryAddEnumerable de-duplicates on implementation type, which a factory descriptor does not carry, so it threw on the second call rather than skipping (mxgw calls AddZbSecrets twice; the Akka replicator calls it internally). 500 tests pass. Rollout status: pre-bump pins were ScadaBridge 0.5.1, the other three 0.2.3 — so three of the four are four-minor jumps, not re-pins. mxgw MERGED + PUSHED 0.6.0 (mxaccessgw origin/main @ 9bc70d1, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up. HistorianGateway has adopted 0.6.2 (4-arg content-root call site included) and re-verified on the exact probe that found the defect (in-content-root path → aborts with no .db/-wal/-shm left behind); merged to main + pushed 2026-08-13 (e42ff31). OtOpcUa has adopted 0.6.2 (b037cc7a, superseding 835e35a8; unpushed) with the heavy part discharged by live evidence rather than changelog trust: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves Replicator.AkkaDotNet, whose pair-local topology was live-gated on 0.2.3 — a re-gate may be the real cost there. 0.6.1 closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target 0.6.1, never 0.6.0. The validator resolves the content root from IHostEnvironment and skipped the under-content-root rule when it was absent — but an app that expands ${secret:} references before building its host composes the subsystem in a throwaway ServiceCollection with no IHostEnvironment in it and then migrates the store from that container. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator created the database, and the boot failed a second later once the real host registered IHostEnvironment. Net effect: a rejected boot that still left an empty store with -wal/-shm at the rejected path — the exact artifact that made the MxGateway outage read as the database is there, it's just empty. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating no IHostEnvironment registered with no content root exists. Fix is an additive overload AddZbSecrets(services, config, sectionPath, contentRootPath); the 3-arg overload is unchanged. THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix. The exposure is pre-host-expander-shaped, not app-shaped: an app with a correctly configured path shows no symptom and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. mxgw and HistorianGateway are CONFIRMED affected (mxgw GatewayApplication.cs:82, HG Program.cs:90 — both new ServiceCollection().AddZbSecrets(...) followed by SqliteSecretsStoreMigrator.MigrateAsync); OtOpcUa is CONFIRMED affected too (Program.cs, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. Containers: the rule deliberately over-rejects. A store on a volume mounted INSIDE the content root (/app/data/… with /app the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and no opt-out flag — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an improvement rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so the same path would pass on one host and fail on another — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used /data); the file sits at the volume root either way, so seeded stores carry over. NU1102 after a bump is a cached-index artifact, not the feed — and it recurs on every publish, consistently stranding Replicator.AkkaDotNet at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. dotnet nuget locals http-cache --clear. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a do-not-simplify-back-to-the-3-arg-overload comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. Verify with positive controls, not a clean boot — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the old relative default is orphaned, not migrated, and nothing signals it — deleting the key, rather than leaving a relative value, is what makes the new default apply. ZB.MOM.WW.Secrets now depends on ZB.MOM.WW.Configuration 0.1.0 for the shared OptionsValidatorBase/ValidationBuilder vocabulary. Secrets.Ui is byte-identical between 0.2.3 and 0.6.0 (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned 0.5.1 (main 68f812ea→4d7f09d5→e697477c, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to ONE run against central's shared SQL store (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, ScadaBridge#34, is FIXED+CLOSED 2026-08-08 (ScadaBridge main @ e9c412e5): the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays running, restart: unless-stopped never fires) was root-caused to dotnet running as container PID 1 — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so abort() could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an AppDomain.UnhandledException handler registered before anything can throw exits 134 via the exit() syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus init: true on all 8 compose nodes for crash paths that bypass the managed event (Environment.FailFast, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with RestartCount climbing under unless-stopped, trace intact, proven with and without init; rig redeployed healthy with docker-init as PID 1. The CoordinatedShutdown no-Environment.Exit source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's run-by-clr-shutdown-hook). Operator note: production overlays running this image under their own compose should also add init: true. Side-find lmxopcua#483 FIXED+CLOSED 2026-07-19 (lmxopcua @ 5f72ff85): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying @rendermode InteractiveServer (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. Family-wide /admin/secrets sweep 2026-07-19: ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared ZB.MOM.WW.Audit.IAuditWriter seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons IAuditWriter) → hard 500 on every render, hidden behind the login wall. FIXED+CLOSED as ScadaBridge#22 (ScadaBridge main @ fc86e8bf): CentralSharedSeamAuditWriter bridges the shared seam to ICentralAuditWriter (central direct-write → dbo.AuditLog, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; secret.add/secret.delete audit rows landed in dbo.AuditLog). The sweep also surfaced a Secrets.Ui library defect, scadaproj#2 — FIXED+CLOSED same day via 0.2.3 (5 packages PUBLISHED to the feed; scadaproj main @ 62ab8f6): ConfirmDeleteModal used bare class="modal" and the RCL shipped no CSS, so Bootstrap 5's .modal{display:none} made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; all four apps are Bootstrap hosts — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: src/ZB.MOM.WW.MxGateway.Server/libman.json pins bootstrap@5.3.3 into wwwroot/lib/bootstrap/ and App.razor:7 links it ahead of <ThemeHead />. What mxgw's CLAUDE.md forbids is Blazor component libraries (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: zb-secrets-modal* class rename + a component-emitted <style> block (Theme tokens w/ fallbacks; testids unchanged; bUnit pins). Scoped .razor.css was deliberately rejected — only ScadaBridge linked a scoped-CSS bundle at the time (2026-07-19), so isolation CSS would silently not load on the other 3 hosts — still the right call, and note the premise has since changed: OtOpcUa's 2026-08-10 AdminUI sweep linked its own dead bundle (App.razor:23), so as of 2026-08-11 two of the four link one, while mxgw and HistorianGateway have zero *.razor.css and emit no bundle at all (the same silent-failure class). Live-verified on ScadaBridge (bumped to 0.2.3 @ 63510778): modal visible display=flex with zero JS forcing, delete flows through the real path. Rollout COMPLETE 2026-07-19 — all four apps on 0.2.3, pushed: OtOpcUa (lmxopcua master @ be87ddeb; AdminUI 665/665 + secrets integration 8/8; docker-dev rig rebuilt, modal gate PASS live), HistorianGateway (main @ 7452eb4, from 0.2.0; plus 6bc005d fixing the pre-existing AngleSharp NU1902 warn-as-error that had made the whole HG test suite unbuildable — test-only pin to 1.5.2, suite now runs: 854 pass / 1 pre-existing VPN-gated live failure, histsdk failures identical to the pre-bump baseline; docker container rebuilt, modal gate PASS live), and mxgw (main @ 8f7ee49, from 0.2.0; 780 pass / 45 macOS-only NamedPipe-harness failures identical before and after the bump — zero delta). mxgw's modal gate was originally SKIPPED on the false "Bootstrap-free so never affected" reasoning, and is now PASS (2026-08-11) — run live against a local rig: modal root class is zb-secrets-modal, document.querySelector('.modal') is null so Bootstrap's .modal{display:none} has nothing to bite, computed display:flex / position:fixed / z-index 1081 over a full-viewport 1080 backdrop, and elementFromPoint at the confirm button's centre returns the button rather than the backdrop (visible ≠ hit-testable — that probe is the one that proves usable). Cancel disarmed cleanly with no delete performed. The exemption reached the right conclusion by luck: mxgw was a Bootstrap host and was affected, but had already taken the 0.2.3 fix independently of the sweep. Lesson: an exemption from a family-wide gate must re-derive the exempting property from the repo at gate time — see admin_ui_cleanup.md 0.3.0 (2026-07-19, merged to main @ 87ffe1b) added a menu-driven Spectre.Console interactive operator console to the secret CLI: bare secret with no verb on a real TTY (redirected I/O or any verb → unchanged headless behavior, no-args-redirected → usage/exit-2) opens a guided session that targets a deployment's store by reading the target app's own appsettings.json (base + env overlay + env vars → Secrets section, so it hits exactly the store the app reads; recents cached at ~/.zb-secrets/recent-targets.json, names/paths only). Eight flows: list/set/reveal/delete plus the two headline ones — Reference audit & seed (classifies every ${secret:NAME} ref Ok/PresentUnverified/Missing/Tombstoned/Undecryptable/InvalidName, then seeds the gaps — deployment setup) and KEK doctor (per-row Ok/WrongKek/Corrupt verdicts + guided rewrap-all or lost-KEK re-set — lockout recovery), plus ciphertext-only bundle export/import. Degraded (no-KEK) sessions open metadata-only and upgrade in place via masked paste/key-file; no login (local trust); key material never a literal arg. 5 packages published at 0.3.0 to the Gitea feed (restore-verified); the four apps still pin the 0.2.3 libraries (the console ships in the CLI tool, not the app packages). Runbook ZB.MOM.WW.Secrets/docs/operations/interactive-console.md + quickstart ZB.MOM.WW.Secrets/docs/interactive-console-quickstart.md. 0.6.3 (2026-09-28, task 1 of the site-secrets-on-artifact-deploy plan) makes every gRPC hub follower sweep observable: GrpcSecretSyncService's background loop now records outcomes on a new internal SecretsHubSyncStatus (registered singleton, exposed publicly as ISecretsHubSyncStatus with LastSweepUtc/LastSuccessUtc/ConsecutiveFailures/LastError — message-only, never a secret value) and logs three new events: EventId 4 (Debug, every sweep that finds nothing to pull), EventId 5 (Information, the same in-sync outcome but throttled to at most once per the new SecretsGrpcHubClientOptions.InSyncLogInterval, default 1 h, validated positive), and EventId 6 (Information, the first sweep to succeed after one or more failures — "recovered after N failed sweep(s)"). EventIds 1–3 are unchanged. AddZbSecretsGrpcHubClient registers SecretsHubSyncStatus as a singleton for both the concrete type and ISecretsHubSyncStatus, resolving to the same instance the sync service writes to. NOT YET PUBLISHED to the Gitea feed — built and tested, merged to scadaproj main (29d0c50); ScadaBridge has not been re-pinned. Superseded for ScadaBridge, not just deferred (2026-09-28): the same site-secrets-via-artifact-deploy work (docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md, ScadaBridge main @ 23a139a4) retires ALL secrets replication in ScadaBridge — the pull-only gRPC hub this 0.6.3 logging targets, and the historical SQL-Server replication mode — so ScadaBridge is no longer a hub consumer at all and 0.6.3 will never be re-pinned there; a site now receives its secrets over Deploy Artifacts instead. 0.6.3 is published to the Gitea feed (2026-09-28, all six packages; CHANGELOG entry added) for the library's other hub consumers (OtOpcUa) — no app pins it yet, and non-hub consumers gain nothing from it over 0.6.2.	Shared ZB.MOM.WW.Secrets lib (5 packages + CLI)	components/secrets/