Files
scadaproj/components/secrets/GAPS.md
T
Joseph Doherty 15ef1f32a8 fix(secrets): anti-entropy never ran — Self read after await in the replication actor
I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.

The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.

Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.

170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-18 04:42:16 -04:00

255 lines
23 KiB
Markdown

# Secrets — GAPS (adoption backlog)
The delta between each project's [`current-state`](current-state/) and the
[`SPEC`](spec/SPEC.md), as prioritized work. The library itself is **built + published +
reference-consumer-proven**; everything here is per-app adoption of an existing lib (not lib
construction).
## Current state (verified 2026-07-18)
- **`0.1.3` PUBLISHED to the Gitea feed 2026-07-18** (all three packages; restore-verified from a
scratch consumer). Carries the G-8 KEK-rotation work (`KekRotationService`, `RewrapReport`,
`secret rewrap-all`, `docs/operations/kek-rotation.md`), committed on `main` at `d82d345`.
(An earlier note claiming G-8 was "not yet committed/pushed" was stale.)
- **`0.1.3` also carries a transitive security fix that `0.1.2` does not.** Packing surfaced
NU1903: `Microsoft.Data.Sqlite 10.0.7` pulls `SQLitePCLRaw.lib.e_sqlite3 2.1.11`, which has a
**high-severity** advisory (GHSA-2m69-gcr7-jv3q). Fixed the way `ZB.MOM.WW.LocalDb` already
did — `CentralPackageTransitivePinningEnabled` + a pin to the patched `2.1.12`. Bumping
`Microsoft.Data.Sqlite` does **not** help (even `10.0.10` still pulls `2.1.11`); the pin is the
fix. Verified: the shipped `.nuspec` declares `SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12` as a direct
dependency, so **consumers get the patched native lib without needing their own pin**, and a
fresh consumer restoring `0.1.3` scans clean.
- **All four apps are still pinned at `0.1.2`** — they get neither KEK rotation nor the security
fix until bumped.
- **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take
`Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`.
- **G-7 clustered replication is BUILT (2026-07-18, lib `0.2.0`) — both options, not just one.**
The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build
covers **both**, as two new packages:
- **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** — `SqlServerSecretStore : ISecretStore` (all 7
members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and *two*
topologies: `AddZbSecretsSqlServerStore` (one shared store, Option A) and
`AddZbSecretsSqlServerReplication` (local store + hub, with a bidirectional sweep).
- **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** — peer-to-peer over distributed pub/sub
(Option B): live broadcast + periodic manifest anti-entropy, an explicit
`SerializerWithStringManifest`, ciphertext-only wire DTOs.
- **Core gained the pieces both need:** `ISecretsStoreMigrator` (so the hosted service + CLI are
store-agnostic), `SecretLastWriterWins` (one definition of the LWW tie-break, shared by every
store and reconciler), `SecretReplicationReconciler`, and `ReplicatingSecretStore`.
- **The `ISecretReplicator` seam had no caller** — nothing invoked `PublishAsync`, so local writes
would never have propagated. `ReplicatingSecretStore` decorates the store, which fixes it for
the UI, the CLI, and any future write path at once.
- Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`);
`AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace.
- Library suite: **170 passing offline (184 with the live SQL suite enabled), 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural
divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster**
(real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions,
late-joiner catch-up, and malformed-peer-input handling.
### Open
- ⬜ Bump the four apps `0.1.2``0.1.3`/`0.2.0` to pick up KEK rotation + the SQLitePCLRaw security fix.
- ⬜ Publish `0.2.0` (5 packages) to the Gitea feed — packed and scanned clean, **not yet pushed**.
- ⬜ Adopt a topology in ScadaBridge / OtOpcUa (config + shared KEK; no code change needed).
- ⬜ Per-app live wonder gates for the G-8 rotation path.
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
drift symptom, not the forwarding filter itself (noted in the test).
### Post-build code review (2026-07-18) — 7 defects found and fixed
An independent review caught real bugs that a green suite had not:
- **CRITICAL — both replication modes were dead on arrival.** The decorators resolve the *undecorated*
local store by concrete type, but core registered `SqliteSecretStore` only behind `ISecretStore`.
Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with
`No service for type 'SqliteSecretStore'`), fixed, and covered by new DI-resolution tests.
- **Unbounded fetch.** A cold-starting node pulls the peer's entire inventory in one call; past
~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the
node never converges. Now chunked at 500.
- **Poison row aborted the batch.** One unapplyable row abandoned every row after it, and since the
pull set recomputes in the same order each sweep, it would poison the same batch forever. Now
isolated per row and logged.
- **`Enum.Parse` on peer input.** An overflowing numeric string throws `OverflowException`, which the
actor's `ArgumentException` filter did not catch → actor restart loop on redelivery; and any
in-range number was accepted as an undefined enum value. Now `TryParse` + `IsDefined`.
- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*).
- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic.
- **CRITICAL — anti-entropy was broken by `Self` read after `await`.** Initially dismissed as a
false positive on the reasoning that `ActorBase` caches `Self` in a field and that three passing
anti-entropy tests traverse the path. **Both premises were wrong.** A direct probe
(`ActorContextAfterAwaitTests`) shows `Self` resolves through `Context`, which is `[ThreadStatic]`,
and throws `NotSupportedException` after a real thread hop. The tests passed only because a local
SQLite store usually completes `await` **synchronously**, keeping the continuation on the mailbox
thread with the context intact — so correctness silently depended on store latency and thread-pool
timing. `GenuinelyAsyncSecretStore` forces the async path;
`Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous` fails on the unfixed code
(20s timeout: "Node A never learned about a secret held by a node whose store is genuinely async")
and passes in 3s after capturing `self` on the actor thread. Every remaining `Self`/`Sender` access
in the actor was then audited.
**Process note worth keeping.** Seven of seven review findings were real. The one dismissal was
wrong, and it was dismissed using *indirect* evidence (a green suite) over a *direct* mechanism
claim. Green tests are evidence a path works under the timing the tests happened to produce, not
evidence it is correct. When a reviewer names a mechanism, probe the mechanism.
## Execution status (2026-07-16)
- **mxaccessgw — G-4 + G-5 + G-6 DONE.** Executed via the plan below (8 tasks, subagent-driven,
classification-driven reviews), **merged + pushed to `origin/main` (mxaccessgw @ `e088dfa`)**.
Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker),
behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev
(real .NET 10.0.8): fail-closed on unseeded `${secret:}` (`SecretNotFoundException`, refuses
to start) + CLI seed/`get` decrypt round-trip + **zero plaintext in the SQLite store at rest**.
(`/admin/secrets` interactive reveal not re-proven on this box — same shared `.Ui` component
already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real
LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP
password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth),
and the live-LDAP integration suite now honors the `MxGateway__Ldap__ServiceAccountPassword`
env override. **G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.**
- **ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE.** Executed via the plan below (10 tasks, subagent-driven,
classification-driven reviews), **FF-merged + pushed to `origin/main` (ScadaBridge @ `128f1596`)**.
Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181,
CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes
isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra
all green). **G-3 LIVE-PROVEN vs the REAL production MxGateway gateway** (`wonder-app-vd03.zmr.zimmer.com:5120`):
a throwaway harness drove the real `MxGatewayDataConnection` through the DI factory + `AddZbSecrets` with a
`secret:`-ref `ApiKey`**dummy key `Unauthenticated`** (proves the resolved value is transmitted, not
fail-closed first) → **real read-scoped key → `Connected` + browsed the real Galaxy root (10 nodes)**. Temp
key created via the gateway dashboard, then **revoked + deleted + functionally confirmed dead**. Notable
deltas from the plan: T3 registers `AddZbSecrets` on **both** the Central branch (UI) and
`SiteServiceRegistration` (DCL adapter) — the plan assumed one container; T6 claim-type resolved **MATCH**
(`ZbClaimTypes.Role == ClaimTypes.Role`, adversarially verified) → `AddSecretsAuthorization()` used directly;
T8 (G-5) shipped as a **docs-only** production-posture runbook (`docs/operations/2026-07-16-secrets-clustered-master-key.md`)
because hardcoding `Source=File` + `/shared` paths would break the Central-boot Host.Tests + local dev (the
pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking
follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single `[Collection]` / disable
parallelization for Host.Tests).
- **OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted).** Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), **FF-merged + pushed to `origin/master` (lmxopcua @ `872cf7e3`, `ec6598ce..872cf7e3`)**. Clean build (0 errors). The full offline suite caught **one real regression**`AddOtOpcUaDriverFactories`' registration now resolves `ISecretResolver` (Galaxy/OpcUaClient secret: refs), so two `ResilienceInvokerFactoryRegistrationTests` that built a minimal container threw `No service for ISecretResolver`; fixed test-only by registering a stub resolver (production always has it via the unconditional `AddZbSecrets`, and `GetRequiredService` correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental `AbCip_Green_AgainstSim` sim-fixture test — zero AbCip code touched). **Three deliberate deviations from the plan's literal text** (verified corrections, same class as the CPM miss the plan itself caught): (1) **G-4 does NOT commit a `${secret:}` token into the default appsettings** — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is `ServerHistorian:ApiKey=""` (empty, disabled-by-default), so a committed token would break every dev/CI/`TwoNodeClusterHarness` boot for zero at-rest benefit; instead documented the token-delivery convention (`docs/security.md`) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → `SecretNotFoundException` at pre-host expansion; seeded → resolves). (2) **G-5 shipped docs-only** (`docs/operations/2026-07-16-secrets-clustered-master-key.md`) — hardcoding `Source=File`+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays `Source=Environment`. Mirrors the ScadaBridge G-5 resolution. (3) **Task 8 resolves in the factory/driver only, NOT the probe** — the v3 `OpcUaClientDriverProbe` is unauthenticated `GetEndpoints`-only and never reads `Password`/`UserCertificatePassword`, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (`GalaxyDriver.BuildClientOptionsAsync` / `OpcUaClientDriver.InitializeAsync`), threaded via ctor + `DriverFactoryBootstrap` (real resolver from DI; `NullSecretResolver` null-object backs test/parse paths only, fail-closed on `secret:`). `OpcUaClientDriverOptions` was converted `sealed class``sealed record` for the `with`-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: `DriverConfig` persists as an opaque JSON string via `RawTreeService`, which has NO resolver dependency. **Live wonder gate PROVEN 2026-07-16 (VPN up):** a throwaway harness drove OtOpcUa's real `GalaxyDriverBrowser` (resolves the `secret:` `ApiKeySecretRef` via `ISecretResolver``GalaxySecretRef.ResolveApiKeyAsync` → connects) against the REAL production MxGateway `wonder-app-vd03:5120`. **Dummy `secret:` ref → `MxGatewayAuthenticationException` (proves the resolved value is transmitted, not fail-closed-first); real `metadata:read` key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…).** Temp key minted via the gateway dashboard, then **revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted** (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. **All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw.**
**Plan re-verified 2026-07-16 against `origin/master` @ `ec6598ce`** (the merged **v3.0 dual-namespace** rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa **is** on Central Package Management (was mis-stated as inline-versioned), `OpcUaClientDriverOptions` lives in `…Driver.OpcUaClient.Contracts` + `GalaxyDriverBrowser` in `…Driver.Galaxy.Browser`, and `DriverHostActor.cs` line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
- **G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17).**
G-8 shipped into `ZB.MOM.WW.Secrets` (lib `0.1.3`): the `Rewrap` DEK primitive, CAS-guarded
`ApplyRewrapAsync`, `KekRotationService.RewrapAllAsync`, the `secret rewrap-all` CLI verb, and an
operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial
crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka
fork is resolved (build the shared SQL-Server `ISecretStore`; Akka replicator deferred phase-2)
with a design doc + executable plan + `.tasks.json`, **no code built**. See the G-7/G-8 backlog
entries below. Not yet committed/pushed (awaiting go-ahead).
## Design + implementation plans (2026-07-16)
G-2 … G-6 are now planned. Shared design + three per-repo executable plans (task-metadata'd,
code-verified anchors, co-located `.tasks.json`):
- [`docs/plans/2026-07-16-secrets-adoption-design.md`](../../docs/plans/2026-07-16-secrets-adoption-design.md)
— shared design: library API, the two resolution layers (pre-host `${secret:}` config
expander vs runtime `ISecretResolver`), wiring template, master-key/clustering fork.
- [`docs/plans/2026-07-16-secrets-adoption-otopcua.md`](../../docs/plans/2026-07-16-secrets-adoption-otopcua.md)
**G-2 + G-4 + G-5 + G-6** (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
- [`docs/plans/2026-07-16-secrets-adoption-scadabridge.md`](../../docs/plans/2026-07-16-secrets-adoption-scadabridge.md)
**G-3 + G-4 + G-5 + G-6** (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
- [`docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md`](../../docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md)
**G-4 + G-5 + G-6** (8 tasks; single-box, no Layer B).
**G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED** (2026-07-17, below).
## Cross-cutting observations
- **No app has `${secret:}` resolution today.** All three assemble config with the stock
`AddJsonFile`+`AddEnvironmentVariables` chain and read secrets verbatim. Two already lean
on the *pattern* the expander formalizes: ScadaBridge's non-functional `${SCADABRIDGE_*}`
placeholders (whole-key env override) and OtOpcUa's driver-level `GalaxySecretRef`
(`env:`/`file:`/`dev:`/literal). These are the natural first swaps.
- **The common high-value gap is plaintext credentials in appsettings/env:** LDAP
service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb,
ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and
API-key peppers (all three, runtime-supplied but plaintext).
- **Secrets already stored in a DB in plaintext are the sharpest edge:** OtOpcUa's OpcUaClient
`Password`/`UserCertificatePassword` and the `dev:`/literal Galaxy API key live cleartext in
the central config DB; ScadaBridge's MxGateway per-endpoint `ApiKey` lives plaintext in the
ConfigDb `DataConnections` JSON. These leak at rest to anyone with DB read.
- **Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets**
(4 encrypted ConfigDb columns via `EncryptedStringConverter`). OtOpcUa/mxaccessgw use Data
Protection only for cookies/tokens. None use DPAPI or `ProtectKeysWith*`; key rings are
unencrypted-at-rest. `ZB.MOM.WW.Secrets` adds envelope encryption with a KEK held **outside**
the store — a stronger boundary than "the DB is the only protection."
- **Peppered-HMAC API-key stores stay bespoke** (mxaccessgw + ScadaBridge): they are hashed,
not reversibly stored — already correct. Secrets only takes over the *pepper value* supply.
## Backlog (prioritized)
### G-1 — HistorianGateway reference consumer ✅ DONE
Adopted + live-proven 2026-07-16 (historian password via `${secret:}`, authenticated read
against the real wonder historian). This is the wiring template for the three apps.
### G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)
Add a `secret:` arm to `GalaxySecretRef` (its own comment anticipates this) and to the
OpcUaClient driver options, resolving through `ISecretResolver` — retire the `dev:`/literal
and plaintext `Password`/`UserCertificatePassword` DB paths.
### G-3 — ScadaBridge: MxGateway `ApiKey` plaintext-in-ConfigDb
Resolve via `ISecretResolver` or extend the existing `EncryptedStringConverter` to that JSON
column.
### G-4 — Pre-host `${secret:}` for plaintext config secrets (all three)
LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa),
peppers — swap to `${secret:…}` tokens, expander running **before** each app's existing
validator (`ConfigPreflight` / `Ldap`+`OpcUa` validators / `GatewayOptionsValidator`).
Delete committed dev plaintext and the loose `*_login.txt` files (ScadaBridge).
### G-5 — Master-key provider per deployment
Env (`ZB_SECRETS_MASTER_KEY`) for containers; DPAPI for the Windows boxes; **File (shared
mounted key)** for clustered pairs (hard requirement — same KEK on every node).
### G-6 — Mount `/admin/secrets` UI (all three)
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib `0.2.0`)
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork was designed as *either/or* with
Option A recommended and Option B deferred. **Both were built** — the fork is now a deployment
choice rather than a library one, and the two packages are independent (an app takes whichever
matches its availability requirement, or neither).
| Package | Topologies | When |
|---|---|---|
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | shared store (Option A) · local + hub | every node can reach a shared DB |
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | peer-to-peer over the cluster (Option B) | a node must resolve while **partitioned** |
Three constraints hold across all of them, and each is documented at the API and in the runbook:
**same KEK on every node** (a mismatch fails closed and looks like corruption); **ciphertext only
crosses trust boundaries**; **re-wraps do not replicate**, so `rewrap-all` runs once per independent
store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no
merge — stated plainly in the README rather than left for an operator to discover.
Design + plan: [`…-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
+ [`…-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md).
Operator runbook: [`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md).
### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
`ZB.MOM.WW.Secrets`. Because the envelope wraps a per-secret DEK under the KEK, rotation **re-wraps
DEKs only** — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster
LWW). Shipped: `ISecretCipher.Rewrap(row, oldKek, newKek)` (fail-closed on wrong old KEK; DEK zeroed
on all paths; also catches malformed-length wraps → `SecretDecryptionException`),
`ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek)` (updates only the 4 wrap
columns + `kek_id`, **compare-and-swap on the current wrapped DEK** so a concurrent write can't
corrupt a row — review-caught TOCTOU), `KekRotationService.RewrapAllAsync` + `RewrapReport`
(enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK), the `secret rewrap-all` CLI verb (key material only via
env-var-name/file-path — never a literal), README section, and the operator runbook
[`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`](../../ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md).
**Verified:** full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke
(migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard).
Version bumped `0.1.2``0.1.3` (additive interface members); **republish to the Gitea feed is a
separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive).** Reviewed
by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent-
write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.
## Out of scope (this component)
- Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening
concern, not secret storage.
- SQL-Server `ISecretStore` provider construction (seam supports it; build with G-7).