scadaproj/components/secrets/GAPS.md

# Secrets — GAPS (adoption backlog)

The delta between each project's [`current-state`](current-state/) and the
[`SPEC`](spec/SPEC.md), as prioritized work. The library itself is **built + published +
reference-consumer-proven**; everything here is per-app adoption of an existing lib (not lib
construction).

## Current state (verified 2026-07-18)

- **`0.1.3` PUBLISHED to the Gitea feed 2026-07-18** (all three packages; restore-verified from a
  scratch consumer). Carries the G-8 KEK-rotation work (`KekRotationService`, `RewrapReport`,
  `secret rewrap-all`, `docs/operations/kek-rotation.md`), committed on `main` at `d82d345`.
  (An earlier note claiming G-8 was "not yet committed/pushed" was stale.)
- **`0.1.3` also carries a transitive security fix that `0.1.2` does not.** Packing surfaced
  NU1903: `Microsoft.Data.Sqlite 10.0.7` pulls `SQLitePCLRaw.lib.e_sqlite3 2.1.11`, which has a
  **high-severity** advisory (GHSA-2m69-gcr7-jv3q). Fixed the way `ZB.MOM.WW.LocalDb` already
  did — `CentralPackageTransitivePinningEnabled` + a pin to the patched `2.1.12`. Bumping
  `Microsoft.Data.Sqlite` does **not** help (even `10.0.10` still pulls `2.1.11`); the pin is the
  fix. Verified: the shipped `.nuspec` declares `SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12` as a direct
  dependency, so **consumers get the patched native lib without needing their own pin**, and a
  fresh consumer restoring `0.1.3` scans clean.
- **All four apps are still pinned at `0.1.2`** — they get neither KEK rotation nor the security
  fix until bumped.
- **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take
  `Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`.
- **G-7 clustered replication is BUILT (2026-07-18, lib `0.2.0`) — both options, not just one.**
  The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build
  covers **both**, as two new packages:
  - **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** — `SqlServerSecretStore : ISecretStore` (all 7
    members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and *two*
    topologies: `AddZbSecretsSqlServerStore` (one shared store, Option A) and
    `AddZbSecretsSqlServerReplication` (local store + hub, with a bidirectional sweep).
  - **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** — peer-to-peer over distributed pub/sub
    (Option B): live broadcast + periodic manifest anti-entropy, an explicit
    `SerializerWithStringManifest`, ciphertext-only wire DTOs.
  - **Core gained the pieces both need:** `ISecretsStoreMigrator` (so the hosted service + CLI are
    store-agnostic), `SecretLastWriterWins` (one definition of the LWW tie-break, shared by every
    store and reconciler), `SecretReplicationReconciler`, and `ReplicatingSecretStore`.
  - **The `ISecretReplicator` seam had no caller** — nothing invoked `PublishAsync`, so local writes
    would never have propagated. `ReplicatingSecretStore` decorates the store, which fixes it for
    the UI, the CLI, and any future write path at once.
  - Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`);
    `AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace.
- Library suite: **170 passing offline (184 with the live SQL suite enabled), 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
  against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural
  divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster**
  (real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions,
  late-joiner catch-up, and malformed-peer-input handling.

### Open

- ✅ **All four apps bumped `0.1.2` → `0.2.1`** (2026-07-18), each on a local `feat/secrets-0.2.0`
  branch, **not pushed**. HistorianGateway `bb9d923`, MxAccessGateway `b021e29`, OtOpcUa `1ccc237c`,
  ScadaBridge `0083ce35`.
  - **Correction:** the bump was originally justified as closing GHSA-2m69-gcr7-jv3q in all four
    apps. Measured against the `0.1.2` baseline in every repo, **all four already resolved the
    patched `SQLitePCLRaw.lib.e_sqlite3 2.1.12`** — two via their own pins, two transitively via
    `Auth.ApiKeys 0.1.5`. The bump delivers **no security change**; its real value is version
    currency, `secret rewrap-all` access, and being the precondition for the replicator packages.
- ✅ **A real SQLite vulnerability WAS found and fixed — but by separate work, not the bump.**
  ScadaBridge's `NuGetAuditSuppress` was masking a **live** high-severity finding: only the Host
  project resolved a patched `2.1.12`; `AuditLog`, `SiteRuntime`, `StoreAndForward`,
  `SiteEventLogging` and 11 test projects still resolved the vulnerable **`2.1.11`**. Fixed with an
  explicit `2.1.12` pin per SQLite-consuming project and the suppression deleted (`2a0faeab`).
  MxAccessGateway's obsolete suppression removed likewise (`2f0cfe3`). Both suppressions rested on
  false premises (*"no patched release exists"* / *"only the 3.x line is patched"*) — `2.1.12`
  patches it within 2.1.x, verified by forced restore. HistorianGateway and OtOpcUa needed nothing.
  - Rejected `CentralPackageTransitivePinningEnabled`: it clears the advisory in one line but makes
    every central version a transitive ceiling, demanding `Google.Protobuf`, `Grpc.Net.Client`,
    `Microsoft.Data.SqlClient` and `Newtonsoft.Json` bumps. That is a gRPC/data-access change to a
    production SCADA platform and deserves its own reviewed commit. **Still open as a follow-up** —
    it is the better long-term posture.
- ✅ `0.2.0` **PUBLISHED to the Gitea feed 2026-07-18** (all 5 packages; restore-verified from a
  scratch consumer — resolves `SQLitePCLRaw.lib.e_sqlite3 2.1.12`, `Microsoft.Data.SqlClient 6.0.2`,
  `Akka.Cluster.Tools 1.5.62`, and scans clean). Merged to `main` @ `15ef1f3`.
- 🟡 **Topology WIRED in both clustered apps, default-OFF, NOT live-validated.**
  **OtOpcUa → Akka peer-to-peer** (one cluster; `DistributedPubSub` already in product use;
  `ActorSystem` in DI via `Akka.Hosting`; Akka 1.5.62 matches the package exactly).
  **ScadaBridge → SQL-Server hub mode**, because central and each site are *separate* Akka clusters
  — `DistributedPubSub` cannot cross that boundary, so an Akka replicator there would produce
  **secret islands that never reconcile while every node reports healthy**.
  - Both gate on `Secrets:Replication:Enabled`, **default false**, so current behaviour is
    unchanged. ScadaBridge additionally requires a non-blank hub connection string — its options
    `Validate()` eagerly, so unconditional wiring would have broken startup on every node.
  - Bootstrap constraint recorded: the hub connstr **cannot come from the hub**; it must be seeded
    in each node's local store or supplied via env.
  - **Task 9 live gate RUN 2026-07-18. Split result — one topology passed, one FAILED.**

### Live gate: SQL-Server hub (ScadaBridge topology) — ✅ 6/6 PASSED

Run against a **real SQL Server 2022** (`10.100.0.35:31433`, DB `ZbSecretsGate`) with two nodes,
each with its own local SQLite store, using `AddZbSecretsSqlServerReplication` — the same extension
`AddScadaBridgeSecrets` calls:

| Check | Result |
|---|---|
| hub schema provisions + sweeps start | ✅ |
| write on A → resolves on B via the hub | ✅ converged through the shared hub |
| delete on A → tombstone on B | ✅ no resurrection |
| write on B → resolves on A | ✅ sweep is bidirectional |
| **partitioned node still resolves from local state** | ✅ hub unreachable, local reads unaffected — **site autonomy holds**, which is the whole reason ScadaBridge uses hub mode |
| **wrong-KEK node fails closed** | ✅ `SecretDecryptionException`, not wrong plaintext |

The KEK control fired spontaneously first: a re-run against a hub still holding rows from a prior
run (different KEK) threw `SecretDecryptionException` rather than returning garbage — unplanned but
real evidence of fail-closed.

### Live gate: Akka peer-to-peer (OtOpcUa topology) — ✅ PASSES 6/6 (root-caused + fixed in `0.2.2`, 2026-07-18)

The `0.2.1` hang is **root-caused and fixed**. It was never Akka at all: the Akka package's DI
wiring had a **circular singleton dependency** the container cannot detect through factory lambdas —
`ISecretStore` (the `ReplicatingSecretStore` decorator) → `ISecretReplicator` →
`SecretReplicationActorProvider` → `ISecretCacheInvalidator` → `DefaultSecretResolver` →
`ISecretStore` again. Resolution recurses around that loop (same-thread `Monitor` re-entry keeps it
alive) until MS.DI's `StackGuard` moves the recursion onto a fresh thread-pool thread to avoid a
stack overflow — and that thread then blocks forever on a singleton call-site lock the first thread
still holds. Managed stacks from a `dotnet-dump` of the hung process show the full cycle repeating
and both threads parked, which is the `Monitor_Wait` the native samples saw. Both earlier candidate
causes (`DistributedPubSub.Get` vs the `Lazy<IActorRef>` lock; missing `Akka.Cluster.Tools` HOCON)
are **disproven** — the actor's constructor was never even reached, and the deadlock reproduces on a
single-node rig with no clustering in play.

**Fix (`0.2.2`):** the one cycle-closing edge is deferred. `SecretReplicationActorProvider` now
receives a `DeferredSecretCacheInvalidator` that resolves the real invalidator on **first
eviction** — which can only happen when a replicated row is applied, strictly after the graph has
finished resolving. The SQL-Server package never had the cycle (its replicator chain never touches
the invalidator), which is exactly why the hub gate passed while Akka hung.

**Verified:** live convergence gate re-run — **6/6 checks** on a real 2-node cluster (write→peer,
delete→tombstone, tombstoned secret no longer resolves on the peer — the check that proves the
deferred invalidator actually evicts — reverse direction, wrong-KEK fail-closed). Full Akka test
project 38/38 green.

- ✅ **Regression test now exists at the exact blind spot:** `HostedProcessResolutionTests` builds
  the graph the way a host does (container-registered `ActorSystem`, hosted services started, then
  resolves `ISecretStore`/`ISecretReplicator`/`ISecretResolver` under a 20 s watchdog). Verified to
  discriminate: it deadlocks/fails on `0.2.1` and passes on `0.2.2`. `DeferredSecretCacheInvalidatorTests`
  pins the wrapper's contract (no eager resolve; evictions forward; lookup runs once). This was the
  **fourth** defect in this library visible only when the DI graph is built inside a real host.
- ✅ **`0.2.2` PUBLISHED to the Gitea feed 2026-07-18** (all 5 packages; restore-verified from a
  scratch consumer — 5 pinned refs, clean restore + build). Also folds in the deterministic rewrite
  of the `ActorContextAfterAwaitTests` pin (its old pool-thread formulation asserted a
  non-guaranteed property and flaked under parallel load).
- ✅ **OtOpcUa bumped to `0.2.2` (2026-07-18, `lmxopcua master @ 2254ae3d`)** and the **gate re-run
  against the published feed packages passes 6/6**. The long-promised
  `The_startup_hook_actually_creates_the_replication_actor` test now exists for real: a
  provider-based resolve of OtOpcUa's own `AddOtOpcUaSecrets` wiring on a self-joined single-node
  cluster, watchdogged at the exact point 0.2.1 deadlocked (`SecretsReplicationRegistrationTests`
  8/8; plain `ActorSystem`, no TestKit — xunit.v3-compatible).
  **Tracked:** [`scadaproj#1`](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) (the defect —
  fixed, CLOSED) and [`lmxopcua#482`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482)
  (open only for the per-environment decision to enable `Secrets:Replication:Enabled`, which stays
  default-false and requires the same KEK on every node first).
- ✅ ScadaBridge's hub topology **is** live-validated and safe to enable (config + a shared KEK).

Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed).

### Upstream defect found during adoption — `0.2.0` Akka replication was inert

- ✅ **Fixed in `0.2.1`, published to the feed 2026-07-18** (all 5 packages; all four apps consume it).
  `AddZbSecretsAkkaReplication` called `AddZbSecrets` **first**, which does
  `TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>()`, so the package's own `TryAdd` was
  silently discarded. Replication published into a no-op sink and **no actor was ever spawned** —
  no exception, no log line. The SQL-Server replicator always had the correct order.
- **Why it escaped:** the SQL-Server package had a DI test asserting its replicator type; the Akka
  package had none. Every Akka test exercised the actor, serializer and reconciler in isolation —
  **nothing built a container**. This is the **third** instance of this defect class in this library
  (inert `ISecretReplicator` seam → unregistered concrete `SqliteSecretStore` → this), all with one
  cause: DI extensions with no test that constructs the graph. `0.2.1` adds 5 such tests, verified
  to discriminate (2 fail on the old order).
- **Testing note:** `Akka.TestKit` is the family convention for non-end-to-end Akka tests, but
  `Akka.TestKit.Xunit2` is **xunit-v2-only** and OtOpcUa's `Host.IntegrationTests` is on xunit.v3
  (`CS0433` if added). OtOpcUa's guards therefore assert on `ServiceCollection` descriptors — which
  is exactly where this defect lives — while actor creation and convergence stay covered by the
  library's own 2-node cluster suite.
- ⬜ Per-app live wonder gates for the G-8 rotation path.
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
  drift symptom, not the forwarding filter itself (noted in the test).

### Post-build code review (2026-07-18) — 7 defects found and fixed

An independent review caught real bugs that a green suite had not:

- **CRITICAL — both replication modes were dead on arrival.** The decorators resolve the *undecorated*
  local store by concrete type, but core registered `SqliteSecretStore` only behind `ISecretStore`.
  Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with
  `No service for type 'SqliteSecretStore'`), fixed, and covered by new DI-resolution tests.
- **Unbounded fetch.** A cold-starting node pulls the peer's entire inventory in one call; past
  ~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the
  node never converges. Now chunked at 500.
- **Poison row aborted the batch.** One unapplyable row abandoned every row after it, and since the
  pull set recomputes in the same order each sweep, it would poison the same batch forever. Now
  isolated per row and logged.
- **`Enum.Parse` on peer input.** An overflowing numeric string throws `OverflowException`, which the
  actor's `ArgumentException` filter did not catch → actor restart loop on redelivery; and any
  in-range number was accepted as an undefined enum value. Now `TryParse` + `IsDefined`.
- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*).
- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic.

- **CRITICAL — anti-entropy was broken by `Self` read after `await`.** Initially dismissed as a
  false positive on the reasoning that `ActorBase` caches `Self` in a field and that three passing
  anti-entropy tests traverse the path. **Both premises were wrong.** A direct probe
  (`ActorContextAfterAwaitTests`) shows `Self` resolves through `Context`, which is `[ThreadStatic]`,
  and throws `NotSupportedException` after a real thread hop. The tests passed only because a local
  SQLite store usually completes `await` **synchronously**, keeping the continuation on the mailbox
  thread with the context intact — so correctness silently depended on store latency and thread-pool
  timing. `GenuinelyAsyncSecretStore` forces the async path;
  `Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous` fails on the unfixed code
  (20s timeout: "Node A never learned about a secret held by a node whose store is genuinely async")
  and passes in 3s after capturing `self` on the actor thread. Every remaining `Self`/`Sender` access
  in the actor was then audited.

**Process note worth keeping.** Seven of seven review findings were real. The one dismissal was
wrong, and it was dismissed using *indirect* evidence (a green suite) over a *direct* mechanism
claim. Green tests are evidence a path works under the timing the tests happened to produce, not
evidence it is correct. When a reviewer names a mechanism, probe the mechanism.

## Execution status (2026-07-16)

- **mxaccessgw — G-4 + G-5 + G-6 DONE.** Executed via the plan below (8 tasks, subagent-driven,
  classification-driven reviews), **merged + pushed to `origin/main` (mxaccessgw @ `e088dfa`)**.
  Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker),
  behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev
  (real .NET 10.0.8): fail-closed on unseeded `${secret:}` (`SecretNotFoundException`, refuses
  to start) + CLI seed/`get` decrypt round-trip + **zero plaintext in the SQLite store at rest**.
  (`/admin/secrets` interactive reveal not re-proven on this box — same shared `.Ui` component
  already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real
  LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP
  password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth),
  and the live-LDAP integration suite now honors the `MxGateway__Ldap__ServiceAccountPassword`
  env override. **G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.**
- **ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE.** Executed via the plan below (10 tasks, subagent-driven,
  classification-driven reviews), **FF-merged + pushed to `origin/main` (ScadaBridge @ `128f1596`)**.
  Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181,
  CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes
  isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra
  all green). **G-3 LIVE-PROVEN vs the REAL production MxGateway gateway** (`wonder-app-vd03.zmr.zimmer.com:5120`):
  a throwaway harness drove the real `MxGatewayDataConnection` through the DI factory + `AddZbSecrets` with a
  `secret:`-ref `ApiKey` → **dummy key `Unauthenticated`** (proves the resolved value is transmitted, not
  fail-closed first) → **real read-scoped key → `Connected` + browsed the real Galaxy root (10 nodes)**. Temp
  key created via the gateway dashboard, then **revoked + deleted + functionally confirmed dead**. Notable
  deltas from the plan: T3 registers `AddZbSecrets` on **both** the Central branch (UI) and
  `SiteServiceRegistration` (DCL adapter) — the plan assumed one container; T6 claim-type resolved **MATCH**
  (`ZbClaimTypes.Role == ClaimTypes.Role`, adversarially verified) → `AddSecretsAuthorization()` used directly;
  T8 (G-5) shipped as a **docs-only** production-posture runbook (`docs/operations/2026-07-16-secrets-clustered-master-key.md`)
  because hardcoding `Source=File` + `/shared` paths would break the Central-boot Host.Tests + local dev (the
  pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking
  follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single `[Collection]` / disable
  parallelization for Host.Tests).
- **OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted).** Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), **FF-merged + pushed to `origin/master` (lmxopcua @ `872cf7e3`, `ec6598ce..872cf7e3`)**. Clean build (0 errors). The full offline suite caught **one real regression** — `AddOtOpcUaDriverFactories`' registration now resolves `ISecretResolver` (Galaxy/OpcUaClient secret: refs), so two `ResilienceInvokerFactoryRegistrationTests` that built a minimal container threw `No service for ISecretResolver`; fixed test-only by registering a stub resolver (production always has it via the unconditional `AddZbSecrets`, and `GetRequiredService` correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental `AbCip_Green_AgainstSim` sim-fixture test — zero AbCip code touched). **Three deliberate deviations from the plan's literal text** (verified corrections, same class as the CPM miss the plan itself caught): (1) **G-4 does NOT commit a `${secret:}` token into the default appsettings** — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is `ServerHistorian:ApiKey=""` (empty, disabled-by-default), so a committed token would break every dev/CI/`TwoNodeClusterHarness` boot for zero at-rest benefit; instead documented the token-delivery convention (`docs/security.md`) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → `SecretNotFoundException` at pre-host expansion; seeded → resolves). (2) **G-5 shipped docs-only** (`docs/operations/2026-07-16-secrets-clustered-master-key.md`) — hardcoding `Source=File`+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays `Source=Environment`. Mirrors the ScadaBridge G-5 resolution. (3) **Task 8 resolves in the factory/driver only, NOT the probe** — the v3 `OpcUaClientDriverProbe` is unauthenticated `GetEndpoints`-only and never reads `Password`/`UserCertificatePassword`, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (`GalaxyDriver.BuildClientOptionsAsync` / `OpcUaClientDriver.InitializeAsync`), threaded via ctor + `DriverFactoryBootstrap` (real resolver from DI; `NullSecretResolver` null-object backs test/parse paths only, fail-closed on `secret:`). `OpcUaClientDriverOptions` was converted `sealed class` → `sealed record` for the `with`-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: `DriverConfig` persists as an opaque JSON string via `RawTreeService`, which has NO resolver dependency. **Live wonder gate PROVEN 2026-07-16 (VPN up):** a throwaway harness drove OtOpcUa's real `GalaxyDriverBrowser` (resolves the `secret:` `ApiKeySecretRef` via `ISecretResolver` → `GalaxySecretRef.ResolveApiKeyAsync` → connects) against the REAL production MxGateway `wonder-app-vd03:5120`. **Dummy `secret:` ref → `MxGatewayAuthenticationException` (proves the resolved value is transmitted, not fail-closed-first); real `metadata:read` key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…).** Temp key minted via the gateway dashboard, then **revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted** (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. **All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw.**
  **Plan re-verified 2026-07-16 against `origin/master` @ `ec6598ce`** (the merged **v3.0 dual-namespace** rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa **is** on Central Package Management (was mis-stated as inline-versioned), `OpcUaClientDriverOptions` lives in `…Driver.OpcUaClient.Contracts` + `GalaxyDriverBrowser` in `…Driver.Galaxy.Browser`, and `DriverHostActor.cs` line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
- **G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17).**
  G-8 shipped into `ZB.MOM.WW.Secrets` (lib `0.1.3`): the `Rewrap` DEK primitive, CAS-guarded
  `ApplyRewrapAsync`, `KekRotationService.RewrapAllAsync`, the `secret rewrap-all` CLI verb, and an
  operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial
  crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka
  fork is resolved (build the shared SQL-Server `ISecretStore`; Akka replicator deferred phase-2)
  with a design doc + executable plan + `.tasks.json`, **no code built**. See the G-7/G-8 backlog
  entries below. Not yet committed/pushed (awaiting go-ahead).

## Design + implementation plans (2026-07-16)

G-2 … G-6 are now planned. Shared design +  three per-repo executable plans (task-metadata'd,
code-verified anchors, co-located `.tasks.json`):

- [`docs/plans/2026-07-16-secrets-adoption-design.md`](../../docs/plans/2026-07-16-secrets-adoption-design.md)
  — shared design: library API, the two resolution layers (pre-host `${secret:}` config
  expander vs runtime `ISecretResolver`), wiring template, master-key/clustering fork.
- [`docs/plans/2026-07-16-secrets-adoption-otopcua.md`](../../docs/plans/2026-07-16-secrets-adoption-otopcua.md)
  — **G-2 + G-4 + G-5 + G-6** (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
- [`docs/plans/2026-07-16-secrets-adoption-scadabridge.md`](../../docs/plans/2026-07-16-secrets-adoption-scadabridge.md)
  — **G-3 + G-4 + G-5 + G-6** (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
- [`docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md`](../../docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md)
  — **G-4 + G-5 + G-6** (8 tasks; single-box, no Layer B).

**G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED** (2026-07-17, below).

## Cross-cutting observations

- **No app has `${secret:}` resolution today.** All three assemble config with the stock
  `AddJsonFile`+`AddEnvironmentVariables` chain and read secrets verbatim. Two already lean
  on the *pattern* the expander formalizes: ScadaBridge's non-functional `${SCADABRIDGE_*}`
  placeholders (whole-key env override) and OtOpcUa's driver-level `GalaxySecretRef`
  (`env:`/`file:`/`dev:`/literal). These are the natural first swaps.
- **The common high-value gap is plaintext credentials in appsettings/env:** LDAP
  service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb,
  ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and
  API-key peppers (all three, runtime-supplied but plaintext).
- **Secrets already stored in a DB in plaintext are the sharpest edge:** OtOpcUa's OpcUaClient
  `Password`/`UserCertificatePassword` and the `dev:`/literal Galaxy API key live cleartext in
  the central config DB; ScadaBridge's MxGateway per-endpoint `ApiKey` lives plaintext in the
  ConfigDb `DataConnections` JSON. These leak at rest to anyone with DB read.
- **Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets**
  (4 encrypted ConfigDb columns via `EncryptedStringConverter`). OtOpcUa/mxaccessgw use Data
  Protection only for cookies/tokens. None use DPAPI or `ProtectKeysWith*`; key rings are
  unencrypted-at-rest. `ZB.MOM.WW.Secrets` adds envelope encryption with a KEK held **outside**
  the store — a stronger boundary than "the DB is the only protection."
- **Peppered-HMAC API-key stores stay bespoke** (mxaccessgw + ScadaBridge): they are hashed,
  not reversibly stored — already correct. Secrets only takes over the *pepper value* supply.

## Backlog (prioritized)

### G-1 — HistorianGateway reference consumer ✅ DONE
Adopted + live-proven 2026-07-16 (historian password via `${secret:}`, authenticated read
against the real wonder historian). This is the wiring template for the three apps.

### G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)
Add a `secret:` arm to `GalaxySecretRef` (its own comment anticipates this) and to the
OpcUaClient driver options, resolving through `ISecretResolver` — retire the `dev:`/literal
and plaintext `Password`/`UserCertificatePassword` DB paths.

### G-3 — ScadaBridge: MxGateway `ApiKey` plaintext-in-ConfigDb
Resolve via `ISecretResolver` or extend the existing `EncryptedStringConverter` to that JSON
column.

### G-4 — Pre-host `${secret:}` for plaintext config secrets (all three)
LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa),
peppers — swap to `${secret:…}` tokens, expander running **before** each app's existing
validator (`ConfigPreflight` / `Ldap`+`OpcUa` validators / `GatewayOptionsValidator`).
Delete committed dev plaintext and the loose `*_login.txt` files (ScadaBridge).

### G-5 — Master-key provider per deployment
Env (`ZB_SECRETS_MASTER_KEY`) for containers; DPAPI for the Windows boxes; **File (shared
mounted key)** for clustered pairs (hard requirement — same KEK on every node).

### G-6 — Mount `/admin/secrets` UI (all three)
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.

### G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib `0.2.0`)
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork was designed as *either/or* with
Option A recommended and Option B deferred. **Both were built** — the fork is now a deployment
choice rather than a library one, and the two packages are independent (an app takes whichever
matches its availability requirement, or neither).

| Package | Topologies | When |
|---|---|---|
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | shared store (Option A) · local + hub | every node can reach a shared DB |
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | peer-to-peer over the cluster (Option B) | a node must resolve while **partitioned** |

Three constraints hold across all of them, and each is documented at the API and in the runbook:
**same KEK on every node** (a mismatch fails closed and looks like corruption); **ciphertext only
crosses trust boundaries**; **re-wraps do not replicate**, so `rewrap-all` runs once per independent
store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no
merge — stated plainly in the README rather than left for an operator to discover.

Design + plan: [`…-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
+ [`…-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md).
Operator runbook: [`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md).

### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
`ZB.MOM.WW.Secrets`. Because the envelope wraps a per-secret DEK under the KEK, rotation **re-wraps
DEKs only** — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster
LWW). Shipped: `ISecretCipher.Rewrap(row, oldKek, newKek)` (fail-closed on wrong old KEK; DEK zeroed
on all paths; also catches malformed-length wraps → `SecretDecryptionException`),
`ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek)` (updates only the 4 wrap
columns + `kek_id`, **compare-and-swap on the current wrapped DEK** so a concurrent write can't
corrupt a row — review-caught TOCTOU), `KekRotationService.RewrapAllAsync` + `RewrapReport`
(enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK), the `secret rewrap-all` CLI verb (key material only via
env-var-name/file-path — never a literal), README section, and the operator runbook
[`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`](../../ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md).
**Verified:** full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke
(migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard).
Version bumped `0.1.2`→`0.1.3` (additive interface members); **republish to the Gitea feed is a
separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive).** Reviewed
by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent-
write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.

## Out of scope (this component)

- Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening
  concern, not secret storage.
- SQL-Server `ISecretStore` provider construction (seam supports it; build with G-7).