# Secrets — GAPS (adoption backlog)

The delta between each project's [`current-state`](current-state/) and the
[`SPEC`](spec/SPEC.md), as prioritized work. The library itself is **built + published +
reference-consumer-proven**; everything here is per-app adoption of an existing lib (not lib
construction).

## Current state (verified 2026-07-18)

- **`0.1.3` PUBLISHED to the Gitea feed 2026-07-18** (all three packages; restore-verified from a
  scratch consumer). Carries the G-8 KEK-rotation work (`KekRotationService`, `RewrapReport`,
  `secret rewrap-all`, `docs/operations/kek-rotation.md`), committed on `main` at `d82d345`.
  (An earlier note claiming G-8 was "not yet committed/pushed" was stale.)
- **`0.1.3` also carries a transitive security fix that `0.1.2` does not.** Packing surfaced
  NU1903: `Microsoft.Data.Sqlite 10.0.7` pulls `SQLitePCLRaw.lib.e_sqlite3 2.1.11`, which has a
  **high-severity** advisory (GHSA-2m69-gcr7-jv3q). Fixed the way `ZB.MOM.WW.LocalDb` already
  did — `CentralPackageTransitivePinningEnabled` + a pin to the patched `2.1.12`. Bumping
  `Microsoft.Data.Sqlite` does **not** help (even `10.0.10` still pulls `2.1.11`); the pin is the
  fix. Verified: the shipped `.nuspec` declares `SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12` as a direct
  dependency, so **consumers get the patched native lib without needing their own pin**, and a
  fresh consumer restoring `0.1.3` scans clean.
- **All four apps are still pinned at `0.1.2`** — they get neither KEK rotation nor the security
  fix until bumped.
- **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take
  `Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`.
- **G-7 clustered replication is designed + planned, not built** — plan at
  `docs/plans/2026-07-17-secrets-g7-*` (resolved to build the shared SQL-Server `ISecretStore`;
  Akka replicator deferred to phase 2). No `ISecretStore` SQL-Server implementation exists yet.
- Library suite: **97 passing, 1 skipped**, green 2026-07-18.

### Open

- ⬜ Bump the four apps `0.1.2` → `0.1.3` to pick up KEK rotation + the SQLitePCLRaw security fix.
- ⬜ Execute the G-7 plan (shared SQL-Server `ISecretStore`).
- ⬜ Per-app live wonder gates for the G-8 rotation path.

## Execution status (2026-07-16)

- **mxaccessgw — G-4 + G-5 + G-6 DONE.** Executed via the plan below (8 tasks, subagent-driven,
  classification-driven reviews), **merged + pushed to `origin/main` (mxaccessgw @ `e088dfa`)**.
  Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker),
  behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev
  (real .NET 10.0.8): fail-closed on unseeded `${secret:}` (`SecretNotFoundException`, refuses
  to start) + CLI seed/`get` decrypt round-trip + **zero plaintext in the SQLite store at rest**.
  (`/admin/secrets` interactive reveal not re-proven on this box — same shared `.Ui` component
  already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real
  LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP
  password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth),
  and the live-LDAP integration suite now honors the `MxGateway__Ldap__ServiceAccountPassword`
  env override. **G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.**
- **ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE.** Executed via the plan below (10 tasks, subagent-driven,
  classification-driven reviews), **FF-merged + pushed to `origin/main` (ScadaBridge @ `128f1596`)**.
  Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181,
  CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes
  isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra
  all green). **G-3 LIVE-PROVEN vs the REAL production MxGateway gateway** (`wonder-app-vd03.zmr.zimmer.com:5120`):
  a throwaway harness drove the real `MxGatewayDataConnection` through the DI factory + `AddZbSecrets` with a
  `secret:`-ref `ApiKey` → **dummy key `Unauthenticated`** (proves the resolved value is transmitted, not
  fail-closed first) → **real read-scoped key → `Connected` + browsed the real Galaxy root (10 nodes)**. Temp
  key created via the gateway dashboard, then **revoked + deleted + functionally confirmed dead**. Notable
  deltas from the plan: T3 registers `AddZbSecrets` on **both** the Central branch (UI) and
  `SiteServiceRegistration` (DCL adapter) — the plan assumed one container; T6 claim-type resolved **MATCH**
  (`ZbClaimTypes.Role == ClaimTypes.Role`, adversarially verified) → `AddSecretsAuthorization()` used directly;
  T8 (G-5) shipped as a **docs-only** production-posture runbook (`docs/operations/2026-07-16-secrets-clustered-master-key.md`)
  because hardcoding `Source=File` + `/shared` paths would break the Central-boot Host.Tests + local dev (the
  pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking
  follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single `[Collection]` / disable
  parallelization for Host.Tests).
- **OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted).** Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), **FF-merged + pushed to `origin/master` (lmxopcua @ `872cf7e3`, `ec6598ce..872cf7e3`)**. Clean build (0 errors). The full offline suite caught **one real regression** — `AddOtOpcUaDriverFactories`' registration now resolves `ISecretResolver` (Galaxy/OpcUaClient secret: refs), so two `ResilienceInvokerFactoryRegistrationTests` that built a minimal container threw `No service for ISecretResolver`; fixed test-only by registering a stub resolver (production always has it via the unconditional `AddZbSecrets`, and `GetRequiredService` correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental `AbCip_Green_AgainstSim` sim-fixture test — zero AbCip code touched). **Three deliberate deviations from the plan's literal text** (verified corrections, same class as the CPM miss the plan itself caught): (1) **G-4 does NOT commit a `${secret:}` token into the default appsettings** — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is `ServerHistorian:ApiKey=""` (empty, disabled-by-default), so a committed token would break every dev/CI/`TwoNodeClusterHarness` boot for zero at-rest benefit; instead documented the token-delivery convention (`docs/security.md`) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → `SecretNotFoundException` at pre-host expansion; seeded → resolves). (2) **G-5 shipped docs-only** (`docs/operations/2026-07-16-secrets-clustered-master-key.md`) — hardcoding `Source=File`+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays `Source=Environment`. Mirrors the ScadaBridge G-5 resolution. (3) **Task 8 resolves in the factory/driver only, NOT the probe** — the v3 `OpcUaClientDriverProbe` is unauthenticated `GetEndpoints`-only and never reads `Password`/`UserCertificatePassword`, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (`GalaxyDriver.BuildClientOptionsAsync` / `OpcUaClientDriver.InitializeAsync`), threaded via ctor + `DriverFactoryBootstrap` (real resolver from DI; `NullSecretResolver` null-object backs test/parse paths only, fail-closed on `secret:`). `OpcUaClientDriverOptions` was converted `sealed class` → `sealed record` for the `with`-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: `DriverConfig` persists as an opaque JSON string via `RawTreeService`, which has NO resolver dependency. **Live wonder gate PROVEN 2026-07-16 (VPN up):** a throwaway harness drove OtOpcUa's real `GalaxyDriverBrowser` (resolves the `secret:` `ApiKeySecretRef` via `ISecretResolver` → `GalaxySecretRef.ResolveApiKeyAsync` → connects) against the REAL production MxGateway `wonder-app-vd03:5120`. **Dummy `secret:` ref → `MxGatewayAuthenticationException` (proves the resolved value is transmitted, not fail-closed-first); real `metadata:read` key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…).** Temp key minted via the gateway dashboard, then **revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted** (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. **All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw.**
  **Plan re-verified 2026-07-16 against `origin/master` @ `ec6598ce`** (the merged **v3.0 dual-namespace** rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa **is** on Central Package Management (was mis-stated as inline-versioned), `OpcUaClientDriverOptions` lives in `…Driver.OpcUaClient.Contracts` + `GalaxyDriverBrowser` in `…Driver.Galaxy.Browser`, and `DriverHostActor.cs` line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
- **G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17).**
  G-8 shipped into `ZB.MOM.WW.Secrets` (lib `0.1.3`): the `Rewrap` DEK primitive, CAS-guarded
  `ApplyRewrapAsync`, `KekRotationService.RewrapAllAsync`, the `secret rewrap-all` CLI verb, and an
  operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial
  crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka
  fork is resolved (build the shared SQL-Server `ISecretStore`; Akka replicator deferred phase-2)
  with a design doc + executable plan + `.tasks.json`, **no code built**. See the G-7/G-8 backlog
  entries below. Not yet committed/pushed (awaiting go-ahead).

## Design + implementation plans (2026-07-16)

G-2 … G-6 are now planned. Shared design +  three per-repo executable plans (task-metadata'd,
code-verified anchors, co-located `.tasks.json`):

- [`docs/plans/2026-07-16-secrets-adoption-design.md`](../../docs/plans/2026-07-16-secrets-adoption-design.md)
  — shared design: library API, the two resolution layers (pre-host `${secret:}` config
  expander vs runtime `ISecretResolver`), wiring template, master-key/clustering fork.
- [`docs/plans/2026-07-16-secrets-adoption-otopcua.md`](../../docs/plans/2026-07-16-secrets-adoption-otopcua.md)
  — **G-2 + G-4 + G-5 + G-6** (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
- [`docs/plans/2026-07-16-secrets-adoption-scadabridge.md`](../../docs/plans/2026-07-16-secrets-adoption-scadabridge.md)
  — **G-3 + G-4 + G-5 + G-6** (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
- [`docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md`](../../docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md)
  — **G-4 + G-5 + G-6** (8 tasks; single-box, no Layer B).

**G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED** (2026-07-17, below).

## Cross-cutting observations

- **No app has `${secret:}` resolution today.** All three assemble config with the stock
  `AddJsonFile`+`AddEnvironmentVariables` chain and read secrets verbatim. Two already lean
  on the *pattern* the expander formalizes: ScadaBridge's non-functional `${SCADABRIDGE_*}`
  placeholders (whole-key env override) and OtOpcUa's driver-level `GalaxySecretRef`
  (`env:`/`file:`/`dev:`/literal). These are the natural first swaps.
- **The common high-value gap is plaintext credentials in appsettings/env:** LDAP
  service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb,
  ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and
  API-key peppers (all three, runtime-supplied but plaintext).
- **Secrets already stored in a DB in plaintext are the sharpest edge:** OtOpcUa's OpcUaClient
  `Password`/`UserCertificatePassword` and the `dev:`/literal Galaxy API key live cleartext in
  the central config DB; ScadaBridge's MxGateway per-endpoint `ApiKey` lives plaintext in the
  ConfigDb `DataConnections` JSON. These leak at rest to anyone with DB read.
- **Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets**
  (4 encrypted ConfigDb columns via `EncryptedStringConverter`). OtOpcUa/mxaccessgw use Data
  Protection only for cookies/tokens. None use DPAPI or `ProtectKeysWith*`; key rings are
  unencrypted-at-rest. `ZB.MOM.WW.Secrets` adds envelope encryption with a KEK held **outside**
  the store — a stronger boundary than "the DB is the only protection."
- **Peppered-HMAC API-key stores stay bespoke** (mxaccessgw + ScadaBridge): they are hashed,
  not reversibly stored — already correct. Secrets only takes over the *pepper value* supply.

## Backlog (prioritized)

### G-1 — HistorianGateway reference consumer ✅ DONE
Adopted + live-proven 2026-07-16 (historian password via `${secret:}`, authenticated read
against the real wonder historian). This is the wiring template for the three apps.

### G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)
Add a `secret:` arm to `GalaxySecretRef` (its own comment anticipates this) and to the
OpcUaClient driver options, resolving through `ISecretResolver` — retire the `dev:`/literal
and plaintext `Password`/`UserCertificatePassword` DB paths.

### G-3 — ScadaBridge: MxGateway `ApiKey` plaintext-in-ConfigDb
Resolve via `ISecretResolver` or extend the existing `EncryptedStringConverter` to that JSON
column.

### G-4 — Pre-host `${secret:}` for plaintext config secrets (all three)
LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa),
peppers — swap to `${secret:…}` tokens, expander running **before** each app's existing
validator (`ConfigPreflight` / `Ldap`+`OpcUa` validators / `GatewayOptionsValidator`).
Delete committed dev plaintext and the loose `*_login.txt` files (ScadaBridge).

### G-5 — Master-key provider per deployment
Env (`ZB_SECRETS_MASTER_KEY`) for containers; DPAPI for the Windows boxes; **File (shared
mounted key)** for clustered pairs (hard requirement — same KEK on every node).

### G-6 — Mount `/admin/secrets` UI (all three)
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.

### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17)
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork is **resolved: build Option A —
a shared SQL-Server `ISecretStore`** (both clustered apps already run shared SQL + a shared
Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure
modes); the Akka replicator (`ZB.MOM.WW.Secrets.Akka`, LWW/anti-entropy/tombstones) is a **deferred
phase-2** to build only on a stated partition-tolerance requirement. Both require a shared KEK; the
`ISecretReplicator` seam stays in place so Option B remains a drop-in later. Design + executable
plan (+ `.tasks.json`): [`docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
+ [`docs/plans/2026-07-17-secrets-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). **No code built yet — plan is ready to execute.**

### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
`ZB.MOM.WW.Secrets`. Because the envelope wraps a per-secret DEK under the KEK, rotation **re-wraps
DEKs only** — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster
LWW). Shipped: `ISecretCipher.Rewrap(row, oldKek, newKek)` (fail-closed on wrong old KEK; DEK zeroed
on all paths; also catches malformed-length wraps → `SecretDecryptionException`),
`ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek)` (updates only the 4 wrap
columns + `kek_id`, **compare-and-swap on the current wrapped DEK** so a concurrent write can't
corrupt a row — review-caught TOCTOU), `KekRotationService.RewrapAllAsync` + `RewrapReport`
(enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK), the `secret rewrap-all` CLI verb (key material only via
env-var-name/file-path — never a literal), README section, and the operator runbook
[`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`](../../ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md).
**Verified:** full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke
(migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard).
Version bumped `0.1.2`→`0.1.3` (additive interface members); **republish to the Gitea feed is a
separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive).** Reviewed
by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent-
write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.

## Out of scope (this component)

- Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening
  concern, not secret storage.
- SQL-Server `ISecretStore` provider construction (seam supports it; build with G-7).