scadaproj/components/secrets/GAPS.md

# Secrets — GAPS (adoption backlog)

The delta between each project's [`current-state`](current-state/) and the
[`SPEC`](spec/SPEC.md), as prioritized work. The library itself is **built + published +
reference-consumer-proven**; everything here is per-app adoption of an existing lib (not lib
construction).

## Current state (verified 2026-07-18)

- **`0.1.3` PUBLISHED to the Gitea feed 2026-07-18** (all three packages; restore-verified from a
  scratch consumer). Carries the G-8 KEK-rotation work (`KekRotationService`, `RewrapReport`,
  `secret rewrap-all`, `docs/operations/kek-rotation.md`), committed on `main` at `d82d345`.
  (An earlier note claiming G-8 was "not yet committed/pushed" was stale.)
- **`0.1.3` also carries a transitive security fix that `0.1.2` does not.** Packing surfaced
  NU1903: `Microsoft.Data.Sqlite 10.0.7` pulls `SQLitePCLRaw.lib.e_sqlite3 2.1.11`, which has a
  **high-severity** advisory (GHSA-2m69-gcr7-jv3q). Fixed the way `ZB.MOM.WW.LocalDb` already
  did — `CentralPackageTransitivePinningEnabled` + a pin to the patched `2.1.12`. Bumping
  `Microsoft.Data.Sqlite` does **not** help (even `10.0.10` still pulls `2.1.11`); the pin is the
  fix. Verified: the shipped `.nuspec` declares `SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12` as a direct
  dependency, so **consumers get the patched native lib without needing their own pin**, and a
  fresh consumer restoring `0.1.3` scans clean.
- **All four apps are still pinned at `0.1.2`** — they get neither KEK rotation nor the security
  fix until bumped.
- **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take
  `Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`.
- **G-7 clustered replication is BUILT (2026-07-18, lib `0.2.0`) — both options, not just one.**
  The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build
  covers **both**, as two new packages:
  - **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** — `SqlServerSecretStore : ISecretStore` (all 7
    members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and *two*
    topologies: `AddZbSecretsSqlServerStore` (one shared store, Option A) and
    `AddZbSecretsSqlServerReplication` (local store + hub, with a bidirectional sweep).
  - **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** — peer-to-peer over distributed pub/sub
    (Option B): live broadcast + periodic manifest anti-entropy, an explicit
    `SerializerWithStringManifest`, ciphertext-only wire DTOs.
  - **Core gained the pieces both need:** `ISecretsStoreMigrator` (so the hosted service + CLI are
    store-agnostic), `SecretLastWriterWins` (one definition of the LWW tie-break, shared by every
    store and reconciler), `SecretReplicationReconciler`, and `ReplicatingSecretStore`.
  - **The `ISecretReplicator` seam had no caller** — nothing invoked `PublishAsync`, so local writes
    would never have propagated. `ReplicatingSecretStore` decorates the store, which fixes it for
    the UI, the CLI, and any future write path at once.
  - Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`);
    `AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace.
- Library suite: **182 passing, 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
  against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural
  divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster**
  (real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions,
  late-joiner catch-up, and malformed-peer-input handling.

### Open

- ⬜ Bump the four apps `0.1.2` → `0.1.3`/`0.2.0` to pick up KEK rotation + the SQLitePCLRaw security fix.
- ⬜ Publish `0.2.0` (5 packages) to the Gitea feed — packed and scanned clean, **not yet pushed**.
- ⬜ Adopt a topology in ScadaBridge / OtOpcUa (config + shared KEK; no code change needed).
- ⬜ Per-app live wonder gates for the G-8 rotation path.
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
  drift symptom, not the forwarding filter itself (noted in the test).

### Post-build code review (2026-07-18) — 6 defects found and fixed

An independent review caught real bugs that a green suite had not:

- **CRITICAL — both replication modes were dead on arrival.** The decorators resolve the *undecorated*
  local store by concrete type, but core registered `SqliteSecretStore` only behind `ISecretStore`.
  Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with
  `No service for type 'SqliteSecretStore'`), fixed, and covered by new DI-resolution tests.
- **Unbounded fetch.** A cold-starting node pulls the peer's entire inventory in one call; past
  ~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the
  node never converges. Now chunked at 500.
- **Poison row aborted the batch.** One unapplyable row abandoned every row after it, and since the
  pull set recomputes in the same order each sweep, it would poison the same batch forever. Now
  isolated per row and logged.
- **`Enum.Parse` on peer input.** An overflowing numeric string throws `OverflowException`, which the
  actor's `ArgumentException` filter did not catch → actor restart loop on redelivery; and any
  in-range number was accepted as an undefined enum value. Now `TryParse` + `IsDefined`.
- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*).
- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic.

One reported "critical" was a **false positive**: `Self` used after `await` in the actor. Akka's
`ActorBase` caches `Self` in an instance field (unlike `Context`, which is `[ThreadStatic]`), and
three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as
written.

## Execution status (2026-07-16)

- **mxaccessgw — G-4 + G-5 + G-6 DONE.** Executed via the plan below (8 tasks, subagent-driven,
  classification-driven reviews), **merged + pushed to `origin/main` (mxaccessgw @ `e088dfa`)**.
  Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker),
  behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev
  (real .NET 10.0.8): fail-closed on unseeded `${secret:}` (`SecretNotFoundException`, refuses
  to start) + CLI seed/`get` decrypt round-trip + **zero plaintext in the SQLite store at rest**.
  (`/admin/secrets` interactive reveal not re-proven on this box — same shared `.Ui` component
  already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real
  LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP
  password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth),
  and the live-LDAP integration suite now honors the `MxGateway__Ldap__ServiceAccountPassword`
  env override. **G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.**
- **ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE.** Executed via the plan below (10 tasks, subagent-driven,
  classification-driven reviews), **FF-merged + pushed to `origin/main` (ScadaBridge @ `128f1596`)**.
  Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181,
  CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes
  isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra
  all green). **G-3 LIVE-PROVEN vs the REAL production MxGateway gateway** (`wonder-app-vd03.zmr.zimmer.com:5120`):
  a throwaway harness drove the real `MxGatewayDataConnection` through the DI factory + `AddZbSecrets` with a
  `secret:`-ref `ApiKey` → **dummy key `Unauthenticated`** (proves the resolved value is transmitted, not
  fail-closed first) → **real read-scoped key → `Connected` + browsed the real Galaxy root (10 nodes)**. Temp
  key created via the gateway dashboard, then **revoked + deleted + functionally confirmed dead**. Notable
  deltas from the plan: T3 registers `AddZbSecrets` on **both** the Central branch (UI) and
  `SiteServiceRegistration` (DCL adapter) — the plan assumed one container; T6 claim-type resolved **MATCH**
  (`ZbClaimTypes.Role == ClaimTypes.Role`, adversarially verified) → `AddSecretsAuthorization()` used directly;
  T8 (G-5) shipped as a **docs-only** production-posture runbook (`docs/operations/2026-07-16-secrets-clustered-master-key.md`)
  because hardcoding `Source=File` + `/shared` paths would break the Central-boot Host.Tests + local dev (the
  pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking
  follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single `[Collection]` / disable
  parallelization for Host.Tests).
- **OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted).** Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), **FF-merged + pushed to `origin/master` (lmxopcua @ `872cf7e3`, `ec6598ce..872cf7e3`)**. Clean build (0 errors). The full offline suite caught **one real regression** — `AddOtOpcUaDriverFactories`' registration now resolves `ISecretResolver` (Galaxy/OpcUaClient secret: refs), so two `ResilienceInvokerFactoryRegistrationTests` that built a minimal container threw `No service for ISecretResolver`; fixed test-only by registering a stub resolver (production always has it via the unconditional `AddZbSecrets`, and `GetRequiredService` correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental `AbCip_Green_AgainstSim` sim-fixture test — zero AbCip code touched). **Three deliberate deviations from the plan's literal text** (verified corrections, same class as the CPM miss the plan itself caught): (1) **G-4 does NOT commit a `${secret:}` token into the default appsettings** — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is `ServerHistorian:ApiKey=""` (empty, disabled-by-default), so a committed token would break every dev/CI/`TwoNodeClusterHarness` boot for zero at-rest benefit; instead documented the token-delivery convention (`docs/security.md`) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → `SecretNotFoundException` at pre-host expansion; seeded → resolves). (2) **G-5 shipped docs-only** (`docs/operations/2026-07-16-secrets-clustered-master-key.md`) — hardcoding `Source=File`+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays `Source=Environment`. Mirrors the ScadaBridge G-5 resolution. (3) **Task 8 resolves in the factory/driver only, NOT the probe** — the v3 `OpcUaClientDriverProbe` is unauthenticated `GetEndpoints`-only and never reads `Password`/`UserCertificatePassword`, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (`GalaxyDriver.BuildClientOptionsAsync` / `OpcUaClientDriver.InitializeAsync`), threaded via ctor + `DriverFactoryBootstrap` (real resolver from DI; `NullSecretResolver` null-object backs test/parse paths only, fail-closed on `secret:`). `OpcUaClientDriverOptions` was converted `sealed class` → `sealed record` for the `with`-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: `DriverConfig` persists as an opaque JSON string via `RawTreeService`, which has NO resolver dependency. **Live wonder gate PROVEN 2026-07-16 (VPN up):** a throwaway harness drove OtOpcUa's real `GalaxyDriverBrowser` (resolves the `secret:` `ApiKeySecretRef` via `ISecretResolver` → `GalaxySecretRef.ResolveApiKeyAsync` → connects) against the REAL production MxGateway `wonder-app-vd03:5120`. **Dummy `secret:` ref → `MxGatewayAuthenticationException` (proves the resolved value is transmitted, not fail-closed-first); real `metadata:read` key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…).** Temp key minted via the gateway dashboard, then **revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted** (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. **All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw.**
  **Plan re-verified 2026-07-16 against `origin/master` @ `ec6598ce`** (the merged **v3.0 dual-namespace** rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa **is** on Central Package Management (was mis-stated as inline-versioned), `OpcUaClientDriverOptions` lives in `…Driver.OpcUaClient.Contracts` + `GalaxyDriverBrowser` in `…Driver.Galaxy.Browser`, and `DriverHostActor.cs` line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
- **G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17).**
  G-8 shipped into `ZB.MOM.WW.Secrets` (lib `0.1.3`): the `Rewrap` DEK primitive, CAS-guarded
  `ApplyRewrapAsync`, `KekRotationService.RewrapAllAsync`, the `secret rewrap-all` CLI verb, and an
  operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial
  crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka
  fork is resolved (build the shared SQL-Server `ISecretStore`; Akka replicator deferred phase-2)
  with a design doc + executable plan + `.tasks.json`, **no code built**. See the G-7/G-8 backlog
  entries below. Not yet committed/pushed (awaiting go-ahead).

## Design + implementation plans (2026-07-16)

G-2 … G-6 are now planned. Shared design +  three per-repo executable plans (task-metadata'd,
code-verified anchors, co-located `.tasks.json`):

- [`docs/plans/2026-07-16-secrets-adoption-design.md`](../../docs/plans/2026-07-16-secrets-adoption-design.md)
  — shared design: library API, the two resolution layers (pre-host `${secret:}` config
  expander vs runtime `ISecretResolver`), wiring template, master-key/clustering fork.
- [`docs/plans/2026-07-16-secrets-adoption-otopcua.md`](../../docs/plans/2026-07-16-secrets-adoption-otopcua.md)
  — **G-2 + G-4 + G-5 + G-6** (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
- [`docs/plans/2026-07-16-secrets-adoption-scadabridge.md`](../../docs/plans/2026-07-16-secrets-adoption-scadabridge.md)
  — **G-3 + G-4 + G-5 + G-6** (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
- [`docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md`](../../docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md)
  — **G-4 + G-5 + G-6** (8 tasks; single-box, no Layer B).

**G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED** (2026-07-17, below).

## Cross-cutting observations

- **No app has `${secret:}` resolution today.** All three assemble config with the stock
  `AddJsonFile`+`AddEnvironmentVariables` chain and read secrets verbatim. Two already lean
  on the *pattern* the expander formalizes: ScadaBridge's non-functional `${SCADABRIDGE_*}`
  placeholders (whole-key env override) and OtOpcUa's driver-level `GalaxySecretRef`
  (`env:`/`file:`/`dev:`/literal). These are the natural first swaps.
- **The common high-value gap is plaintext credentials in appsettings/env:** LDAP
  service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb,
  ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and
  API-key peppers (all three, runtime-supplied but plaintext).
- **Secrets already stored in a DB in plaintext are the sharpest edge:** OtOpcUa's OpcUaClient
  `Password`/`UserCertificatePassword` and the `dev:`/literal Galaxy API key live cleartext in
  the central config DB; ScadaBridge's MxGateway per-endpoint `ApiKey` lives plaintext in the
  ConfigDb `DataConnections` JSON. These leak at rest to anyone with DB read.
- **Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets**
  (4 encrypted ConfigDb columns via `EncryptedStringConverter`). OtOpcUa/mxaccessgw use Data
  Protection only for cookies/tokens. None use DPAPI or `ProtectKeysWith*`; key rings are
  unencrypted-at-rest. `ZB.MOM.WW.Secrets` adds envelope encryption with a KEK held **outside**
  the store — a stronger boundary than "the DB is the only protection."
- **Peppered-HMAC API-key stores stay bespoke** (mxaccessgw + ScadaBridge): they are hashed,
  not reversibly stored — already correct. Secrets only takes over the *pepper value* supply.

## Backlog (prioritized)

### G-1 — HistorianGateway reference consumer ✅ DONE
Adopted + live-proven 2026-07-16 (historian password via `${secret:}`, authenticated read
against the real wonder historian). This is the wiring template for the three apps.

### G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)
Add a `secret:` arm to `GalaxySecretRef` (its own comment anticipates this) and to the
OpcUaClient driver options, resolving through `ISecretResolver` — retire the `dev:`/literal
and plaintext `Password`/`UserCertificatePassword` DB paths.

### G-3 — ScadaBridge: MxGateway `ApiKey` plaintext-in-ConfigDb
Resolve via `ISecretResolver` or extend the existing `EncryptedStringConverter` to that JSON
column.

### G-4 — Pre-host `${secret:}` for plaintext config secrets (all three)
LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa),
peppers — swap to `${secret:…}` tokens, expander running **before** each app's existing
validator (`ConfigPreflight` / `Ldap`+`OpcUa` validators / `GatewayOptionsValidator`).
Delete committed dev plaintext and the loose `*_login.txt` files (ScadaBridge).

### G-5 — Master-key provider per deployment
Env (`ZB_SECRETS_MASTER_KEY`) for containers; DPAPI for the Windows boxes; **File (shared
mounted key)** for clustered pairs (hard requirement — same KEK on every node).

### G-6 — Mount `/admin/secrets` UI (all three)
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.

### G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib `0.2.0`)
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork was designed as *either/or* with
Option A recommended and Option B deferred. **Both were built** — the fork is now a deployment
choice rather than a library one, and the two packages are independent (an app takes whichever
matches its availability requirement, or neither).

| Package | Topologies | When |
|---|---|---|
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | shared store (Option A) · local + hub | every node can reach a shared DB |
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | peer-to-peer over the cluster (Option B) | a node must resolve while **partitioned** |

Three constraints hold across all of them, and each is documented at the API and in the runbook:
**same KEK on every node** (a mismatch fails closed and looks like corruption); **ciphertext only
crosses trust boundaries**; **re-wraps do not replicate**, so `rewrap-all` runs once per independent
store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no
merge — stated plainly in the README rather than left for an operator to discover.

Design + plan: [`…-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
+ [`…-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md).
Operator runbook: [`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md).

### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
`ZB.MOM.WW.Secrets`. Because the envelope wraps a per-secret DEK under the KEK, rotation **re-wraps
DEKs only** — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster
LWW). Shipped: `ISecretCipher.Rewrap(row, oldKek, newKek)` (fail-closed on wrong old KEK; DEK zeroed
on all paths; also catches malformed-length wraps → `SecretDecryptionException`),
`ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek)` (updates only the 4 wrap
columns + `kek_id`, **compare-and-swap on the current wrapped DEK** so a concurrent write can't
corrupt a row — review-caught TOCTOU), `KekRotationService.RewrapAllAsync` + `RewrapReport`
(enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK), the `secret rewrap-all` CLI verb (key material only via
env-var-name/file-path — never a literal), README section, and the operator runbook
[`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`](../../ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md).
**Verified:** full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke
(migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard).
Version bumped `0.1.2`→`0.1.3` (additive interface members); **republish to the Gitea feed is a
separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive).** Reviewed
by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent-
write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.

## Out of scope (this component)

- Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening
  concern, not secret storage.
- SQL-Server `ISecretStore` provider construction (seam supports it; build with G-7).