Joseph Doherty
3dd7aa40ff
Publishes the three ZB.MOM.WW.LocalDb packages to the Gitea feed (restore-verified from a scratch consumer) and adds the build/push.sh the other shared libs already have. The status prose across CLAUDE.md, README.md and components/*/GAPS.md had drifted from reality, so it was re-derived from the feed listing and the actual PackageReferences + registration calls on each consumer's default branch rather than from prior claims. Five claims were false: Health "not yet adopted" (all four apps wire MapZbHealth), GalaxyRepository's mxaccessgw adoption "a follow-on" (its Server wires AddZbGalaxyRepository), Configuration "not yet pushed", Secrets G-8 "not yet committed", and Theme pinned at 0.2.0 (all four are on 0.3.1). Every doc also said "three apps" while HistorianGateway is a fourth consumer of seven libs, and all eight libraries' test counts were stale (re-ran each suite; all green). Surfaces one previously unrecorded gap: Secrets source is at 0.1.3 with KEK rotation committed, but the feed tops out at 0.1.2, so no app can consume rotation until it is published. Health and observability divergence tables are labelled historical, not re-verified — the libraries are proven wired, but per-app probe coverage vs spec was not re-walked. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
177 lines
17 KiB
Markdown
177 lines
17 KiB
Markdown
# Secrets — GAPS (adoption backlog)
|
||
|
||
The delta between each project's [`current-state`](current-state/) and the
|
||
[`SPEC`](spec/SPEC.md), as prioritized work. The library itself is **built + published +
|
||
reference-consumer-proven**; everything here is per-app adoption of an existing lib (not lib
|
||
construction).
|
||
|
||
## Current state (verified 2026-07-18)
|
||
|
||
- **Source is at `0.1.3`; the feed is at `0.1.2`.** `ZB.MOM.WW.Secrets/Directory.Build.props`
|
||
declares `<Version>0.1.3</Version>` and the G-8 KEK-rotation work (`KekRotationService`,
|
||
`RewrapReport`, `secret rewrap-all`, `docs/operations/kek-rotation.md`) is **committed and
|
||
pushed** on `main` (`d82d345`). The Gitea feed carries only `0.1.0`–`0.1.2` for all three
|
||
packages. **→ `0.1.3` is unpublished: no app can consume KEK rotation yet.** (An earlier note
|
||
claiming G-8 was "not yet committed/pushed" was stale — it is committed; only the *publish*
|
||
is outstanding.)
|
||
- **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take
|
||
`Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`.
|
||
- **G-7 clustered replication is designed + planned, not built** — plan at
|
||
`docs/plans/2026-07-17-secrets-g7-*` (resolved to build the shared SQL-Server `ISecretStore`;
|
||
Akka replicator deferred to phase 2). No `ISecretStore` SQL-Server implementation exists yet.
|
||
- Library suite: **97 passing, 1 skipped**, green 2026-07-18.
|
||
|
||
### Open
|
||
|
||
- ⬜ Publish `ZB.MOM.WW.Secrets` `0.1.3` (+ `.Abstractions`, `.Ui`) to the Gitea feed.
|
||
- ⬜ Execute the G-7 plan (shared SQL-Server `ISecretStore`).
|
||
- ⬜ Per-app live wonder gates for the G-8 rotation path.
|
||
|
||
## Execution status (2026-07-16)
|
||
|
||
- **mxaccessgw — G-4 + G-5 + G-6 DONE.** Executed via the plan below (8 tasks, subagent-driven,
|
||
classification-driven reviews), **merged + pushed to `origin/main` (mxaccessgw @ `e088dfa`)**.
|
||
Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker),
|
||
behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev
|
||
(real .NET 10.0.8): fail-closed on unseeded `${secret:}` (`SecretNotFoundException`, refuses
|
||
to start) + CLI seed/`get` decrypt round-trip + **zero plaintext in the SQLite store at rest**.
|
||
(`/admin/secrets` interactive reveal not re-proven on this box — same shared `.Ui` component
|
||
already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real
|
||
LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP
|
||
password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth),
|
||
and the live-LDAP integration suite now honors the `MxGateway__Ldap__ServiceAccountPassword`
|
||
env override. **G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.**
|
||
- **ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE.** Executed via the plan below (10 tasks, subagent-driven,
|
||
classification-driven reviews), **FF-merged + pushed to `origin/main` (ScadaBridge @ `128f1596`)**.
|
||
Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181,
|
||
CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes
|
||
isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra
|
||
all green). **G-3 LIVE-PROVEN vs the REAL production MxGateway gateway** (`wonder-app-vd03.zmr.zimmer.com:5120`):
|
||
a throwaway harness drove the real `MxGatewayDataConnection` through the DI factory + `AddZbSecrets` with a
|
||
`secret:`-ref `ApiKey` → **dummy key `Unauthenticated`** (proves the resolved value is transmitted, not
|
||
fail-closed first) → **real read-scoped key → `Connected` + browsed the real Galaxy root (10 nodes)**. Temp
|
||
key created via the gateway dashboard, then **revoked + deleted + functionally confirmed dead**. Notable
|
||
deltas from the plan: T3 registers `AddZbSecrets` on **both** the Central branch (UI) and
|
||
`SiteServiceRegistration` (DCL adapter) — the plan assumed one container; T6 claim-type resolved **MATCH**
|
||
(`ZbClaimTypes.Role == ClaimTypes.Role`, adversarially verified) → `AddSecretsAuthorization()` used directly;
|
||
T8 (G-5) shipped as a **docs-only** production-posture runbook (`docs/operations/2026-07-16-secrets-clustered-master-key.md`)
|
||
because hardcoding `Source=File` + `/shared` paths would break the Central-boot Host.Tests + local dev (the
|
||
pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking
|
||
follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single `[Collection]` / disable
|
||
parallelization for Host.Tests).
|
||
- **OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted).** Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), **FF-merged + pushed to `origin/master` (lmxopcua @ `872cf7e3`, `ec6598ce..872cf7e3`)**. Clean build (0 errors). The full offline suite caught **one real regression** — `AddOtOpcUaDriverFactories`' registration now resolves `ISecretResolver` (Galaxy/OpcUaClient secret: refs), so two `ResilienceInvokerFactoryRegistrationTests` that built a minimal container threw `No service for ISecretResolver`; fixed test-only by registering a stub resolver (production always has it via the unconditional `AddZbSecrets`, and `GetRequiredService` correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental `AbCip_Green_AgainstSim` sim-fixture test — zero AbCip code touched). **Three deliberate deviations from the plan's literal text** (verified corrections, same class as the CPM miss the plan itself caught): (1) **G-4 does NOT commit a `${secret:}` token into the default appsettings** — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is `ServerHistorian:ApiKey=""` (empty, disabled-by-default), so a committed token would break every dev/CI/`TwoNodeClusterHarness` boot for zero at-rest benefit; instead documented the token-delivery convention (`docs/security.md`) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → `SecretNotFoundException` at pre-host expansion; seeded → resolves). (2) **G-5 shipped docs-only** (`docs/operations/2026-07-16-secrets-clustered-master-key.md`) — hardcoding `Source=File`+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays `Source=Environment`. Mirrors the ScadaBridge G-5 resolution. (3) **Task 8 resolves in the factory/driver only, NOT the probe** — the v3 `OpcUaClientDriverProbe` is unauthenticated `GetEndpoints`-only and never reads `Password`/`UserCertificatePassword`, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (`GalaxyDriver.BuildClientOptionsAsync` / `OpcUaClientDriver.InitializeAsync`), threaded via ctor + `DriverFactoryBootstrap` (real resolver from DI; `NullSecretResolver` null-object backs test/parse paths only, fail-closed on `secret:`). `OpcUaClientDriverOptions` was converted `sealed class` → `sealed record` for the `with`-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: `DriverConfig` persists as an opaque JSON string via `RawTreeService`, which has NO resolver dependency. **Live wonder gate PROVEN 2026-07-16 (VPN up):** a throwaway harness drove OtOpcUa's real `GalaxyDriverBrowser` (resolves the `secret:` `ApiKeySecretRef` via `ISecretResolver` → `GalaxySecretRef.ResolveApiKeyAsync` → connects) against the REAL production MxGateway `wonder-app-vd03:5120`. **Dummy `secret:` ref → `MxGatewayAuthenticationException` (proves the resolved value is transmitted, not fail-closed-first); real `metadata:read` key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…).** Temp key minted via the gateway dashboard, then **revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted** (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. **All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw.**
|
||
**Plan re-verified 2026-07-16 against `origin/master` @ `ec6598ce`** (the merged **v3.0 dual-namespace** rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa **is** on Central Package Management (was mis-stated as inline-versioned), `OpcUaClientDriverOptions` lives in `…Driver.OpcUaClient.Contracts` + `GalaxyDriverBrowser` in `…Driver.Galaxy.Browser`, and `DriverHostActor.cs` line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
|
||
- **G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17).**
|
||
G-8 shipped into `ZB.MOM.WW.Secrets` (lib `0.1.3`): the `Rewrap` DEK primitive, CAS-guarded
|
||
`ApplyRewrapAsync`, `KekRotationService.RewrapAllAsync`, the `secret rewrap-all` CLI verb, and an
|
||
operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial
|
||
crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka
|
||
fork is resolved (build the shared SQL-Server `ISecretStore`; Akka replicator deferred phase-2)
|
||
with a design doc + executable plan + `.tasks.json`, **no code built**. See the G-7/G-8 backlog
|
||
entries below. Not yet committed/pushed (awaiting go-ahead).
|
||
|
||
## Design + implementation plans (2026-07-16)
|
||
|
||
G-2 … G-6 are now planned. Shared design + three per-repo executable plans (task-metadata'd,
|
||
code-verified anchors, co-located `.tasks.json`):
|
||
|
||
- [`docs/plans/2026-07-16-secrets-adoption-design.md`](../../docs/plans/2026-07-16-secrets-adoption-design.md)
|
||
— shared design: library API, the two resolution layers (pre-host `${secret:}` config
|
||
expander vs runtime `ISecretResolver`), wiring template, master-key/clustering fork.
|
||
- [`docs/plans/2026-07-16-secrets-adoption-otopcua.md`](../../docs/plans/2026-07-16-secrets-adoption-otopcua.md)
|
||
— **G-2 + G-4 + G-5 + G-6** (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
|
||
- [`docs/plans/2026-07-16-secrets-adoption-scadabridge.md`](../../docs/plans/2026-07-16-secrets-adoption-scadabridge.md)
|
||
— **G-3 + G-4 + G-5 + G-6** (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
|
||
- [`docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md`](../../docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md)
|
||
— **G-4 + G-5 + G-6** (8 tasks; single-box, no Layer B).
|
||
|
||
**G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED** (2026-07-17, below).
|
||
|
||
## Cross-cutting observations
|
||
|
||
- **No app has `${secret:}` resolution today.** All three assemble config with the stock
|
||
`AddJsonFile`+`AddEnvironmentVariables` chain and read secrets verbatim. Two already lean
|
||
on the *pattern* the expander formalizes: ScadaBridge's non-functional `${SCADABRIDGE_*}`
|
||
placeholders (whole-key env override) and OtOpcUa's driver-level `GalaxySecretRef`
|
||
(`env:`/`file:`/`dev:`/literal). These are the natural first swaps.
|
||
- **The common high-value gap is plaintext credentials in appsettings/env:** LDAP
|
||
service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb,
|
||
ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and
|
||
API-key peppers (all three, runtime-supplied but plaintext).
|
||
- **Secrets already stored in a DB in plaintext are the sharpest edge:** OtOpcUa's OpcUaClient
|
||
`Password`/`UserCertificatePassword` and the `dev:`/literal Galaxy API key live cleartext in
|
||
the central config DB; ScadaBridge's MxGateway per-endpoint `ApiKey` lives plaintext in the
|
||
ConfigDb `DataConnections` JSON. These leak at rest to anyone with DB read.
|
||
- **Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets**
|
||
(4 encrypted ConfigDb columns via `EncryptedStringConverter`). OtOpcUa/mxaccessgw use Data
|
||
Protection only for cookies/tokens. None use DPAPI or `ProtectKeysWith*`; key rings are
|
||
unencrypted-at-rest. `ZB.MOM.WW.Secrets` adds envelope encryption with a KEK held **outside**
|
||
the store — a stronger boundary than "the DB is the only protection."
|
||
- **Peppered-HMAC API-key stores stay bespoke** (mxaccessgw + ScadaBridge): they are hashed,
|
||
not reversibly stored — already correct. Secrets only takes over the *pepper value* supply.
|
||
|
||
## Backlog (prioritized)
|
||
|
||
### G-1 — HistorianGateway reference consumer ✅ DONE
|
||
Adopted + live-proven 2026-07-16 (historian password via `${secret:}`, authenticated read
|
||
against the real wonder historian). This is the wiring template for the three apps.
|
||
|
||
### G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)
|
||
Add a `secret:` arm to `GalaxySecretRef` (its own comment anticipates this) and to the
|
||
OpcUaClient driver options, resolving through `ISecretResolver` — retire the `dev:`/literal
|
||
and plaintext `Password`/`UserCertificatePassword` DB paths.
|
||
|
||
### G-3 — ScadaBridge: MxGateway `ApiKey` plaintext-in-ConfigDb
|
||
Resolve via `ISecretResolver` or extend the existing `EncryptedStringConverter` to that JSON
|
||
column.
|
||
|
||
### G-4 — Pre-host `${secret:}` for plaintext config secrets (all three)
|
||
LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa),
|
||
peppers — swap to `${secret:…}` tokens, expander running **before** each app's existing
|
||
validator (`ConfigPreflight` / `Ldap`+`OpcUa` validators / `GatewayOptionsValidator`).
|
||
Delete committed dev plaintext and the loose `*_login.txt` files (ScadaBridge).
|
||
|
||
### G-5 — Master-key provider per deployment
|
||
Env (`ZB_SECRETS_MASTER_KEY`) for containers; DPAPI for the Windows boxes; **File (shared
|
||
mounted key)** for clustered pairs (hard requirement — same KEK on every node).
|
||
|
||
### G-6 — Mount `/admin/secrets` UI (all three)
|
||
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
|
||
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.
|
||
|
||
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17)
|
||
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork is **resolved: build Option A —
|
||
a shared SQL-Server `ISecretStore`** (both clustered apps already run shared SQL + a shared
|
||
Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure
|
||
modes); the Akka replicator (`ZB.MOM.WW.Secrets.Akka`, LWW/anti-entropy/tombstones) is a **deferred
|
||
phase-2** to build only on a stated partition-tolerance requirement. Both require a shared KEK; the
|
||
`ISecretReplicator` seam stays in place so Option B remains a drop-in later. Design + executable
|
||
plan (+ `.tasks.json`): [`docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
|
||
+ [`docs/plans/2026-07-17-secrets-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). **No code built yet — plan is ready to execute.**
|
||
|
||
### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
|
||
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
|
||
`ZB.MOM.WW.Secrets`. Because the envelope wraps a per-secret DEK under the KEK, rotation **re-wraps
|
||
DEKs only** — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster
|
||
LWW). Shipped: `ISecretCipher.Rewrap(row, oldKek, newKek)` (fail-closed on wrong old KEK; DEK zeroed
|
||
on all paths; also catches malformed-length wraps → `SecretDecryptionException`),
|
||
`ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek)` (updates only the 4 wrap
|
||
columns + `kek_id`, **compare-and-swap on the current wrapped DEK** so a concurrent write can't
|
||
corrupt a row — review-caught TOCTOU), `KekRotationService.RewrapAllAsync` + `RewrapReport`
|
||
(enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
|
||
fail-closed on unknown/identical KEK), the `secret rewrap-all` CLI verb (key material only via
|
||
env-var-name/file-path — never a literal), README section, and the operator runbook
|
||
[`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`](../../ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md).
|
||
**Verified:** full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke
|
||
(migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard).
|
||
Version bumped `0.1.2`→`0.1.3` (additive interface members); **republish to the Gitea feed is a
|
||
separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive).** Reviewed
|
||
by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent-
|
||
write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.
|
||
|
||
## Out of scope (this component)
|
||
|
||
- Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening
|
||
concern, not secret storage.
|
||
- SQL-Server `ISecretStore` provider construction (seam supports it; build with G-7).
|