scadaproj/components/secrets/GAPS.md

Secrets — GAPS (adoption backlog)

The delta between each project's current-state and the SPEC, as prioritized work. The library itself is built + published + reference-consumer-proven; everything here is per-app adoption of an existing lib (not lib construction).

Current state (verified 2026-07-18)

• 0.1.3 PUBLISHED to the Gitea feed 2026-07-18 (all three packages; restore-verified from a scratch consumer). Carries the G-8 KEK-rotation work (KekRotationService, RewrapReport, secret rewrap-all, docs/operations/kek-rotation.md), committed on main at d82d345. (An earlier note claiming G-8 was "not yet committed/pushed" was stale.)
• 0.1.3 also carries a transitive security fix that 0.1.2 does not. Packing surfaced NU1903: Microsoft.Data.Sqlite 10.0.7 pulls SQLitePCLRaw.lib.e_sqlite3 2.1.11, which has a high-severity advisory (GHSA-2m69-gcr7-jv3q). Fixed the way ZB.MOM.WW.LocalDb already did — CentralPackageTransitivePinningEnabled + a pin to the patched 2.1.12. Bumping Microsoft.Data.Sqlite does not help (even 10.0.10 still pulls 2.1.11); the pin is the fix. Verified: the shipped .nuspec declares SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12 as a direct dependency, so consumers get the patched native lib without needing their own pin, and a fresh consumer restoring 0.1.3 scans clean.
• All four apps are still pinned at 0.1.2 — they get neither KEK rotation nor the security fix until bumped.
• All four apps are pinned to 0.1.2 — OtOpcUa, MxAccessGateway, ScadaBridge take Secrets + .Abstractions + .Ui; HistorianGateway takes Secrets + .Ui.
• G-7 clustered replication is designed + planned, not built — plan at docs/plans/2026-07-17-secrets-g7-* (resolved to build the shared SQL-Server ISecretStore; Akka replicator deferred to phase 2). No ISecretStore SQL-Server implementation exists yet.
• Library suite: 97 passing, 1 skipped, green 2026-07-18.

Open

• ⬜ Bump the four apps 0.1.2 → 0.1.3 to pick up KEK rotation + the SQLitePCLRaw security fix.
• ⬜ Execute the G-7 plan (shared SQL-Server ISecretStore).
• ⬜ Per-app live wonder gates for the G-8 rotation path.

Execution status (2026-07-16)

• mxaccessgw — G-4 + G-5 + G-6 DONE. Executed via the plan below (8 tasks, subagent-driven, classification-driven reviews), merged + pushed to origin/main (mxaccessgw @ e088dfa). Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker), behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev (real .NET 10.0.8): fail-closed on unseeded ${secret:} (SecretNotFoundException, refuses to start) + CLI seed/get decrypt round-trip + zero plaintext in the SQLite store at rest. (/admin/secrets interactive reveal not re-proven on this box — same shared .Ui component already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth), and the live-LDAP integration suite now honors the MxGateway__Ldap__ServiceAccountPassword env override. G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.
• ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE. Executed via the plan below (10 tasks, subagent-driven, classification-driven reviews), FF-merged + pushed to origin/main (ScadaBridge @ 128f1596). Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181, CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra all green). G-3 LIVE-PROVEN vs the REAL production MxGateway gateway (wonder-app-vd03.zmr.zimmer.com:5120): a throwaway harness drove the real MxGatewayDataConnection through the DI factory + AddZbSecrets with a secret:-ref ApiKey → dummy key Unauthenticated (proves the resolved value is transmitted, not fail-closed first) → real read-scoped key → Connected + browsed the real Galaxy root (10 nodes). Temp key created via the gateway dashboard, then revoked + deleted + functionally confirmed dead. Notable deltas from the plan: T3 registers AddZbSecrets on both the Central branch (UI) and SiteServiceRegistration (DCL adapter) — the plan assumed one container; T6 claim-type resolved MATCH (ZbClaimTypes.Role == ClaimTypes.Role, adversarially verified) → AddSecretsAuthorization() used directly; T8 (G-5) shipped as a docs-only production-posture runbook (docs/operations/2026-07-16-secrets-clustered-master-key.md) because hardcoding Source=File + /shared paths would break the Central-boot Host.Tests + local dev (the pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single [Collection] / disable parallelization for Host.Tests).
• OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted). Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), FF-merged + pushed to origin/master (lmxopcua @ 872cf7e3, ec6598ce..872cf7e3). Clean build (0 errors). The full offline suite caught one real regression — AddOtOpcUaDriverFactories' registration now resolves ISecretResolver (Galaxy/OpcUaClient secret: refs), so two ResilienceInvokerFactoryRegistrationTests that built a minimal container threw No service for ISecretResolver; fixed test-only by registering a stub resolver (production always has it via the unconditional AddZbSecrets, and GetRequiredService correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental AbCip_Green_AgainstSim sim-fixture test — zero AbCip code touched). Three deliberate deviations from the plan's literal text (verified corrections, same class as the CPM miss the plan itself caught): (1) G-4 does NOT commit a ${secret:} token into the default appsettings — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is ServerHistorian:ApiKey="" (empty, disabled-by-default), so a committed token would break every dev/CI/TwoNodeClusterHarness boot for zero at-rest benefit; instead documented the token-delivery convention (docs/security.md) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → SecretNotFoundException at pre-host expansion; seeded → resolves). (2) G-5 shipped docs-only (docs/operations/2026-07-16-secrets-clustered-master-key.md) — hardcoding Source=File+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays Source=Environment. Mirrors the ScadaBridge G-5 resolution. (3) Task 8 resolves in the factory/driver only, NOT the probe — the v3 OpcUaClientDriverProbe is unauthenticated GetEndpoints-only and never reads Password/UserCertificatePassword, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (GalaxyDriver.BuildClientOptionsAsync / OpcUaClientDriver.InitializeAsync), threaded via ctor + DriverFactoryBootstrap (real resolver from DI; NullSecretResolver null-object backs test/parse paths only, fail-closed on secret:). OpcUaClientDriverOptions was converted sealed class → sealed record for the with-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: DriverConfig persists as an opaque JSON string via RawTreeService, which has NO resolver dependency. Live wonder gate PROVEN 2026-07-16 (VPN up): a throwaway harness drove OtOpcUa's real GalaxyDriverBrowser (resolves the secret: ApiKeySecretRef via ISecretResolver → GalaxySecretRef.ResolveApiKeyAsync → connects) against the REAL production MxGateway wonder-app-vd03:5120. Dummy secret: ref → MxGatewayAuthenticationException (proves the resolved value is transmitted, not fail-closed-first); real metadata:read key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…). Temp key minted via the gateway dashboard, then revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw. Plan re-verified 2026-07-16 against origin/master @ ec6598ce (the merged v3.0 dual-namespace rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa is on Central Package Management (was mis-stated as inline-versioned), OpcUaClientDriverOptions lives in …Driver.OpcUaClient.Contracts + GalaxyDriverBrowser in …Driver.Galaxy.Browser, and DriverHostActor.cs line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
• G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17). G-8 shipped into ZB.MOM.WW.Secrets (lib 0.1.3): the Rewrap DEK primitive, CAS-guarded ApplyRewrapAsync, KekRotationService.RewrapAllAsync, the secret rewrap-all CLI verb, and an operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka fork is resolved (build the shared SQL-Server ISecretStore; Akka replicator deferred phase-2) with a design doc + executable plan + .tasks.json, no code built. See the G-7/G-8 backlog entries below. Not yet committed/pushed (awaiting go-ahead).

Design + implementation plans (2026-07-16)

G-2 … G-6 are now planned. Shared design + three per-repo executable plans (task-metadata'd, code-verified anchors, co-located .tasks.json):

• docs/plans/2026-07-16-secrets-adoption-design.md — shared design: library API, the two resolution layers (pre-host ${secret:} config expander vs runtime ISecretResolver), wiring template, master-key/clustering fork.
• docs/plans/2026-07-16-secrets-adoption-otopcua.md — G-2 + G-4 + G-5 + G-6 (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
• docs/plans/2026-07-16-secrets-adoption-scadabridge.md — G-3 + G-4 + G-5 + G-6 (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
• docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md — G-4 + G-5 + G-6 (8 tasks; single-box, no Layer B).

G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED (2026-07-17, below).

Cross-cutting observations

• No app has ${secret:} resolution today. All three assemble config with the stock AddJsonFile+AddEnvironmentVariables chain and read secrets verbatim. Two already lean on the pattern the expander formalizes: ScadaBridge's non-functional ${SCADABRIDGE_*} placeholders (whole-key env override) and OtOpcUa's driver-level GalaxySecretRef (env:/file:/dev:/literal). These are the natural first swaps.
• The common high-value gap is plaintext credentials in appsettings/env: LDAP service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb, ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and API-key peppers (all three, runtime-supplied but plaintext).
• Secrets already stored in a DB in plaintext are the sharpest edge: OtOpcUa's OpcUaClient Password/UserCertificatePassword and the dev:/literal Galaxy API key live cleartext in the central config DB; ScadaBridge's MxGateway per-endpoint ApiKey lives plaintext in the ConfigDb DataConnections JSON. These leak at rest to anyone with DB read.
• Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets (4 encrypted ConfigDb columns via EncryptedStringConverter). OtOpcUa/mxaccessgw use Data Protection only for cookies/tokens. None use DPAPI or ProtectKeysWith*; key rings are unencrypted-at-rest. ZB.MOM.WW.Secrets adds envelope encryption with a KEK held outside the store — a stronger boundary than "the DB is the only protection."
• Peppered-HMAC API-key stores stay bespoke (mxaccessgw + ScadaBridge): they are hashed, not reversibly stored — already correct. Secrets only takes over the pepper value supply.

Backlog (prioritized)

G-1 — HistorianGateway reference consumer ✅ DONE

Adopted + live-proven 2026-07-16 (historian password via ${secret:}, authenticated read against the real wonder historian). This is the wiring template for the three apps.

G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)

Add a secret: arm to GalaxySecretRef (its own comment anticipates this) and to the OpcUaClient driver options, resolving through ISecretResolver — retire the dev:/literal and plaintext Password/UserCertificatePassword DB paths.

G-3 — ScadaBridge: MxGateway ApiKey plaintext-in-ConfigDb

Resolve via ISecretResolver or extend the existing EncryptedStringConverter to that JSON column.

G-4 — Pre-host ${secret:} for plaintext config secrets (all three)

LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa), peppers — swap to ${secret:…} tokens, expander running before each app's existing validator (ConfigPreflight / Ldap+OpcUa validators / GatewayOptionsValidator). Delete committed dev plaintext and the loose *_login.txt files (ScadaBridge).

G-5 — Master-key provider per deployment

Env (ZB_SECRETS_MASTER_KEY) for containers; DPAPI for the Windows boxes; File (shared mounted key) for clustered pairs (hard requirement — same KEK on every node).

G-6 — Mount /admin/secrets UI (all three)

Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI) and map each app's admin role onto secrets:manage / secrets:reveal.

G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17)

The SPEC's "shared SQL-Server store vs Akka replicator" fork is resolved: build Option A — a shared SQL-Server ISecretStore (both clustered apps already run shared SQL + a shared Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure modes); the Akka replicator (ZB.MOM.WW.Secrets.Akka, LWW/anti-entropy/tombstones) is a deferred phase-2 to build only on a stated partition-tolerance requirement. Both require a shared KEK; the ISecretReplicator seam stays in place so Option B remains a drop-in later. Design + executable plan (+ .tasks.json): docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md

• docs/plans/2026-07-17-secrets-g7-sqlserver-store.md. No code built yet — plan is ready to execute.

G-8 — KEK-rotation RewrapAll + runbook ✅ BUILT (2026-07-17, lib 0.1.3)

The RewrapAll(oldKek, newKek) admin primitive + operator runbook are built + fully tested in ZB.MOM.WW.Secrets. Because the envelope wraps a per-secret DEK under the KEK, rotation re-wraps DEKs only — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster LWW). Shipped: ISecretCipher.Rewrap(row, oldKek, newKek) (fail-closed on wrong old KEK; DEK zeroed on all paths; also catches malformed-length wraps → SecretDecryptionException), ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek) (updates only the 4 wrap columns + kek_id, compare-and-swap on the current wrapped DEK so a concurrent write can't corrupt a row — review-caught TOCTOU), KekRotationService.RewrapAllAsync + RewrapReport (enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry, fail-closed on unknown/identical KEK), the secret rewrap-all CLI verb (key material only via env-var-name/file-path — never a literal), README section, and the operator runbook ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md. Verified: full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke (migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard). Version bumped 0.1.2→0.1.3 (additive interface members); republish to the Gitea feed is a separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive). Reviewed by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent- write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.

Out of scope (this component)

• Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening concern, not secret storage.
• SQL-Server ISecretStore provider construction (seam supports it; build with G-7).