scadaproj/components/secrets/GAPS.md

Secrets — GAPS (adoption backlog)

The delta between each project's current-state and the SPEC, as prioritized work. The library itself is built + published + reference-consumer-proven; everything here is per-app adoption of an existing lib (not lib construction).

Current state (verified 2026-07-18)

• 0.1.3 PUBLISHED to the Gitea feed 2026-07-18 (all three packages; restore-verified from a scratch consumer). Carries the G-8 KEK-rotation work (KekRotationService, RewrapReport, secret rewrap-all, docs/operations/kek-rotation.md), committed on main at d82d345. (An earlier note claiming G-8 was "not yet committed/pushed" was stale.)
• 0.1.3 also carries a transitive security fix that 0.1.2 does not. Packing surfaced NU1903: Microsoft.Data.Sqlite 10.0.7 pulls SQLitePCLRaw.lib.e_sqlite3 2.1.11, which has a high-severity advisory (GHSA-2m69-gcr7-jv3q). Fixed the way ZB.MOM.WW.LocalDb already did — CentralPackageTransitivePinningEnabled + a pin to the patched 2.1.12. Bumping Microsoft.Data.Sqlite does not help (even 10.0.10 still pulls 2.1.11); the pin is the fix. Verified: the shipped .nuspec declares SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12 as a direct dependency, so consumers get the patched native lib without needing their own pin, and a fresh consumer restoring 0.1.3 scans clean.
• All four apps are still pinned at 0.1.2 — they get neither KEK rotation nor the security fix until bumped.
• All four apps are pinned to 0.1.2 — OtOpcUa, MxAccessGateway, ScadaBridge take Secrets + .Abstractions + .Ui; HistorianGateway takes Secrets + .Ui.
• G-7 clustered replication is BUILT (2026-07-18, lib 0.2.0) — both options, not just one. The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build covers both, as two new packages:
  • ZB.MOM.WW.Secrets.Replicator.SqlServer — SqlServerSecretStore : ISecretStore (all 7 members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and two topologies: AddZbSecretsSqlServerStore (one shared store, Option A) and AddZbSecretsSqlServerReplication (local store + hub, with a bidirectional sweep).
  • ZB.MOM.WW.Secrets.Replicator.AkkaDotNet — peer-to-peer over distributed pub/sub (Option B): live broadcast + periodic manifest anti-entropy, an explicit SerializerWithStringManifest, ciphertext-only wire DTOs.
  • Core gained the pieces both need: ISecretsStoreMigrator (so the hosted service + CLI are store-agnostic), SecretLastWriterWins (one definition of the LWW tie-break, shared by every store and reconciler), SecretReplicationReconciler, and ReplicatingSecretStore.
  • The ISecretReplicator seam had no caller — nothing invoked PublishAsync, so local writes would never have propagated. ReplicatingSecretStore decorates the store, which fixes it for the UI, the CLI, and any future write path at once.
  • Naming: the user chose .Replicator.* (not the plan's .Replication.* / ZB.MOM.WW.Secrets.Akka); AkkaDotNet rather than Akka also avoids a namespace collision with the Akka root namespace.
• Library suite: 182 passing, 1 skipped, green 2026-07-18 — including 15 live SQL-Server tests against a real SQL Server 2022 (the SQLite store's suite ported case-for-case, so any behavioural divergence between the two stores fails a test) and a 9-test in-process 2-node Akka cluster (real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up, and malformed-peer-input handling.

Open

• ⬜ Bump the four apps 0.1.2 → 0.1.3/0.2.0 to pick up KEK rotation + the SQLitePCLRaw security fix.
• ⬜ Publish 0.2.0 (5 packages) to the Gitea feed — packed and scanned clean, not yet pushed.
• ⬜ Adopt a topology in ScadaBridge / OtOpcUa (config + shared KEK; no code change needed).
• ⬜ Per-app live wonder gates for the G-8 rotation path.
• ⬜ Akka self-echo forwarding is only observable with three nodes; the 2-node rig covers the drift symptom, not the forwarding filter itself (noted in the test).

Post-build code review (2026-07-18) — 6 defects found and fixed

An independent review caught real bugs that a green suite had not:

• CRITICAL — both replication modes were dead on arrival. The decorators resolve the undecorated local store by concrete type, but core registered SqliteSecretStore only behind ISecretStore. Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with No service for type 'SqliteSecretStore'), fixed, and covered by new DI-resolution tests.
• Unbounded fetch. A cold-starting node pulls the peer's entire inventory in one call; past ~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the node never converges. Now chunked at 500.
• Poison row aborted the batch. One unapplyable row abandoned every row after it, and since the pull set recomputes in the same order each sweep, it would poison the same batch forever. Now isolated per row and logged.
• Enum.Parse on peer input. An overflowing numeric string throws OverflowException, which the actor's ArgumentException filter did not catch → actor restart loop on redelivery; and any in-range number was accepted as an undefined enum value. Now TryParse + IsDefined.
• Null crypto blobs passed the boundary (required byte[] means present, not non-null).
• A failed pull-read replied with silence — the peer waited forever with no diagnostic.

One reported "critical" was a false positive: Self used after await in the actor. Akka's ActorBase caches Self in an instance field (unlike Context, which is [ThreadStatic]), and three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as written.

Execution status (2026-07-16)

• mxaccessgw — G-4 + G-5 + G-6 DONE. Executed via the plan below (8 tasks, subagent-driven, classification-driven reviews), merged + pushed to origin/main (mxaccessgw @ e088dfa). Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker), behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev (real .NET 10.0.8): fail-closed on unseeded ${secret:} (SecretNotFoundException, refuses to start) + CLI seed/get decrypt round-trip + zero plaintext in the SQLite store at rest. (/admin/secrets interactive reveal not re-proven on this box — same shared .Ui component already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth), and the live-LDAP integration suite now honors the MxGateway__Ldap__ServiceAccountPassword env override. G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.
• ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE. Executed via the plan below (10 tasks, subagent-driven, classification-driven reviews), FF-merged + pushed to origin/main (ScadaBridge @ 128f1596). Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181, CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra all green). G-3 LIVE-PROVEN vs the REAL production MxGateway gateway (wonder-app-vd03.zmr.zimmer.com:5120): a throwaway harness drove the real MxGatewayDataConnection through the DI factory + AddZbSecrets with a secret:-ref ApiKey → dummy key Unauthenticated (proves the resolved value is transmitted, not fail-closed first) → real read-scoped key → Connected + browsed the real Galaxy root (10 nodes). Temp key created via the gateway dashboard, then revoked + deleted + functionally confirmed dead. Notable deltas from the plan: T3 registers AddZbSecrets on both the Central branch (UI) and SiteServiceRegistration (DCL adapter) — the plan assumed one container; T6 claim-type resolved MATCH (ZbClaimTypes.Role == ClaimTypes.Role, adversarially verified) → AddSecretsAuthorization() used directly; T8 (G-5) shipped as a docs-only production-posture runbook (docs/operations/2026-07-16-secrets-clustered-master-key.md) because hardcoding Source=File + /shared paths would break the Central-boot Host.Tests + local dev (the pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single [Collection] / disable parallelization for Host.Tests).
• OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted). Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), FF-merged + pushed to origin/master (lmxopcua @ 872cf7e3, ec6598ce..872cf7e3). Clean build (0 errors). The full offline suite caught one real regression — AddOtOpcUaDriverFactories' registration now resolves ISecretResolver (Galaxy/OpcUaClient secret: refs), so two ResilienceInvokerFactoryRegistrationTests that built a minimal container threw No service for ISecretResolver; fixed test-only by registering a stub resolver (production always has it via the unconditional AddZbSecrets, and GetRequiredService correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental AbCip_Green_AgainstSim sim-fixture test — zero AbCip code touched). Three deliberate deviations from the plan's literal text (verified corrections, same class as the CPM miss the plan itself caught): (1) G-4 does NOT commit a ${secret:} token into the default appsettings — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is ServerHistorian:ApiKey="" (empty, disabled-by-default), so a committed token would break every dev/CI/TwoNodeClusterHarness boot for zero at-rest benefit; instead documented the token-delivery convention (docs/security.md) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → SecretNotFoundException at pre-host expansion; seeded → resolves). (2) G-5 shipped docs-only (docs/operations/2026-07-16-secrets-clustered-master-key.md) — hardcoding Source=File+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays Source=Environment. Mirrors the ScadaBridge G-5 resolution. (3) Task 8 resolves in the factory/driver only, NOT the probe — the v3 OpcUaClientDriverProbe is unauthenticated GetEndpoints-only and never reads Password/UserCertificatePassword, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (GalaxyDriver.BuildClientOptionsAsync / OpcUaClientDriver.InitializeAsync), threaded via ctor + DriverFactoryBootstrap (real resolver from DI; NullSecretResolver null-object backs test/parse paths only, fail-closed on secret:). OpcUaClientDriverOptions was converted sealed class → sealed record for the with-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: DriverConfig persists as an opaque JSON string via RawTreeService, which has NO resolver dependency. Live wonder gate PROVEN 2026-07-16 (VPN up): a throwaway harness drove OtOpcUa's real GalaxyDriverBrowser (resolves the secret: ApiKeySecretRef via ISecretResolver → GalaxySecretRef.ResolveApiKeyAsync → connects) against the REAL production MxGateway wonder-app-vd03:5120. Dummy secret: ref → MxGatewayAuthenticationException (proves the resolved value is transmitted, not fail-closed-first); real metadata:read key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…). Temp key minted via the gateway dashboard, then revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw. Plan re-verified 2026-07-16 against origin/master @ ec6598ce (the merged v3.0 dual-namespace rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa is on Central Package Management (was mis-stated as inline-versioned), OpcUaClientDriverOptions lives in …Driver.OpcUaClient.Contracts + GalaxyDriverBrowser in …Driver.Galaxy.Browser, and DriverHostActor.cs line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
• G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17). G-8 shipped into ZB.MOM.WW.Secrets (lib 0.1.3): the Rewrap DEK primitive, CAS-guarded ApplyRewrapAsync, KekRotationService.RewrapAllAsync, the secret rewrap-all CLI verb, and an operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka fork is resolved (build the shared SQL-Server ISecretStore; Akka replicator deferred phase-2) with a design doc + executable plan + .tasks.json, no code built. See the G-7/G-8 backlog entries below. Not yet committed/pushed (awaiting go-ahead).

Design + implementation plans (2026-07-16)

G-2 … G-6 are now planned. Shared design + three per-repo executable plans (task-metadata'd, code-verified anchors, co-located .tasks.json):

• docs/plans/2026-07-16-secrets-adoption-design.md — shared design: library API, the two resolution layers (pre-host ${secret:} config expander vs runtime ISecretResolver), wiring template, master-key/clustering fork.
• docs/plans/2026-07-16-secrets-adoption-otopcua.md — G-2 + G-4 + G-5 + G-6 (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
• docs/plans/2026-07-16-secrets-adoption-scadabridge.md — G-3 + G-4 + G-5 + G-6 (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
• docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md — G-4 + G-5 + G-6 (8 tasks; single-box, no Layer B).

G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED (2026-07-17, below).

Cross-cutting observations

• No app has ${secret:} resolution today. All three assemble config with the stock AddJsonFile+AddEnvironmentVariables chain and read secrets verbatim. Two already lean on the pattern the expander formalizes: ScadaBridge's non-functional ${SCADABRIDGE_*} placeholders (whole-key env override) and OtOpcUa's driver-level GalaxySecretRef (env:/file:/dev:/literal). These are the natural first swaps.
• The common high-value gap is plaintext credentials in appsettings/env: LDAP service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb, ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and API-key peppers (all three, runtime-supplied but plaintext).
• Secrets already stored in a DB in plaintext are the sharpest edge: OtOpcUa's OpcUaClient Password/UserCertificatePassword and the dev:/literal Galaxy API key live cleartext in the central config DB; ScadaBridge's MxGateway per-endpoint ApiKey lives plaintext in the ConfigDb DataConnections JSON. These leak at rest to anyone with DB read.
• Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets (4 encrypted ConfigDb columns via EncryptedStringConverter). OtOpcUa/mxaccessgw use Data Protection only for cookies/tokens. None use DPAPI or ProtectKeysWith*; key rings are unencrypted-at-rest. ZB.MOM.WW.Secrets adds envelope encryption with a KEK held outside the store — a stronger boundary than "the DB is the only protection."
• Peppered-HMAC API-key stores stay bespoke (mxaccessgw + ScadaBridge): they are hashed, not reversibly stored — already correct. Secrets only takes over the pepper value supply.

Backlog (prioritized)

G-1 — HistorianGateway reference consumer ✅ DONE

Adopted + live-proven 2026-07-16 (historian password via ${secret:}, authenticated read against the real wonder historian). This is the wiring template for the three apps.

G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)

Add a secret: arm to GalaxySecretRef (its own comment anticipates this) and to the OpcUaClient driver options, resolving through ISecretResolver — retire the dev:/literal and plaintext Password/UserCertificatePassword DB paths.

G-3 — ScadaBridge: MxGateway ApiKey plaintext-in-ConfigDb

Resolve via ISecretResolver or extend the existing EncryptedStringConverter to that JSON column.

G-4 — Pre-host ${secret:} for plaintext config secrets (all three)

LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa), peppers — swap to ${secret:…} tokens, expander running before each app's existing validator (ConfigPreflight / Ldap+OpcUa validators / GatewayOptionsValidator). Delete committed dev plaintext and the loose *_login.txt files (ScadaBridge).

G-5 — Master-key provider per deployment

Env (ZB_SECRETS_MASTER_KEY) for containers; DPAPI for the Windows boxes; File (shared mounted key) for clustered pairs (hard requirement — same KEK on every node).

G-6 — Mount /admin/secrets UI (all three)

Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI) and map each app's admin role onto secrets:manage / secrets:reveal.

G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib 0.2.0)

The SPEC's "shared SQL-Server store vs Akka replicator" fork was designed as either/or with Option A recommended and Option B deferred. Both were built — the fork is now a deployment choice rather than a library one, and the two packages are independent (an app takes whichever matches its availability requirement, or neither).

Package	Topologies	When
ZB.MOM.WW.Secrets.Replicator.SqlServer	shared store (Option A) · local + hub	every node can reach a shared DB
ZB.MOM.WW.Secrets.Replicator.AkkaDotNet	peer-to-peer over the cluster (Option B)	a node must resolve while partitioned

Three constraints hold across all of them, and each is documented at the API and in the runbook: same KEK on every node (a mismatch fails closed and looks like corruption); ciphertext only crosses trust boundaries; re-wraps do not replicate, so rewrap-all runs once per independent store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no merge — stated plainly in the README rather than left for an operator to discover.

Design + plan: …-g7-clustered-replication-design.md

• …-g7-sqlserver-store.md. Operator runbook: ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md.

G-8 — KEK-rotation RewrapAll + runbook ✅ BUILT (2026-07-17, lib 0.1.3)

The RewrapAll(oldKek, newKek) admin primitive + operator runbook are built + fully tested in ZB.MOM.WW.Secrets. Because the envelope wraps a per-secret DEK under the KEK, rotation re-wraps DEKs only — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster LWW). Shipped: ISecretCipher.Rewrap(row, oldKek, newKek) (fail-closed on wrong old KEK; DEK zeroed on all paths; also catches malformed-length wraps → SecretDecryptionException), ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek) (updates only the 4 wrap columns + kek_id, compare-and-swap on the current wrapped DEK so a concurrent write can't corrupt a row — review-caught TOCTOU), KekRotationService.RewrapAllAsync + RewrapReport (enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry, fail-closed on unknown/identical KEK), the secret rewrap-all CLI verb (key material only via env-var-name/file-path — never a literal), README section, and the operator runbook ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md. Verified: full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke (migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard). Version bumped 0.1.2→0.1.3 (additive interface members); republish to the Gitea feed is a separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive). Reviewed by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent- write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.

Out of scope (this component)

• Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening concern, not secret storage.
• SQL-Server ISecretStore provider construction (seam supports it; build with G-7).