scadaproj/components/secrets/GAPS.md

Secrets — GAPS (adoption backlog)

The delta between each project's current-state and the SPEC, as prioritized work. The library itself is built + published + reference-consumer-proven; everything here is per-app adoption of an existing lib (not lib construction).

Current state (verified 2026-07-18)

• 0.1.3 PUBLISHED to the Gitea feed 2026-07-18 (all three packages; restore-verified from a scratch consumer). Carries the G-8 KEK-rotation work (KekRotationService, RewrapReport, secret rewrap-all, docs/operations/kek-rotation.md), committed on main at d82d345. (An earlier note claiming G-8 was "not yet committed/pushed" was stale.)
• 0.1.3 also carries a transitive security fix that 0.1.2 does not. Packing surfaced NU1903: Microsoft.Data.Sqlite 10.0.7 pulls SQLitePCLRaw.lib.e_sqlite3 2.1.11, which has a high-severity advisory (GHSA-2m69-gcr7-jv3q). Fixed the way ZB.MOM.WW.LocalDb already did — CentralPackageTransitivePinningEnabled + a pin to the patched 2.1.12. Bumping Microsoft.Data.Sqlite does not help (even 10.0.10 still pulls 2.1.11); the pin is the fix. Verified: the shipped .nuspec declares SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12 as a direct dependency, so consumers get the patched native lib without needing their own pin, and a fresh consumer restoring 0.1.3 scans clean.
• All four apps are still pinned at 0.1.2 — they get neither KEK rotation nor the security fix until bumped.
• All four apps are pinned to 0.1.2 — OtOpcUa, MxAccessGateway, ScadaBridge take Secrets + .Abstractions + .Ui; HistorianGateway takes Secrets + .Ui.
• G-7 clustered replication is BUILT (2026-07-18, lib 0.2.0) — both options, not just one. The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build covers both, as two new packages:
  • ZB.MOM.WW.Secrets.Replicator.SqlServer — SqlServerSecretStore : ISecretStore (all 7 members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and two topologies: AddZbSecretsSqlServerStore (one shared store, Option A) and AddZbSecretsSqlServerReplication (local store + hub, with a bidirectional sweep).
  • ZB.MOM.WW.Secrets.Replicator.AkkaDotNet — peer-to-peer over distributed pub/sub (Option B): live broadcast + periodic manifest anti-entropy, an explicit SerializerWithStringManifest, ciphertext-only wire DTOs.
  • Core gained the pieces both need: ISecretsStoreMigrator (so the hosted service + CLI are store-agnostic), SecretLastWriterWins (one definition of the LWW tie-break, shared by every store and reconciler), SecretReplicationReconciler, and ReplicatingSecretStore.
  • The ISecretReplicator seam had no caller — nothing invoked PublishAsync, so local writes would never have propagated. ReplicatingSecretStore decorates the store, which fixes it for the UI, the CLI, and any future write path at once.
  • Naming: the user chose .Replicator.* (not the plan's .Replication.* / ZB.MOM.WW.Secrets.Akka); AkkaDotNet rather than Akka also avoids a namespace collision with the Akka root namespace.
• Library suite: 170 passing offline (184 with the live SQL suite enabled), 1 skipped, green 2026-07-18 — including 15 live SQL-Server tests against a real SQL Server 2022 (the SQLite store's suite ported case-for-case, so any behavioural divergence between the two stores fails a test) and a 9-test in-process 2-node Akka cluster (real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up, and malformed-peer-input handling.

Open

• ✅ All four apps bumped 0.1.2 → 0.2.1 (2026-07-18), each on a local feat/secrets-0.2.0 branch, not pushed. HistorianGateway bb9d923, MxAccessGateway b021e29, OtOpcUa 1ccc237c, ScadaBridge 0083ce35.
  • Correction: the bump was originally justified as closing GHSA-2m69-gcr7-jv3q in all four apps. Measured against the 0.1.2 baseline in every repo, all four already resolved the patched SQLitePCLRaw.lib.e_sqlite3 2.1.12 — two via their own pins, two transitively via Auth.ApiKeys 0.1.5. The bump delivers no security change; its real value is version currency, secret rewrap-all access, and being the precondition for the replicator packages.
• ✅ A real SQLite vulnerability WAS found and fixed — but by separate work, not the bump. ScadaBridge's NuGetAuditSuppress was masking a live high-severity finding: only the Host project resolved a patched 2.1.12; AuditLog, SiteRuntime, StoreAndForward, SiteEventLogging and 11 test projects still resolved the vulnerable 2.1.11. Fixed with an explicit 2.1.12 pin per SQLite-consuming project and the suppression deleted (2a0faeab). MxAccessGateway's obsolete suppression removed likewise (2f0cfe3). Both suppressions rested on false premises ("no patched release exists" / "only the 3.x line is patched") — 2.1.12 patches it within 2.1.x, verified by forced restore. HistorianGateway and OtOpcUa needed nothing.
  • Rejected CentralPackageTransitivePinningEnabled: it clears the advisory in one line but makes every central version a transitive ceiling, demanding Google.Protobuf, Grpc.Net.Client, Microsoft.Data.SqlClient and Newtonsoft.Json bumps. That is a gRPC/data-access change to a production SCADA platform and deserves its own reviewed commit. Still open as a follow-up — it is the better long-term posture.
• ✅ 0.2.0 PUBLISHED to the Gitea feed 2026-07-18 (all 5 packages; restore-verified from a scratch consumer — resolves SQLitePCLRaw.lib.e_sqlite3 2.1.12, Microsoft.Data.SqlClient 6.0.2, Akka.Cluster.Tools 1.5.62, and scans clean). Merged to main @ 15ef1f3.
• 🟡 Topology WIRED in both clustered apps, default-OFF, NOT live-validated. OtOpcUa → Akka peer-to-peer (one cluster; DistributedPubSub already in product use; ActorSystem in DI via Akka.Hosting; Akka 1.5.62 matches the package exactly). ScadaBridge → SQL-Server hub mode, because central and each site are separate Akka clusters — DistributedPubSub cannot cross that boundary, so an Akka replicator there would produce secret islands that never reconcile while every node reports healthy.
  • Both gate on Secrets:Replication:Enabled, default false, so current behaviour is unchanged. ScadaBridge additionally requires a non-blank hub connection string — its options Validate() eagerly, so unconditional wiring would have broken startup on every node.
  • Bootstrap constraint recorded: the hub connstr cannot come from the hub; it must be seeded in each node's local store or supplied via env.
  • Task 9 live gate RUN 2026-07-18. Split result — one topology passed, one FAILED.

Live gate: SQL-Server hub (ScadaBridge topology) — ✅ 6/6 PASSED

Run against a real SQL Server 2022 (10.100.0.35:31433, DB ZbSecretsGate) with two nodes, each with its own local SQLite store, using AddZbSecretsSqlServerReplication — the same extension AddScadaBridgeSecrets calls:

Check	Result
hub schema provisions + sweeps start	✅
write on A → resolves on B via the hub	✅ converged through the shared hub
delete on A → tombstone on B	✅ no resurrection
write on B → resolves on A	✅ sweep is bidirectional
partitioned node still resolves from local state	✅ hub unreachable, local reads unaffected — site autonomy holds, which is the whole reason ScadaBridge uses hub mode
wrong-KEK node fails closed	✅ SecretDecryptionException, not wrong plaintext

The KEK control fired spontaneously first: a re-run against a hub still holding rows from a prior run (different KEK) threw SecretDecryptionException rather than returning garbage — unplanned but real evidence of fail-closed.

Live gate: Akka peer-to-peer (OtOpcUa topology) — ✅ PASSES 6/6 (root-caused + fixed in 0.2.2, 2026-07-18)

The 0.2.1 hang is root-caused and fixed. It was never Akka at all: the Akka package's DI wiring had a circular singleton dependency the container cannot detect through factory lambdas — ISecretStore (the ReplicatingSecretStore decorator) → ISecretReplicator → SecretReplicationActorProvider → ISecretCacheInvalidator → DefaultSecretResolver → ISecretStore again. Resolution recurses around that loop (same-thread Monitor re-entry keeps it alive) until MS.DI's StackGuard moves the recursion onto a fresh thread-pool thread to avoid a stack overflow — and that thread then blocks forever on a singleton call-site lock the first thread still holds. Managed stacks from a dotnet-dump of the hung process show the full cycle repeating and both threads parked, which is the Monitor_Wait the native samples saw. Both earlier candidate causes (DistributedPubSub.Get vs the Lazy<IActorRef> lock; missing Akka.Cluster.Tools HOCON) are disproven — the actor's constructor was never even reached, and the deadlock reproduces on a single-node rig with no clustering in play.

Fix (0.2.2): the one cycle-closing edge is deferred. SecretReplicationActorProvider now receives a DeferredSecretCacheInvalidator that resolves the real invalidator on first eviction — which can only happen when a replicated row is applied, strictly after the graph has finished resolving. The SQL-Server package never had the cycle (its replicator chain never touches the invalidator), which is exactly why the hub gate passed while Akka hung.

Verified: live convergence gate re-run — 6/6 checks on a real 2-node cluster (write→peer, delete→tombstone, tombstoned secret no longer resolves on the peer — the check that proves the deferred invalidator actually evicts — reverse direction, wrong-KEK fail-closed). Full Akka test project 38/38 green.

• ✅ Regression test now exists at the exact blind spot: HostedProcessResolutionTests builds the graph the way a host does (container-registered ActorSystem, hosted services started, then resolves ISecretStore/ISecretReplicator/ISecretResolver under a 20 s watchdog). Verified to discriminate: it deadlocks/fails on 0.2.1 and passes on 0.2.2. DeferredSecretCacheInvalidatorTests pins the wrapper's contract (no eager resolve; evictions forward; lookup runs once). This was the fourth defect in this library visible only when the DI graph is built inside a real host.
• ✅ 0.2.2 PUBLISHED to the Gitea feed 2026-07-18 (all 5 packages; restore-verified from a scratch consumer — 5 pinned refs, clean restore + build). Also folds in the deterministic rewrite of the ActorContextAfterAwaitTests pin (its old pool-thread formulation asserted a non-guaranteed property and flaked under parallel load).
• ✅ OtOpcUa bumped to 0.2.2 (2026-07-18, lmxopcua master @ 2254ae3d) and the gate re-run against the published feed packages passes 6/6. The long-promised The_startup_hook_actually_creates_the_replication_actor test now exists for real: a provider-based resolve of OtOpcUa's own AddOtOpcUaSecrets wiring on a self-joined single-node cluster, watchdogged at the exact point 0.2.1 deadlocked (SecretsReplicationRegistrationTests 8/8; plain ActorSystem, no TestKit — xunit.v3-compatible). Tracked: scadaproj#1 (the defect — fixed, CLOSED) and lmxopcua#482 (CLOSED 2026-07-18 — fully adopted).
• ✅ ENABLED on the OtOpcUa docker-dev rig (2026-07-18, lmxopcua master @ c878fbbd) — all six host nodes run Secrets:Replication:Enabled=true with one shared dev KEK (compose x-secrets-env anchor, ${OTOPCUA_SECRETS_KEK:-…} dev default). Live-proven on the rig: no startup deadlock across repeated restarts; a secret seeded into site-a-1's store alone converged to all five peers within ≤60 s with byte-identical ciphertext on all six nodes (same sha256/kek_id), decrypt-verified on central-2 + site-b-2; site-b-2 restart rejoined with the row intact. Found during verification and filed as lmxopcua#483 (FIXED + CLOSED 2026-07-19, lmxopcua master @ 5f72ff85): OtOpcUa's /admin/secrets was non-interactive because its static per-page-rendermode router routed straight to the RCL page, which cannot carry @rendermode itself (the other three apps render it under globally interactive routers where a nested render mode throws — verified, so no RCL change). Fix = host-side wrapper page owning the route with @rendermode InteractiveServer + policy/route-parity pin tests (SecretsPageWiringTests); Playwright-verified live on the rig (circuit + working Add-secret). The wrapper is the reference pattern for any static-router consumer of Secrets.Ui. Production/wonder enablement is deliberately NOT done — needs its own decision + real KEK distribution (never the committed dev key) per the clustered-secrets runbook.
• ✅ ScadaBridge's hub topology is live-validated and safe to enable (config + a shared KEK).

Harness: scratchpad/livegate/ (two scenarios, akka / sql; not committed).

Upstream defect found during adoption — 0.2.0 Akka replication was inert

• ✅ Fixed in 0.2.1, published to the feed 2026-07-18 (all 5 packages; all four apps consume it). AddZbSecretsAkkaReplication called AddZbSecrets first, which does TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(), so the package's own TryAdd was silently discarded. Replication published into a no-op sink and no actor was ever spawned — no exception, no log line. The SQL-Server replicator always had the correct order.
• Why it escaped: the SQL-Server package had a DI test asserting its replicator type; the Akka package had none. Every Akka test exercised the actor, serializer and reconciler in isolation — nothing built a container. This is the third instance of this defect class in this library (inert ISecretReplicator seam → unregistered concrete SqliteSecretStore → this), all with one cause: DI extensions with no test that constructs the graph. 0.2.1 adds 5 such tests, verified to discriminate (2 fail on the old order).
• Testing note: Akka.TestKit is the family convention for non-end-to-end Akka tests, but Akka.TestKit.Xunit2 is xunit-v2-only and OtOpcUa's Host.IntegrationTests is on xunit.v3 (CS0433 if added). OtOpcUa's guards therefore assert on ServiceCollection descriptors — which is exactly where this defect lives — while actor creation and convergence stay covered by the library's own 2-node cluster suite.
• ⬜ Per-app live wonder gates for the G-8 rotation path.
• ⬜ Akka self-echo forwarding is only observable with three nodes; the 2-node rig covers the drift symptom, not the forwarding filter itself (noted in the test).

Post-build code review (2026-07-18) — 7 defects found and fixed

An independent review caught real bugs that a green suite had not:

• CRITICAL — both replication modes were dead on arrival. The decorators resolve the undecorated local store by concrete type, but core registered SqliteSecretStore only behind ISecretStore. Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with No service for type 'SqliteSecretStore'), fixed, and covered by new DI-resolution tests.

• Unbounded fetch. A cold-starting node pulls the peer's entire inventory in one call; past ~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the node never converges. Now chunked at 500.

• Poison row aborted the batch. One unapplyable row abandoned every row after it, and since the pull set recomputes in the same order each sweep, it would poison the same batch forever. Now isolated per row and logged.

• Enum.Parse on peer input. An overflowing numeric string throws OverflowException, which the actor's ArgumentException filter did not catch → actor restart loop on redelivery; and any in-range number was accepted as an undefined enum value. Now TryParse + IsDefined.

• Null crypto blobs passed the boundary (required byte[] means present, not non-null).

• A failed pull-read replied with silence — the peer waited forever with no diagnostic.

• CRITICAL — anti-entropy was broken by Self read after await. Initially dismissed as a false positive on the reasoning that ActorBase caches Self in a field and that three passing anti-entropy tests traverse the path. Both premises were wrong. A direct probe (ActorContextAfterAwaitTests) shows Self resolves through Context, which is [ThreadStatic], and throws NotSupportedException after a real thread hop. The tests passed only because a local SQLite store usually completes await synchronously, keeping the continuation on the mailbox thread with the context intact — so correctness silently depended on store latency and thread-pool timing. GenuinelyAsyncSecretStore forces the async path; Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous fails on the unfixed code (20s timeout: "Node A never learned about a secret held by a node whose store is genuinely async") and passes in 3s after capturing self on the actor thread. Every remaining Self/Sender access in the actor was then audited.

Process note worth keeping. Seven of seven review findings were real. The one dismissal was wrong, and it was dismissed using indirect evidence (a green suite) over a direct mechanism claim. Green tests are evidence a path works under the timing the tests happened to produce, not evidence it is correct. When a reviewer names a mechanism, probe the mechanism.

Execution status (2026-07-16)

• mxaccessgw — G-4 + G-5 + G-6 DONE. Executed via the plan below (8 tasks, subagent-driven, classification-driven reviews), merged + pushed to origin/main (mxaccessgw @ e088dfa). Offline: 0-warning build, full suite 781 pass / 44 pre-existing worker-COM (0 non-worker), behavioral claim-type authz test, no Data-Protection disturbance. Box-verified on windev (real .NET 10.0.8): fail-closed on unseeded ${secret:} (SecretNotFoundException, refuses to start) + CLI seed/get decrypt round-trip + zero plaintext in the SQLite store at rest. (/admin/secrets interactive reveal not re-proven on this box — same shared .Ui component already live-proven via HistorianGateway; box gateway runs LDAP-disabled/anonymous so a real LDAP-via-secret login wasn't exercised there.) Two review-caught fixes folded in: the LDAP password change's test/doc blast radius (validator-helper + host-test-bootstrap + doc truth), and the live-LDAP integration suite now honors the MxGateway__Ldap__ServiceAccountPassword env override. G-4/G-5/G-6 below are the remaining OtOpcUa + ScadaBridge scope.
• ScadaBridge — G-3 + G-4 + G-5 + G-6 DONE. Executed via the plan below (10 tasks, subagent-driven, classification-driven reviews), FF-merged + pushed to origin/main (ScadaBridge @ 128f1596). Offline: 0-warning build, all suites green (Host 279, DCL 235 incl. 5 G-3 resolver tests, Security 181, CentralUI 925, InboundAPI 269, SiteRuntime 522/523 [1 pre-existing ScriptActor timing-flake, passes isolated + branch touched no SiteRuntime src], Commons/ConfigDb/ManagementService/Communication/ClusterInfra all green). G-3 LIVE-PROVEN vs the REAL production MxGateway gateway (wonder-app-vd03.zmr.zimmer.com:5120): a throwaway harness drove the real MxGatewayDataConnection through the DI factory + AddZbSecrets with a secret:-ref ApiKey → dummy key Unauthenticated (proves the resolved value is transmitted, not fail-closed first) → real read-scoped key → Connected + browsed the real Galaxy root (10 nodes). Temp key created via the gateway dashboard, then revoked + deleted + functionally confirmed dead. Notable deltas from the plan: T3 registers AddZbSecrets on both the Central branch (UI) and SiteServiceRegistration (DCL adapter) — the plan assumed one container; T6 claim-type resolved MATCH (ZbClaimTypes.Role == ClaimTypes.Role, adversarially verified) → AddSecretsAuthorization() used directly; T8 (G-5) shipped as a docs-only production-posture runbook (docs/operations/2026-07-16-secrets-clustered-master-key.md) because hardcoding Source=File + /shared paths would break the Central-boot Host.Tests + local dev (the pre-host expander migrates the store unconditionally every Central boot). One review-flagged non-blocking follow-up: a pre-existing xUnit env-mutation CI-flake that T4 amplifies (single [Collection] / disable parallelization for Host.Tests).
• OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted). Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), FF-merged + pushed to origin/master (lmxopcua @ 872cf7e3, ec6598ce..872cf7e3). Clean build (0 errors). The full offline suite caught one real regression — AddOtOpcUaDriverFactories' registration now resolves ISecretResolver (Galaxy/OpcUaClient secret: refs), so two ResilienceInvokerFactoryRegistrationTests that built a minimal container threw No service for ISecretResolver; fixed test-only by registering a stub resolver (production always has it via the unconditional AddZbSecrets, and GetRequiredService correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental AbCip_Green_AgainstSim sim-fixture test — zero AbCip code touched). Three deliberate deviations from the plan's literal text (verified corrections, same class as the CPM miss the plan itself caught): (1) G-4 does NOT commit a ${secret:} token into the default appsettings — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is ServerHistorian:ApiKey="" (empty, disabled-by-default), so a committed token would break every dev/CI/TwoNodeClusterHarness boot for zero at-rest benefit; instead documented the token-delivery convention (docs/security.md) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → SecretNotFoundException at pre-host expansion; seeded → resolves). (2) G-5 shipped docs-only (docs/operations/2026-07-16-secrets-clustered-master-key.md) — hardcoding Source=File+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays Source=Environment. Mirrors the ScadaBridge G-5 resolution. (3) Task 8 resolves in the factory/driver only, NOT the probe — the v3 OpcUaClientDriverProbe is unauthenticated GetEndpoints-only and never reads Password/UserCertificatePassword, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (GalaxyDriver.BuildClientOptionsAsync / OpcUaClientDriver.InitializeAsync), threaded via ctor + DriverFactoryBootstrap (real resolver from DI; NullSecretResolver null-object backs test/parse paths only, fail-closed on secret:). OpcUaClientDriverOptions was converted sealed class → sealed record for the with-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: DriverConfig persists as an opaque JSON string via RawTreeService, which has NO resolver dependency. Live wonder gate PROVEN 2026-07-16 (VPN up): a throwaway harness drove OtOpcUa's real GalaxyDriverBrowser (resolves the secret: ApiKeySecretRef via ISecretResolver → GalaxySecretRef.ResolveApiKeyAsync → connects) against the REAL production MxGateway wonder-app-vd03:5120. Dummy secret: ref → MxGatewayAuthenticationException (proves the resolved value is transmitted, not fail-closed-first); real metadata:read key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…). Temp key minted via the gateway dashboard, then revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw. Plan re-verified 2026-07-16 against origin/master @ ec6598ce (the merged v3.0 dual-namespace rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa is on Central Package Management (was mis-stated as inline-versioned), OpcUaClientDriverOptions lives in …Driver.OpcUaClient.Contracts + GalaxyDriverBrowser in …Driver.Galaxy.Browser, and DriverHostActor.cs line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
• G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17). G-8 shipped into ZB.MOM.WW.Secrets (lib 0.1.3): the Rewrap DEK primitive, CAS-guarded ApplyRewrapAsync, KekRotationService.RewrapAllAsync, the secret rewrap-all CLI verb, and an operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka fork is resolved (build the shared SQL-Server ISecretStore; Akka replicator deferred phase-2) with a design doc + executable plan + .tasks.json, no code built. See the G-7/G-8 backlog entries below. Not yet committed/pushed (awaiting go-ahead).

Design + implementation plans (2026-07-16)

G-2 … G-6 are now planned. Shared design + three per-repo executable plans (task-metadata'd, code-verified anchors, co-located .tasks.json):

• docs/plans/2026-07-16-secrets-adoption-design.md — shared design: library API, the two resolution layers (pre-host ${secret:} config expander vs runtime ISecretResolver), wiring template, master-key/clustering fork.
• docs/plans/2026-07-16-secrets-adoption-otopcua.md — G-2 + G-4 + G-5 + G-6 (10 tasks, 2 slices; Layer-A config + Layer-B driver secrets).
• docs/plans/2026-07-16-secrets-adoption-scadabridge.md — G-3 + G-4 + G-5 + G-6 (10 tasks; incl. claim-type verification + committed-plaintext cleanup).
• docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md — G-4 + G-5 + G-6 (8 tasks; single-box, no Layer B).

G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED (2026-07-17, below).

Cross-cutting observations

• No app has ${secret:} resolution today. All three assemble config with the stock AddJsonFile+AddEnvironmentVariables chain and read secrets verbatim. Two already lean on the pattern the expander formalizes: ScadaBridge's non-functional ${SCADABRIDGE_*} placeholders (whole-key env override) and OtOpcUa's driver-level GalaxySecretRef (env:/file:/dev:/literal). These are the natural first swaps.
• The common high-value gap is plaintext credentials in appsettings/env: LDAP service-account passwords (all three), SQL connection-string passwords (OtOpcUa ConfigDb, ScadaBridge ConfigDb + MachineDataDb), JWT/HS256 signing keys (OtOpcUa, ScadaBridge), and API-key peppers (all three, runtime-supplied but plaintext).
• Secrets already stored in a DB in plaintext are the sharpest edge: OtOpcUa's OpcUaClient Password/UserCertificatePassword and the dev:/literal Galaxy API key live cleartext in the central config DB; ScadaBridge's MxGateway per-endpoint ApiKey lives plaintext in the ConfigDb DataConnections JSON. These leak at rest to anyone with DB read.
• Existing at-rest crypto is Data Protection, and only ScadaBridge applies it to secrets (4 encrypted ConfigDb columns via EncryptedStringConverter). OtOpcUa/mxaccessgw use Data Protection only for cookies/tokens. None use DPAPI or ProtectKeysWith*; key rings are unencrypted-at-rest. ZB.MOM.WW.Secrets adds envelope encryption with a KEK held outside the store — a stronger boundary than "the DB is the only protection."
• Peppered-HMAC API-key stores stay bespoke (mxaccessgw + ScadaBridge): they are hashed, not reversibly stored — already correct. Secrets only takes over the pepper value supply.

Backlog (prioritized)

G-1 — HistorianGateway reference consumer ✅ DONE

Adopted + live-proven 2026-07-16 (historian password via ${secret:}, authenticated read against the real wonder historian). This is the wiring template for the three apps.

G-2 — OtOpcUa: cleartext-in-DB driver secrets (highest value)

Add a secret: arm to GalaxySecretRef (its own comment anticipates this) and to the OpcUaClient driver options, resolving through ISecretResolver — retire the dev:/literal and plaintext Password/UserCertificatePassword DB paths.

G-3 — ScadaBridge: MxGateway ApiKey plaintext-in-ConfigDb

Resolve via ISecretResolver or extend the existing EncryptedStringConverter to that JSON column.

G-4 — Pre-host ${secret:} for plaintext config secrets (all three)

LDAP passwords, SQL connstr passwords, JWT signing keys, deploy API key (OtOpcUa), peppers — swap to ${secret:…} tokens, expander running before each app's existing validator (ConfigPreflight / Ldap+OpcUa validators / GatewayOptionsValidator). Delete committed dev plaintext and the loose *_login.txt files (ScadaBridge).

G-5 — Master-key provider per deployment

Env (ZB_SECRETS_MASTER_KEY) for containers; DPAPI for the Windows boxes; File (shared mounted key) for clustered pairs (hard requirement — same KEK on every node).

G-6 — Mount /admin/secrets UI (all three)

Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI) and map each app's admin role onto secrets:manage / secrets:reveal.

G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib 0.2.0)

The SPEC's "shared SQL-Server store vs Akka replicator" fork was designed as either/or with Option A recommended and Option B deferred. Both were built — the fork is now a deployment choice rather than a library one, and the two packages are independent (an app takes whichever matches its availability requirement, or neither).

Package	Topologies	When
ZB.MOM.WW.Secrets.Replicator.SqlServer	shared store (Option A) · local + hub	every node can reach a shared DB
ZB.MOM.WW.Secrets.Replicator.AkkaDotNet	peer-to-peer over the cluster (Option B)	a node must resolve while partitioned

Three constraints hold across all of them, and each is documented at the API and in the runbook: same KEK on every node (a mismatch fails closed and looks like corruption); ciphertext only crosses trust boundaries; re-wraps do not replicate, so rewrap-all runs once per independent store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no merge — stated plainly in the README rather than left for an operator to discover.

Design + plan: …-g7-clustered-replication-design.md

• …-g7-sqlserver-store.md. Operator runbook: ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md.

G-8 — KEK-rotation RewrapAll + runbook ✅ BUILT (2026-07-17, lib 0.1.3)

The RewrapAll(oldKek, newKek) admin primitive + operator runbook are built + fully tested in ZB.MOM.WW.Secrets. Because the envelope wraps a per-secret DEK under the KEK, rotation re-wraps DEKs only — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster LWW). Shipped: ISecretCipher.Rewrap(row, oldKek, newKek) (fail-closed on wrong old KEK; DEK zeroed on all paths; also catches malformed-length wraps → SecretDecryptionException), ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek) (updates only the 4 wrap columns + kek_id, compare-and-swap on the current wrapped DEK so a concurrent write can't corrupt a row — review-caught TOCTOU), KekRotationService.RewrapAllAsync + RewrapReport (enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry, fail-closed on unknown/identical KEK), the secret rewrap-all CLI verb (key material only via env-var-name/file-path — never a literal), README section, and the operator runbook ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md. Verified: full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke (migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard). Version bumped 0.1.2→0.1.3 (additive interface members); republish to the Gitea feed is a separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive). Reviewed by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent- write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.

Out of scope (this component)

• Akka remoting authentication/TLS (ScadaBridge plain TCP remoting) — a separate hardening concern, not secret storage.
• SQL-Server ISecretStore provider construction (seam supports it; build with G-7).