scadaproj/docs/plans/2026-07-16-secrets-adoption-otopcua.md

OtOpcUa — Secrets Adoption (G-2, G-4, G-5, G-6) Implementation Plan

✅ EXECUTED 2026-07-16 — merged to origin/master @ 872cf7e3, G-2 live-proven against the real production MxGateway. This document is the historical record of that adoption; the 0.1.2 pins throughout are what was landed then, not what to use now.

Follow-on (not yet executed): bump to 0.2.0 and adopt clustered replication — 2026-07-18-secrets-0.2.0-upgrade-and-clustering.md, Tasks 3, 5, 6, 9. OtOpcUa's recommended topology is Akka peer-to-peer (ZB.MOM.WW.Secrets.Replicator.AkkaDotNet): it is a single Akka cluster on 1.5.62, already uses DistributedPubSub in product code, registers its ActorSystem in DI via Akka.Hosting, and its driver-role nodes benefit from resolving Layer-B driver secrets while partitioned.

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Shared context: docs/plans/2026-07-16-secrets-adoption-design.md.

Goal: Adopt the already-built, already-published ZB.MOM.WW.Secrets library (@ 0.1.2, Gitea feed, reference-consumer-proven in HistorianGateway) into OtOpcUa — retiring plaintext credentials at rest. Covers G-4 (pre-host ${secret:} config-secret expansion), G-6 (mount /admin/secrets UI), G-5 (clustered master-key posture), and the highest-value gap G-2 (retire cleartext-in-DB driver secrets via runtime ISecretResolver).

Architecture: No library construction — every task is per-app wiring of an existing package, following the HistorianGateway template (design §3) verbatim. OtOpcUa is the heaviest app because it spans both resolution layers (design §2):

• Layer A — pre-host config expansion (${secret:}) — G-4. Config-resident secrets (Security:Jwt:SigningKey, Security:Ldap:ServiceAccountPassword, Security:DeployApiKey, ConfigDb connstr, ServerHistorian:ApiKey) are rewritten once, before the host is built, by a throwaway bootstrap provider. Also underpins G-6 (UI).
• Layer B — runtime resolution (ISecretResolver.GetAsync) — G-2. Per-driver DriverConfig JSON secrets (Galaxy API key, OpcUaClient Password/UserCertificatePassword) live as data in a ConfigDb row, are re-deserialized on every deploy/delta, and are resolved at the point of use in driver-factory code. The pre-host expander never sees these.

Critical clustering fact: OtOpcUa is Akka-clustered (Program.cs:287-297). Layer-B driver secrets resolve on DRIVER-role nodes, which do NOT register auth / Data-Protection / AdminUI (those are admin-role-gated, Program.cs:309-324). Therefore AddZbSecrets (the runtime ISecretResolver + its KEK access) must be registered UNCONDITIONALLY, and every node needs the same KEK + same store rows (design §5).

Tech Stack: .NET 10, C#, ZB.MOM.WW.Secrets{,.Abstractions,.Ui} @ 0.1.2 from the Gitea NuGet feed, AES-256-GCM envelope crypto, SQLite store, ASP.NET Core authorization + Blazor RCL. Solution ZB.MOM.WW.OtOpcUa.slnx.

Anchors re-verified 2026-07-16 against origin/master @ ec6598ce (the merged v3.0 dual-namespace rewrite, PR #472). Key findings that changed this plan vs the original design:

• The v3.0 rewrite did NOT touch the driver-secret flow. GalaxySecretRef (still env:/file:/dev:/literal, no secret: arm), OpcUaClientDriverOptions.Password/UserCertificatePassword (still plaintext-in-JSON), and DriverHostActor's spec.DriverConfig passing all survived structurally intact. So the whole approach (Layer A + Layer B) still holds. The new Calculation driver has no secrets surface (out of scope).
• CPM correction (was backwards): OtOpcUa IS on Central Package Management — Directory.Packages.props at repo root, ManagePackageVersionsCentrally=true, 110 <PackageVersion> rows, and consuming csprojs carry no Version attribute. New packages need a <PackageVersion> entry plus a bare <PackageReference> (an inline-versioned ref will not resolve). Task 1 rewritten accordingly.
• Project-path corrections: OpcUaClientDriverOptions is in …Driver.OpcUaClient.Contracts; GalaxyDriverBrowser is in its own …Driver.Galaxy.Browser project (AdminUI references it), not inside AdminUI.
• DriverHostActor.cs line shifts (path unchanged, src/Server/…Runtime/Drivers/DriverHostActor.cs): spec.DriverConfig create 1704→1766, InitializeRequested 1756→1818, ApplyDelta 1790→1852.
• LdapOptionsValidator.ValidateOnStart was hoisted to run unconditionally (Program.cs:97-102, out of the old hasDriver gate) so admin-only nodes also validate LDAP — a precedent for registering AddZbSecrets unconditionally (Task 3).
• Program.cs, Security/, AdminUI/, driver Options/Probe/Factory files, and NuGet.config anchors otherwise survived essentially unchanged in path and line number.

---

Branch & sequencing

• Branch feat/adopt-zb-secrets off origin/master (OtOpcUa's default branch is master, not main — remote origin = https://gitea.dohertylan.com/dohertj2/lmxopcua).
• Mirror how auth / theme / audit were adopted across the family: commit + review on the feature branch, then FF-merge into master and push to Gitea. Keep the feat/* branch locally as history.
• Do not push mid-slice; land Slice 1 reviewed-and-green, then Slice 2.

Slices

• Slice 1 — Layer A + UI + plumbing + master-key (lands first, one reviewable slice): Task 1 (package plumbing) → Task 2 (pre-host expander, G-4 mechanism) → Task 3 (unconditional runtime AddZbSecrets) → Task 4 (switch config secrets to ${secret:}, G-4) → Task 5 (mount /admin/secrets, G-6) → Task 6 (File master-key posture for clustered nodes, G-5). Review + FF-merge + push.
• Slice 2 — Layer B driver secrets (G-2), separate reviewable slice: Task 7 (GalaxySecretRef secret: arm) → Task 8 (OpcUaClient factory + probe hooks — both call sites) → Task 9 (AdminUI editor round-trip preserves refs) → Task 10 (verification incl. live Galaxy secret-ref). Review + FF-merge + push.

Slice 2 depends on Slice 1's Task 3 (the runtime ISecretResolver registration) being in place. Within a slice, only the noted Parallelizable with tasks are independent.

---

Task 1 — Package references + NuGet.config mapping

Classification: small/trivial Estimated implement time: 4 min Parallelizable with: none (gates every other task — restore must succeed first)

Add the three ZB.MOM.WW.Secrets packages where they are consumed:

• Host (…OtOpcUa.Host) — pre-host expander + runtime registration + master-key options: ZB.MOM.WW.Secrets, ZB.MOM.WW.Secrets.Abstractions.
• AdminUI (…OtOpcUa.AdminUI) — the /admin/secrets UI + authorization: ZB.MOM.WW.Secrets.Ui (transitively pulls .Abstractions; brings ZB.MOM.WW.Theme + ZB.MOM.WW.Auth.AspNetCore + ZB.MOM.WW.Audit, all already referenced).
• Drivers projects — for the Layer-B ISecretResolver seam: the Drivers.Galaxy.Contracts project (GalaxySecretRef) and the Drivers.OpcUaClient project (factory/probe) need ZB.MOM.WW.Secrets.Abstractions (the ISecretResolver/SecretName seam lives there — do NOT reference the full .Secrets impl or .Ui from Drivers).

OtOpcUa uses Central Package Management (Directory.Packages.props, ManagePackageVersionsCentrally=true) — versions go in <PackageVersion> rows there, and each consuming csproj carries a bare <PackageReference Include="…" /> (NO Version attribute — an inline version won't resolve under CPM). This is the ScadaBridge pattern, NOT inline like the original plan assumed.

Files:

• Directory.Packages.props (repo root, near the existing ZB.MOM.WW.Auth.* rows at ~:132-134) — add three <PackageVersion> rows @ 0.1.2: ZB.MOM.WW.Secrets, ZB.MOM.WW.Secrets.Abstractions, ZB.MOM.WW.Secrets.Ui
• src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj — bare <PackageReference> for ZB.MOM.WW.Secrets + ZB.MOM.WW.Secrets.Abstractions (pre-host expander + runtime registration + master-key options)
• src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj — bare <PackageReference> for ZB.MOM.WW.Secrets.Ui (transitively pulls .Abstractions; brings ZB.MOM.WW.Theme + ZB.MOM.WW.Auth.AspNetCore + ZB.MOM.WW.Audit, all already referenced)
• src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts.csproj — bare <PackageReference> for ZB.MOM.WW.Secrets.Abstractions (this project defines GalaxySecretRef.ResolveApiKey, which gains an ISecretResolver param; the two callers Driver.Galaxy/Driver.Galaxy.Browser inherit the ref transitively)
• src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.csproj — bare <PackageReference> for ZB.MOM.WW.Secrets.Abstractions (the factory + probe that deserialize OpcUaClientDriverOptions live here; do NOT reference the full .Secrets impl or .Ui from Drivers)
• NuGet.config (repo root) — the dohertj2-gitea packageSourceMapping block (~:16-28; explicit patterns, NO ZB.MOM.WW.* wildcard — verified)

Steps:

1. Under dohertj2-gitea in NuGet.config (after the last existing pattern ZB.MOM.WW.HistorianGateway.Client at ~:27, before the block's closing </packageSource>) add:

   <package pattern="ZB.MOM.WW.Secrets" />
   <package pattern="ZB.MOM.WW.Secrets.*" />
   

   Without these, restore 404s (there is no wildcard mapping — verified).
2. Add the three <PackageVersion Include="ZB.MOM.WW.Secrets[.…]" Version="0.1.2" /> rows to Directory.Packages.props, then the bare <PackageReference> rows to the four csprojs above.
3. dotnet restore ZB.MOM.WW.OtOpcUa.slnx → expect all three packages resolve from dohertj2-gitea.
4. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings, 0 errors (packages referenced but not yet used).

---

Task 2 — Pre-host ${secret:} expander in Host/Program.cs (G-4 mechanism)

Classification: standard Estimated implement time: 5 min Parallelizable with: Task 5, Task 6 (different regions/files)

Insert the Layer-A pre-host expansion (design §3a) into the assembled configuration provider stack. Insertion point: immediately after Host/Program.cs:73 — the role-overlay block (:59-73) is complete, and env-vars (:71) + command-line (:72) have been re-appended to the top of the provider stack, so ${secret:} tokens supplied by ANY source are visible. It must run before AddOtOpcUaConfigDb(builder.Configuration) at :94 (reads the ConfigDb connstr) and before the first ValidateOnStart bind at :102 (LdapOptionsValidator).

This is the mechanism only — no config value is switched to a token yet (Task 4 does that), so this task is behavior-neutral (no ${secret:} tokens present → expander is a no-op pass).

Files:

• src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:73 (insert after) — before :94 AddOtOpcUaConfigDb, before :102 first validated bind
• usings: ZB.MOM.WW.Secrets.Abstractions, ZB.MOM.WW.Secrets.Configuration, ZB.MOM.WW.Secrets.DependencyInjection, ZB.MOM.WW.Secrets.Sqlite

Steps:

1. After Program.cs:73, insert the throwaway-provider block (design §3a verbatim):

   #pragma warning disable ASP0000 // deliberate throwaway container, disposed here, shares no singletons
   await using (var secretsProvider = new ServiceCollection()
       .AddZbSecrets(builder.Configuration, "Secrets")
       .BuildServiceProvider())
   #pragma warning restore ASP0000
   {
       await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>().MigrateAsync(default);
       var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
       await new SecretReferenceExpander(resolver)
           .ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, default);
   }
   

   Note the cast to IConfigurationRoot (a WebApplicationBuilder.Configuration is a ConfigurationManager, which implements IConfigurationRoot).
2. Add a Secrets section to appsettings.json (design §1 block) — SqlitePath: "otopcua-secrets.db", MasterKey.Source: "Environment", EnvVarName: "ZB_SECRETS_MASTER_KEY", RunMigrationsOnStartup: true, ResolveCacheTtl: "00:00:30". (Task 6 revisits MasterKey for the clustered posture.)
3. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings.
4. Boot smoke (no tokens present yet, ZB_SECRETS_MASTER_KEY seeded with a base64 32-byte dev key): dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host → clean startup, secrets store migrates, expander is a no-op. Expected: identical boot to baseline.

---

Task 3 — Register AddZbSecrets on the runtime host UNCONDITIONALLY (surprise f)

Classification: high-risk (cross-node resolution; touches the conditional role-gated registration structure) Estimated implement time: 5 min Parallelizable with: none within Slice 1 (Slice 2's Layer-B tasks depend on this)

Register the runtime ISecretResolver on the real host container (design §3b). Registration asymmetry — the biggest structural gotcha (surprise f): Auth (AddOtOpcUaAuth, Program.cs:312), Data Protection (ServiceCollectionExtensions.cs:73-75), and AdminUI (Program.cs:313,352) are all admin-role-gated (Program.cs:309-324 block). But Layer-B driver-secret resolution (Slice 2) runs on DRIVER-role nodes (the hasDriver block) — a driver-only node has no auth/DP/AdminUI, yet must resolve DriverConfig secret: refs.

Therefore register AddZbSecrets(builder.Configuration, "Secrets") unconditionally (outside both the admin-only and driver-only role blocks), so ISecretResolver + SecretsMigrationHostedService are present on every node regardless of role. This is independent of the admin-only DP registration — do NOT reconfigure Data Protection (gotcha §8.5).

Role-gate layout (verified): the hasDriver block is Program.cs:104-283; the hasAdmin block is :309-324 (condition if (hasAdmin) at :309); var app = builder.Build(); is at :329. The clean unconditional gap is :325-328 — right after the hasAdmin block closes, alongside the already-unconditional builder.Services.AddOtOpcUaHealth(); (~:326).

Files:

• src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:~326 — add a single builder.Services.AddZbSecrets(builder.Configuration, "Secrets"); on the unconditional path (after the hasAdmin block ends :324, outside the hasDriver block, before builder.Build() :329); usings from Task 2

Steps:

1. In the unconditional region between the hasAdmin block close (:324) and var app = builder.Build(); (:329) — next to AddOtOpcUaHealth() — add builder.Services.AddZbSecrets(builder.Configuration, "Secrets");.
2. Add a code comment: // Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.
3. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings.
4. Boot smoke, admin+driver node (default): clean boot, ISecretResolver resolvable, SecretsMigrationHostedService runs once.
5. Boot smoke, driver-only role overlay (roleSuffix → driver-only appsettings, no admin block): assert the app boots and ISecretResolver is present (no auth/DP registered). Expected: clean boot, resolver available.

---

Task 4 — G-4: switch config secrets to ${secret:} + seed + boot smoke + negative control

Classification: standard Estimated implement time: 5 min Parallelizable with: Task 5 (different files)

Switch OtOpcUa's Layer-A config-resident secrets to ${secret:} tokens, seeding each secret first (fail-closed — gotcha §8.1). Targets (design §4 / current-state):

Config key	Owner (SectionName / const)	Seed name
Security:Jwt:SigningKey	JwtOptions Security/Jwt/JwtOptions.cs (SectionName :5, MinSigningKeyBytes = 32 :6, SigningKey :9)	otopcua/jwt/signing-key
Security:Ldap:ServiceAccountPassword	LdapOptions Security/Ldap/LdapOptions.cs:22,63	otopcua/ldap/service-account-password
Security:DeployApiKey	DeployApiEndpoints const AdminUI/Api/DeployApiEndpoints.cs:32	otopcua/deploy/api-key
ConfigDb connstr	AddOtOpcUaConfigDb Configuration/ServiceCollectionExtensions.cs:10,19-25	otopcua/sql/configdb-connstr (or password-only ref inside the connstr)
ServerHistorian:ApiKey	ServerHistorianOptions SectionName ServerHistorian Runtime/Historian/ServerHistorianOptions.cs:24,46	otopcua/historian/api-key

Committed-appsettings reality (verified): of the five targets, only ServerHistorian:ApiKey is physically present in a committed appsettings — an empty-string placeholder at Host/appsettings.json:18 with a sibling _ApiKeyComment at :19. The other four (Jwt:SigningKey, Ldap:ServiceAccountPassword, Security:DeployApiKey, ConnectionStrings:ConfigDb) are not committed anywhere — supplied via env only. appsettings.Development.json sets Security:Ldap:DevStubMode = true, so dev boots without an LDAP password. So: switch the ServerHistorian:ApiKey placeholder to a ${secret:otopcua/historian/api-key} token in appsettings.json:18; for the other four, the token goes wherever the value is supplied (env delivery, or the deployment appsettings) — document it, don't invent a committed plaintext to replace.

Files:

• src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json:18 — swap the empty ServerHistorian:ApiKey placeholder for "${secret:otopcua/historian/api-key}" (keep/adjust the _ApiKeyComment at :19, which is _-prefixed → skipped per gotcha §8.2)
• env-delivery docs / deployment appsettings — set the other four targets to their "${secret:<name>}" tokens where each value is supplied
• src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs:40-45 — update the "env-var-only, NEVER commit" doc comment to note the value now also accepts a ${secret:} token resolved pre-host (surprise g)

Steps:

1. Seed each secret in the store via the secret CLI (or the /admin/secrets UI once Task 5 lands) — e.g. secret set otopcua/jwt/signing-key <value> against the same SqlitePath + KEK the Host uses.
2. Switch each config value to its ${secret:<name>} token.
3. Update the ServerHistorianOptions.cs:40-45 doc comment (do not change validation — it stays warn-only).
4. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings.
5. Boot smoke (positive): all five secrets seeded → dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host boots clean; LdapOptionsValidator (Program.cs:102), ServerHistorianOptionsValidator (:127-128), OpcUaApplicationHostOptionsValidator (:275-276) all bind resolved plaintext.
6. Negative control (fail-closed): unseed one secret (e.g. otopcua/jwt/signing-key) → boot must throw SecretNotFoundException during pre-host expansion, before any validator. Confirms design §8.1. Re-seed to restore.
7. Rollback note: reverting a token to the literal/env value still works (env override path intact) — keep this documented in the task's commit.

---

Task 5 — G-6: AddSecretsAuthorization() + mount /admin/secrets (both anchors)

Classification: standard Estimated implement time: 5 min Parallelizable with: Task 4 (different files)

Grant admins the secrets policies and mount the UI. OtOpcUa gates on role claims already (FleetAdmin → role Administrator at Security/ServiceCollectionExtensions.cs:160), and AddSecretsAuthorization() uses AdminRole = "Administrator" — so an existing Administrator satisfies secrets:manage/secrets:reveal with NO new role mapping (GroupToRole/DB grants already emit "Administrator"; no claim-type mismatch, unlike ScadaBridge).

The RCL assembly must be registered in BOTH places (design §3d / §6) or /admin/secrets 404s — SSR discovery via .AddAdditionalAssemblies AND interactive routing via the router's AdditionalAssemblies param.

Files:

• src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:143-173 — inside the AddAuthorization(...) block (or right after AddOtOpcUaAuth at Program.cs:312), add o.AddSecretsAuthorization(); (the admin-role block is admin-gated, which is correct for the UI — the UI only mounts on admin nodes)
• src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs:31 — add .AddAdditionalAssemblies(typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly) onto the MapRazorComponents<TApp>().AddInteractiveServerRenderMode() chain (:31-32)
• src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/App.razor:22 — pass AdditionalAssemblies="@(new[] { typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly })" into <Routes/> (currently a bare <Routes/>; the param is already plumbed through Components/Routes.razor — [Parameter] AdditionalAssemblies at :37-38, consumed by <Router AdditionalAssemblies=…> at :8)
• src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Layout/MainLayout.razor:22 — add <NavRailItem Href="/admin/secrets" Text="Secrets" /> after the Role grants item (:22) in the Navigation NavRailSection (:13-23). Nav items here are NOT markup-gated — the page's own [Authorize(Policy = ManagePolicy)] gates access, so an unconditional link is correct (non-admins get redirected).
• Do NOT touch the stale Host/Routes.razor (plain RouteView, still present but dead/unreferenced) — App.razor's <Routes/> resolves to the AdminUI RCL one

Steps:

1. Add AddSecretsAuthorization() in the AdminUI authorization block (via Configure<AuthorizationOptions> or directly in the AddAuthorization callback at ServiceCollectionExtensions.cs:143-173). Confirm it composes additively with the existing FleetAdmin/ConfigEditor/etc. policies. Claim-type is a confirmed MATCH — FleetAdmin uses RequireRole("Administrator") (ServiceCollectionExtensions.cs:160) reading the standard ClaimTypes.Role, same claim AddSecretsAuthorization()'s AdminRole="Administrator" reads (independently confirmed via MainLayout.razor:40-41 filtering ClaimTypes.Role). No new role mapping, no ScadaBridge-style mismatch.
2. Add .AddAdditionalAssemblies(...) at EndpointRouteBuilderExtensions.cs:31.
3. Add the AdditionalAssemblies arg to <Routes/> at App.razor:22.
4. Add the <NavRailItem> at MainLayout.razor:22.
5. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings.
6. Browser verify (AdminUI on :9000): unauthenticated /admin/secrets → login redirect (policy enforced); Administrator → metadata-only list + reveal shows plaintext; confirm an audit row is written with no plaintext (gotcha §8.4 — reveal audit proven plaintext-free in HistorianGateway). Use superpowers-extended-cc:executing-plans' browser step or the multi-role test user (multi-role/password).

---

Task 6 — G-5: File master-key posture for clustered nodes (the fork)

Classification: small/trivial (config + docs; the interim posture ships no new code) Estimated implement time: 4 min Parallelizable with: Task 4, Task 5

OtOpcUa is Akka-clustered (Program.cs:287-297, AddAkka("otopcua", …)) → every node needs the same KEK and the same store rows (design §5). The library today ships a SQLite-only store + NoOpSecretReplicator (no built-in shared-SQL store, no cross-node replication).

Interim posture (recommended, ships G-5 with no new library code):

• MasterKey.Source = File, FilePath = a read-only mounted secret file identical on every node (not Environment, so all nodes read the same key material without per-node env drift).
• SqlitePath → a single shared SQLite store on a shared/replicated volume (secrets are written rarely, human-driven, from one node's UI/CLI; reads dominate — acceptable for SQLite's single-writer limits).

Fork (design §5): the "right" long-term shape is a ConfigDb-backed ISecretStore mirroring OtOpcUa's existing AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>() pattern (ServiceCollectionExtensions.cs:73-75 — DP key ring is persisted to ConfigDb precisely to share key material across nodes). That requires building a custom ISecretStore, overlaps G-7, and is deferred there — NOT attempted in this cut. Flag the hand-off in the commit.

Files:

• appsettings.json (or deployment/role-overlay appsettings) — Secrets.MasterKey.Source: "File", FilePath: "<shared-mount>/otopcua-master.key"; Secrets.SqlitePath: "<shared-mount>/otopcua-secrets.db"
• deployment runbook / docs — document the shared mount + KEK delivery (KEK never committed, gotcha §8.6); note the G-7 hand-off

Steps:

1. Set MasterKey.Source=File + shared FilePath + shared SqlitePath in the deployment appsettings (dev can stay Environment for single-node testing).
2. Add a doc note: interim File-KEK + shared-SQLite now; ConfigDb-backed ISecretStore tracked under G-7.
3. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings.
4. Multi-node smoke (if a 2-node docker-dev rig is available): seed a secret from node A's UI, resolve it on node B → same value (proves shared store + shared KEK). Otherwise document as a live-gate for rollout.

Slice 1 gate: review, FF-merge feat/adopt-zb-secrets → master, push to Gitea. Then proceed to Slice 2.

---

Task 7 — G-2a: add secret: arm to GalaxySecretRef (Layer B)

Classification: high-risk (signature change on a static method consumed by the driver + AdminUI; touches the driver model) Estimated implement time: 5 min Parallelizable with: none (Task 8 shares the OpcUaClient path but is a separate file; can proceed after this if reviewers want serial)

Add a secret: arm to GalaxySecretRef.ResolveApiKey (Driver.Galaxy.Contracts/GalaxySecretRef.cs), resolving via ISecretResolver.GetAsync, replacing the cleartext-in-DB dev:/literal path. The class doc (:23-24) anticipates exactly this kind of swap ("a future PR can swap any of these arms … without changing the call site"). Structural change: ResolveApiKey is still public static string ResolveApiKey(string secretRef, ILogger? logger = null) (:43) with no DI — it must accept an ISecretResolver parameter (signature change). Both call sites (per the remarks at :26-30) must be updated to thread the resolver in:

• Driver.Galaxy/GalaxyDriver.cs:537 — ApiKey = GalaxySecretRef.ResolveApiKey(gw.ApiKeySecretRef, _logger)
• Driver.Galaxy.Browser/GalaxyDriverBrowser.cs:125 — same call. Note: GalaxyDriverBrowser is in its own project …Driver.Galaxy.Browser (AdminUI references it and registers it as IDriverBrowser at AdminUI/ServiceCollectionExtensions.cs:59), not inside AdminUI.

Current arms in the file: env: (:47-55), file: (:57-70), dev: (:72-77), literal fall-through (:79-87, return secretRef; at :87) with the "cleartext in DB" warning at :83-86.

TDD. Write the fake-ISecretResolver unit test first.

Files:

• src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxySecretRef.cs:43-87 — add secret: arm before the literal fall-through (before :79); change signature to accept ISecretResolver (and go async Task<string> since GetAsync is async — or resolve synchronously via the resolver's cache if a sync seam exists; prefer async and propagate)
• call site 1: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriver.cs:537 — thread ISecretResolver (ctor-injected)
• call site 2: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser/GalaxyDriverBrowser.cs:125 — thread ISecretResolver (DI-injected)
• test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/ — extend the existing GalaxyDriverApiKeyResolverTests.cs (there is no dedicated …Contracts.Tests project; this suite already references the Contracts project) with the secret:-arm cases

Steps:

1. Test first: add GalaxySecretRefTests — a fake ISecretResolver returning a known value for secret:galaxy/inst1/apikey; assert (a) secret: ref resolves to the fake value, (b) an unknown secret: ref fails closed (resolver returns null → method throws/returns fail-closed, not silent literal fall-through), (c) existing env:/file:/dev:/literal arms unchanged. dotnet test --filter "FullyQualifiedName~GalaxySecretRefTests" → RED.
2. Add the secret: arm: if (secretRef.StartsWith("secret:")) { var name = secretRef["secret:".Length..]; var val = await resolver.GetAsync(new SecretName(name), ct); return val ?? throw/fail-closed; } before the literal fall-through.
3. Change the signature to accept ISecretResolver (+ CancellationToken); make async as needed.
4. Update GalaxyDriver + GalaxyDriverBrowser call sites to inject + pass the resolver.
5. dotnet test --filter "FullyQualifiedName~GalaxySecretRefTests" → GREEN.
6. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings; dotnet test ZB.MOM.WW.OtOpcUa.slnx → driver unit suite green.

---

Task 8 — G-2b: ISecretResolver hook in OpcUaClient factory AND probe (both call sites)

Classification: high-risk (driver-factory + actor model; TWO call sites must adopt or Test-Connect and runtime diverge) Estimated implement time: 5 min Parallelizable with: Task 7 (separate driver)

Resolve secret: refs for the OpcUaClient driver's Password (…Driver.OpcUaClient.Contracts/OpcUaClientDriverOptions.cs:76) and UserCertificatePassword (:93) at the deserialization hook — where the DriverConfig JSON is deserialized into OpcUaClientDriverOptions. Note the Options POCO lives in the .Contracts project; the two hook sites below live in the Driver.OpcUaClient project. Two hook sites, both mandatory (surprise d — resolve at BOTH or Test-Connect and runtime diverge):

• (a) OpcUaClientDriverFactoryExtensions.CreateInstance(driverInstanceId, driverConfigJson, ILoggerFactory?) (signature :48-49, body to :59) — JsonSerializer.Deserialize<OpcUaClientDriverOptions>(...) at :54 (shared JsonOptions :27-32). Runtime path: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:1766 passes spec.DriverConfig into TryCreate, re-sent on InitializeRequested :1818 / ApplyDelta :1852 — surprise b. (These shifted ~+62 lines in v3; path unchanged.)
• (b) OpcUaClientDriverProbe.cs:38 (AdminUI Test-Connect), same shape, opts at :24.

Structural gotcha (surprise c): the Drivers project has no DI/IConfiguration — ISecretResolver must be threaded in as a ctor/param dependency. The factory is registered via AddOtOpcUaDriverFactories() at Program.cs:216; wire the resolver into the factory registration. The resolver is guaranteed present because Task 3 registered AddZbSecrets unconditionally.

TDD. Fake-ISecretResolver unit test first.

Files:

• src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriverFactoryExtensions.cs:48-59 (deserialize at :54) — after deserialize, resolve secret:-prefixed Password/UserCertificatePassword via injected ISecretResolver
• src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriverProbe.cs:38 (opts at :24) — same resolution
• src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:216 — AddOtOpcUaDriverFactories() registration threads ISecretResolver into the factory
• src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs — consumers (BuildUserIdentity :376, using options.Password at :382; LoadPkcs12FromFile(options.UserCertificatePath, options.UserCertificatePassword) at :495) receive already-resolved plaintext (no change needed if resolution happens at deserialization)
• new test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/ — factory + probe resolve secret: refs

Steps:

1. Test first: fake ISecretResolver; assert factory CreateInstance with DriverConfig JSON where Password="secret:opcua/inst/password" yields options carrying the resolved plaintext; assert unknown ref fails closed; mirror for the probe. dotnet test --filter "FullyQualifiedName~OpcUaClientDriver" → RED.
2. Add a shared helper (e.g. ResolveSecretRefs(OpcUaClientDriverOptions, ISecretResolver, ct)) that resolves secret:-prefixed Password/UserCertificatePassword; call it from BOTH CreateInstance:54 and Probe:38.
3. Thread ISecretResolver into the factory (via AddOtOpcUaDriverFactories() at Program.cs:216) and into the probe's DI.
4. dotnet test --filter "FullyQualifiedName~OpcUaClientDriver" → GREEN.
5. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings; dotnet test ZB.MOM.WW.OtOpcUa.slnx → driver suite green.

---

Task 9 — G-2c: AdminUI driver-config editor round-trip preserves secret: refs (surprise e)

Classification: standard (verify/guard; risk is silent cleartext-resave) Estimated implement time: 5 min Parallelizable with: none (validates Tasks 7-8)

Driver secrets are stored cleartext in the ConfigDb today (GalaxySecretRef.cs:84-86 warns). The AdminUI driver-config editors emit/read that same DriverConfig JSON. The UI round-trip must PRESERVE the secret:/env:/file: refs — it must NOT resolve-then-resave cleartext (which would defeat the whole point and re-leak the secret into the DB). Verify the editor persists the raw ref string, and add a guard/test.

Files:

• AdminUI driver-config editor components (Galaxy + OpcUaClient driver config edit pages) + their save path — confirm they persist the raw DriverConfig JSON (with secret: refs intact), not a resolved-plaintext round-trip
• GalaxyDriverBrowser (Task 7 call site) — ensure browse/test resolves for the live call but does NOT write the resolved value back
• test: editor save round-trip preserves a secret: ref verbatim

Steps:

1. Trace the AdminUI editor save path for Galaxy + OpcUaClient driver configs; confirm the persisted DriverConfig JSON is the user-entered ref, unresolved.
2. If any path resolves-then-saves, fix it to persist the ref; resolution happens only at driver-instantiation/test time (Tasks 7-8).
3. Add a round-trip test: save {"Password":"secret:opcua/x/pw"} → reload → assert the stored value is still secret:opcua/x/pw (not plaintext).
4. dotnet test ZB.MOM.WW.OtOpcUa.slnx → green.
5. Browser check: edit a driver config with a secret: ref, save, reopen → ref shown, not plaintext; ConfigDb row holds the ref.

---

Task 10 — Verification (Slice 2 gate)

Classification: standard Estimated implement time: 5 min Parallelizable with: none (final gate)

Steps:

1. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings (repo builds under warnings-as-errors discipline).
2. dotnet test ZB.MOM.WW.OtOpcUa.slnx → full suite green (driver unit tests incl. the new GalaxySecretRef + OpcUaClient secret-resolution tests).
3. Boot smoke, admin+driver node: /admin/secrets reachable, config secrets resolve, clean boot.
4. Boot smoke, driver-only node: resolves a DriverConfig secret: ref with no auth/DP/AdminUI present (proves surprise f / Task 3).
5. Live (VPN, wonder): configure a Galaxy driver instance with an ApiKeySecretRef = "secret:galaxy/<inst>/apikey", seed the real gateway key, deploy → assert the driver authenticates real MxAccess traffic through the resolved key (mirrors HistorianGateway's live proof; requires the shared KEK present on the box). Also verify an OpcUaClient driver with a secret: Password connects.
6. Review, FF-merge feat/adopt-zb-secrets → master, push to Gitea.

---

Testing & rollout

• Offline (per design §10): fake-ISecretResolver unit tests for GalaxySecretRef + OpcUaClient factory/probe (assert secret: resolves + unknown fails closed); 0-warning build with the three new package refs.
• Boot smoke: one config value switched to ${secret:…} + seeded → clean boot; negative control (unseed → SecretNotFoundException at startup, fail-closed).
• UI: browser-verify /admin/secrets — unauth → login redirect; Administrator → metadata list + reveal shows plaintext + audit row carries NO plaintext.
• Live (VPN): real Galaxy + OpcUaClient secret: refs authenticate real traffic on wonder, with the shared KEK on the box.
• Rollout: Slice 1 (Layer A + UI + master-key) merges first; Slice 2 (Layer B driver secrets) merges after review. Each slice: commit + review → FF-merge to master → push to Gitea. Seed all referenced secrets before switching any config value to a token (fail-closed staging, gotcha §8.1).

Deferred / out of scope (per components/secrets/GAPS.md)

• G-7 — clustered shared store / replicator. The interim posture (Task 6) is File-KEK + shared-SQLite on a shared volume. The integrated ConfigDb-backed ISecretStore (mirroring OtOpcUa's DP PersistKeysToDbContext pattern) and the Akka ZB.MOM.WW.Secrets.Akka replicator are G-7 — NOT built here. Task 6 flags the hand-off.
• G-8 — RewrapAll KEK rotation. Rewrapping all envelopes under a new master key is deferred to G-8.
• Not touched (keep bespoke): the Data-Protection cookie/JWT key ring (do NOT reconfigure DP — gotcha §8.5), in-memory end-user OPC-UA passwords, and the file-system OPC UA PKI Directory stores.