scadaproj/docs/plans/2026-07-16-secrets-adoption-design.md

Secrets Adoption (G-2 … G-6) — Shared Design

✅ EXECUTED 2026-07-16 at lib 0.1.2 — all four apps adopted. Version numbers and the Layer-A/Layer-B wiring template below are the historical record of that adoption and are left as-written; do not read 0.1.2 here as current. Current version is 0.2.0, and the clustering guidance in §5 has been superseded — see 2026-07-18-secrets-0.2.0-upgrade-and-clustering.md for the version bump and the per-app clustered topology. Everything else here (API surface §1, the two resolution layers §2, the wiring template §3, the gotchas §8) remains accurate and is still the reference for any new consumer.

For Claude: This is the shared design that the three per-repo implementation plans depend on. It is NOT itself executed. The executable plans are:

• 2026-07-16-secrets-adoption-otopcua.md — G-2, G-4, G-5, G-6
• 2026-07-16-secrets-adoption-scadabridge.md — G-3, G-4, G-5, G-6
• 2026-07-16-secrets-adoption-mxaccessgw.md — G-4, G-5, G-6

Goal: Adopt the already-built, already-published ZB.MOM.WW.Secrets library (envelope-encrypted secrets manager + ${secret:} config expander + runtime ISecretResolver

• /admin/secrets Blazor UI) into the three remaining family apps — OtOpcUa, ScadaBridge, MxAccessGateway — retiring plaintext credentials at rest.

Architecture: No library construction — the lib is built, published to the Gitea feed at 0.1.2, and reference-consumer-proven in HistorianGateway (live wonder end-to-end, 2026-07-16). Every task below is per-app wiring of an existing package, following the HistorianGateway template verbatim. The gaps map to two distinct resolution layers (below).

Tech Stack: .NET 10, C#, ZB.MOM.WW.Secrets{,.Abstractions,.Ui} @ 0.1.2 from the Gitea NuGet feed, AES-256-GCM envelope crypto, SQLite store, ASP.NET Core authorization + Blazor RCL.

Backlog source: components/secrets/GAPS.md. Per-app baselines: components/secrets/current-state/.

---

1. What is being adopted — the library API surface

Verified against scadaproj/ZB.MOM.WW.Secrets/ (the source of truth). These are the exact identifiers every plan calls; do not guess variants.

Concern	Type / member	Notes
DI registration	SecretsServiceCollectionExtensions.AddZbSecrets(this IServiceCollection, IConfiguration config, string sectionPath)	One overload only. Both the pre-host throwaway provider and the runtime registration call this identically, section "Secrets".
Pre-host expander	SecretReferenceExpander(ISecretResolver) → Task ExpandConfigurationAsync(IConfigurationRoot config, CancellationToken ct)	Rewrites ${secret:name} values in place via the IConfigurationRoot indexer. Fail-closed: unknown ref throws SecretNotFoundException. Skips any key whose leaf segment (after the last :) starts with _ (the _comment convention).
Store migrator	SqliteSecretsStoreMigrator.MigrateAsync(CancellationToken)	Idempotent (CREATE … IF NOT EXISTS). AddZbSecrets also registers SecretsMigrationHostedService to run it at startup, but the pre-host path must call MigrateAsync explicitly before the first resolve.
Runtime resolver	ISecretResolver.GetAsync(SecretName name, CancellationToken ct) → Task<string?>	GetAsync, not ResolveAsync. Returns decrypted plaintext, or null if absent/tombstoned. This is the seam G-2/G-3 driver-secret resolution calls.
Secret key type	readonly record struct SecretName — new SecretName("some/name")	Normalizes Trim().ToLowerInvariant(); validates allow-list [a-z0-9._/-], no .., no leading/trailing///. Implicit conversion to string only.
Authorization	SecretsAuthorization.AddSecretsAuthorization(this AuthorizationOptions) — in the .Ui package	Adds policies secrets:manage (ManagePolicy) + secrets:reveal (RevealPolicy). AdminRole = nameof(CanonicalRole.Administrator) == "Administrator", so an existing Administrator satisfies both. Additive — composes with existing Configure<AuthorizationOptions> callbacks.
UI page	ZB.MOM.WW.Secrets.Ui.SecretsPage — @page "/admin/secrets", [Authorize(Policy = ManagePolicy)]	Mount handle: typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly. RCL depends on ZB.MOM.WW.Theme + ZB.MOM.WW.Auth.AspNetCore + ZB.MOM.WW.Audit.
Master key	IMasterKeyProvider via MasterKeyProviderFactory.Create(MasterKeyOptions)	MasterKeySource { Environment, File, Dpapi }; default Environment, env var ZB_SECRETS_MASTER_KEY (base64 32 bytes). File/DPAPI use FilePath; optional explicit KekId.
Options	SecretsOptions bound from the "Secrets" section	SqlitePath (default secrets.db), MasterKey, RunMigrationsOnStartup (default true), ResolveCacheTtl (default 30 s).

The Secrets appsettings block (matches HistorianGateway):

"Secrets": {
  "SqlitePath": "<app>-secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true,
  "ResolveCacheTtl": "00:00:30"
}

---

2. The two resolution layers — the load-bearing distinction

The gaps split cleanly across two different layers that must not be conflated. This is the single most important design point, surfaced by the OtOpcUa recon (driver secrets are not IConfiguration).

Layer A — pre-host config expansion (${secret:}) — covers G-4 (all apps) + parts of G-2/G-3

Secrets that live in IConfiguration (appsettings / env): LDAP passwords, SQL connection strings, JWT signing keys, deploy/API keys, peppers. These are rewritten once, before the host is built, by a throwaway bootstrap provider — the HistorianGateway template (§3). A config value becomes the literal token "${secret:sql/<app>/db-password}"; the expander replaces it with the decrypted value before any options validator binds.

Layer B — runtime resolution (ISecretResolver.GetAsync) — covers G-2, G-3

Secrets that live in a database row as data, not in config: OtOpcUa's per-driver DriverConfig JSON (Galaxy API key, OpcUaClient Password/UserCertificatePassword), and ScadaBridge's MxGateway per-endpoint ApiKey inside the DataConnection JSON blob. The pre-host expander never sees these — they are read at driver-instantiation / connection time. The stored value becomes a secret:<name> reference; the consuming code calls ISecretResolver.GetAsync(new SecretName(name), ct) at the point of use.

Rule of thumb: if the secret is in appsettings/env → Layer A. If it is a column/JSON field in the app's own database → Layer B. G-4 is entirely Layer A. G-2 and G-3 are Layer B. G-6 (UI) and G-5 (master key) underpin both.

---

3. The proven wiring template (from HistorianGateway Program.cs)

Every app reuses these four pieces. Insertion points differ per app (see each plan); the code shape is identical.

(a) Pre-host ${secret:} expansion — before builder.Build() / before any options bind:

#pragma warning disable ASP0000 // deliberate throwaway container, disposed here, shares no singletons
await using (var secretsProvider = new ServiceCollection()
    .AddZbSecrets(builder.Configuration, "Secrets")
    .BuildServiceProvider())
#pragma warning restore ASP0000
{
    await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>().MigrateAsync(default);
    var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
    await new SecretReferenceExpander(resolver)
        .ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, default);
}

Usings: ZB.MOM.WW.Secrets.Abstractions, .Configuration, .DependencyInjection, .Sqlite. Note the cast to IConfigurationRoot. ScadaBridge differs — it assembles config in a bare ConfigurationBuilder (not WebApplication.CreateBuilder) before the host exists, so its expander runs against that IConfigurationRoot local, still via the same throwaway provider.

(b) Runtime registration — on the real host container (backs Layer B + the UI):

builder.Services.AddZbSecrets(builder.Configuration, "Secrets");

(c) Authorization — additive:

builder.Services.Configure<AuthorizationOptions>(o => o.AddSecretsAuthorization());

(d) UI mount — register the RCL assembly in both places or /admin/secrets 404s:

// Router.razor / Routes.razor:
//   AdditionalAssemblies="@(new[] { typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly })"
endpoints.MapRazorComponents<App>()
    .AddInteractiveServerRenderMode()
    .AddAdditionalAssemblies(typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly);

Package plumbing (every app): add ZB.MOM.WW.Secrets, .Abstractions, .Ui as PackageReferences at 0.1.2; add nuget.config packageSourceMapping patterns ZB.MOM.WW.Secrets and ZB.MOM.WW.Secrets.* under the dohertj2-gitea source (none of the three has a ZB.MOM.WW.* wildcard — the mapping is explicit, so restore fails without these). ScadaBridge additionally needs <PackageVersion> rows in Directory.Packages.props (CPM).

---

4. G-4 — pre-host config-secret expansion (all three apps)

Same Layer-A mechanism (§3a); per-app insertion point and target keys:

App	Insert expander at	Runs before	Config-secret targets
OtOpcUa	after Host/Program.cs:73 (role overlay done, env/cmdline re-appended)	AddOtOpcUaConfigDb (:94) + first ValidateOnStart (:102)	Security:Jwt:SigningKey, Security:Ldap:ServiceAccountPassword, Security:DeployApiKey, ConfigDb connstr, ServerHistorian:ApiKey
ScadaBridge	between Host/Program.cs:43 (.Build()) and :46 (StartupValidator.Validate)	StartupValidator/ConfigPreflight (:46)	Database:ConfigurationDb, Database:MachineDataDb, Security:Ldap:ServiceAccountPassword, Security:JwtSigningKey, InboundApi:ApiKeyPepper — plus the docker-per-node committed plaintext (see its plan)
mxaccessgw	after GatewayApplication.cs:64 (CreateBuilder), before :67 (TLS)	AddGatewayConfiguration (:71), AddZbGalaxyRepository (:103), AddZbApiKeyAuth	MxGateway:Ldap:ServiceAccountPassword, MxGateway:Galaxy:ConnectionString, MxGateway:ApiKeyPepper

Each plan also deletes the committed dev plaintext (and the loose *_login.txt files in ScadaBridge) and, where a hardcoded default exists in an options class (mxaccessgw LdapOptions.cs:61, serviceaccount123), removes it so a blanked appsettings can't fall back to a leaked default.

---

5. G-5 — master-key provider & clustering (the real fork)

The KEK never lives in the store — it is supplied out-of-band, base64 32 bytes, and never committed. The clustering posture differs sharply between the single-box gateway and the two Akka-clustered apps.

mxaccessgw — single box, no clustering

• Recommended: Environment provider (ZB_SECRETS_MASTER_KEY delivered via the NSSM service env, exactly how MxGateway__Ldap__ServiceAccountPassword is already delivered). Consistent with HistorianGateway; simplest operationally. Best-practice fit: container/NSSM env delivery is the standard 12-factor secret-bootstrap; matches the app's existing out-of-band password convention.
• Alternative: Dpapi provider (machine-bound key file). Stronger at-rest binding (the key is unusable off the box) but ties the store to one machine and complicates DR restore.

OtOpcUa & ScadaBridge — Akka-clustered → every node needs the same KEK and the same rows

Both apps resolve secrets on multiple nodes (OtOpcUa driver-role nodes resolve Layer-B driver secrets; both central nodes of a ScadaBridge pair resolve Layer-A config secrets at boot). Two constraints follow: (1) identical KEK on every node, (2) identical store contents on every node.

⚠️ UPDATED 2026-07-18 — the interim posture below is superseded. This section originally said the library "ships a SQLite-only store with a NoOpSecretReplicator — there is no built-in shared-SQL store and no cross-node replication," and recommended a shared-mounted-SQLite workaround. That is no longer true. G-7 shipped in 0.2.0 with two real replication packages. The KEK guidance below still stands verbatim; the store guidance does not. The shared-SQLite-on-a-network-share workaround should not be adopted — it was always a caveat-laden stopgap, and there is now a supported alternative.

Current topology options and the per-app recommendation live in 2026-07-18-secrets-0.2.0-upgrade-and-clustering.md §2. In brief: OtOpcUa → Akka peer-to-peer (one cluster, pub/sub already in product use); ScadaBridge → SQL-Server hub mode (central and each site are separate Akka clusters, so DistributedPubSub cannot cross that boundary — an Akka replicator there would create secret islands that never reconcile while every node reports healthy).

KEK requirement (unchanged, and independent of store choice): every node must resolve the same master key. Deliver it via MasterKey.Source=File with a read-only mounted key file identical on every node, or the same ZB_SECRETS_MASTER_KEY env value. A node with a different KEK stores replicated rows fine and then fails closed on every resolve with a kek_id mismatch — it reads like data corruption but is a deployment error.

Store options as of 0.2.0 (all three preserve ciphertext-only-at-rest):

• Shared SQL store (AddZbSecretsSqlServerStore) — one database, one copy of each row, no replication and no reconciliation. Simplest; the trade is that a node partitioned from the DB falls back only to the resolver's short TTL cache.
• Local store + SQL hub (AddZbSecretsSqlServerReplication) — each node keeps a local store and syncs bidirectionally with a central hub. Survives partition; converges eventually.
• Akka peer-to-peer (AddZbSecretsAkkaReplication) — no shared database at all; live broadcast plus periodic manifest anti-entropy within a single Akka cluster.

Rotation is per independent store. G-8 rewrap-all runs once for a shared store; once per node and against the hub in hub mode; once per node in Akka mode. Re-wraps deliberately do not replicate (they leave revision/updated_utc untouched so LWW ignores them).

---

6. G-6 — mount /admin/secrets (all three apps)

Same mechanism (§3d) in each dashboard:

App	Dashboard project	Register RCL assembly in
OtOpcUa	…OtOpcUa.AdminUI	EndpointRouteBuilderExtensions.cs:31 (MapRazorComponents<TApp>() — add .AddAdditionalAssemblies) and thread AdditionalAssemblies from App.razor:22 into <Routes/> (param already plumbed at Routes.razor:8,37-38)
ScadaBridge	…ScadaBridge.CentralUI	EndpointExtensions.cs:28 .AddAdditionalAssemblies(...) and Host/Components/Routes.razor:3 AdditionalAssemblies
mxaccessgw	…MxGateway.Server (Dashboard)	DashboardEndpointRouteBuilderExtensions.cs:133-135 (add .AddAdditionalAssemblies) and Dashboard/Components/Routes.razor:1 (add AdditionalAssemblies)

All three already have an Administrator canonical role, so AddSecretsAuthorization() grants secrets access to admins with no new role mapping — except verify the claim-type match below.

Claim-type verification (ScadaBridge especially): the library's policies use RequireRole(...) (reads ClaimsIdentity.RoleClaimType / ZbClaimTypes.Role). OtOpcUa and mxaccessgw already gate on role claims, so they match. ScadaBridge gates on RequireClaim(JwtTokenService.RoleClaimType, …) — its plan includes a task to confirm JwtTokenService.RoleClaimType equals the principal's role-claim type the library reads, or an Administrator won't satisfy secrets:manage/secrets:reveal even though ScadaBridge's own RequireAdmin works.

---

7. G-2 / G-3 — Layer-B database-resident secrets

G-2 — OtOpcUa driver secrets (highest value; its own plan, full detail)

Three call sites read DriverConfig/API-key JSON and must resolve secret: refs at runtime: OpcUaClientDriverFactoryExtensions.CreateInstance (:54), OpcUaClientDriverProbe (:38, AdminUI Test-Connect), and GalaxySecretRef.ResolveApiKey (:43, currently a static method with env:/file:/dev:/literal arms — add a secret: arm). Structural gotchas: the Drivers project does not reference …Security, so ISecretResolver must be threaded in as a dependency (and GalaxySecretRef needs a signature change off static); the resolver must be registered unconditionally (driver-role nodes have no auth/Data-Protection/AdminUI); the AdminUI editors must round-trip secret: refs rather than resolving-then-resaving cleartext.

G-3 — ScadaBridge MxGateway ApiKey (its own plan)

The ApiKey is a field inside the DataConnection.PrimaryConfiguration/BackupConfiguration JSON blob (MxGatewayEndpointConfig.ApiKey), consumed at MxGatewayDataConnection.cs:96. Because it is a field within a JSON column (not its own column), the existing EncryptedStringConverter cannot target it directly. Two options:

• Recommended: store a secret:<name> ref in the ApiKey field and resolve via ISecretResolver.GetAsync at consumption (MxGatewayDataConnection.cs:96). Consistent with G-2's Layer-B pattern; leaves the JSON blob human-readable; no schema change. Best-practice fit: same indirection model as OtOpcUa's driver secrets — one mental model across the family.
• Alternative: encrypt the whole PrimaryConfiguration/BackupConfiguration column with EncryptedStringConverter. Fewer code touch-points but encrypts a mixed blob (the scrubber and any display/edit path must decrypt first), and the column is currently read in several places — higher blast radius.

---

8. Cross-cutting gotchas (apply to every plan)

1. Fail-closed at boot. A ${secret:missing} throws SecretNotFoundException during the pre-host expansion — the app won't start. Seed every referenced secret (via the secret CLI or the UI) before switching a config value to a token. Plans stage this: add the token and seed the secret in the same task, and keep a rollback (the env-var override still works if the token is reverted).
2. _-prefixed comment keys are skipped — keep documentation example tokens under a _secretsComment-style key (the family already uses _secrets/_nodeName); they won't be resolved. Confirmed safe in every app's appsettings.
3. Register AddZbSecrets on the runtime host too (§3b) even for apps that only need Layer A today — the UI (G-6) and any future Layer-B use need the runtime resolver, and SecretsMigrationHostedService keeps the store schema current.
4. Audit is automatic. AddZbSecrets lazily binds IAuditWriter — all three apps have ZB.MOM.WW.Audit registered, so reveals/resolves audit through the app's existing writer with no ordering constraint (falls back to no-op otherwise). The reveal audit event was proven to carry no plaintext in HistorianGateway.
5. Do not disturb existing Data Protection key rings. OtOpcUa/ScadaBridge persist DP keys to their ConfigDb (SetApplicationName on OtOpcUa; none on ScadaBridge); mxaccessgw uses the framework-default provider for SignalR hub tokens only. Secrets envelope crypto is independent of Data Protection — do not reconfigure DP as part of this work, or you risk invalidating cookies/hub-tokens.
6. KEK is never committed and never printed. Same discipline as $GITEA_TOKEN.

---

9. Recommended rollout order

1. mxaccessgw first — simplest (single box, no clustering, no Layer-B). Proves the template a second time after HistorianGateway on the least-risky app.
2. OtOpcUa — highest security value (retires cleartext-in-DB driver secrets), but the heaviest (Layer A + Layer B + clustered KEK). Do Layer A (G-4) + UI (G-6) first, then the Layer-B driver work (G-2) as a separate reviewable slice.
3. ScadaBridge — heaviest config-secret surface (incl. docker-committed plaintext) + the claim-type verification + G-3. Benefits from the OtOpcUa Layer-B pattern being settled first.

Each app is an independent branch (feat/adopt-zb-secrets per repo), mirroring how auth / theme / audit were adopted across the family — commit + review, then FF-merge to the repo default and push to Gitea.

10. Testing strategy (per app)

• Offline: unit test the driver/connection resolver hook (G-2/G-3) with a fake ISecretResolver; assert secret: refs resolve and unknown refs fail closed. Confirm the app builds 0-warning with the three new package refs.
• Boot smoke: start the app with one config value switched to ${secret:…} and the secret seeded; assert clean boot. Negative control: unseed the secret → assert SecretNotFoundException at startup (proves fail-closed, exactly as HistorianGateway did).
• UI: browser-verify /admin/secrets — unauthenticated → login redirect (policy enforced); Administrator → metadata-only list + reveal shows plaintext + an audit row with no plaintext.
• Live (VPN): for OtOpcUa/ScadaBridge, prove a real driver/connection secret (secret: ref) authenticates real traffic, mirroring the HistorianGateway live proof. Requires the shared KEK present on the box.

11. Task tracking

Each per-repo plan co-locates its *.tasks.json beside the .md. The items this design deferred — G-7 (replication) and G-8 (RewrapAll KEK rotation) — were both built and shipped in 0.2.0 on 2026-07-18, G-7 as two packages (.Replicator.SqlServer and .Replicator.AkkaDotNet) rather than the single deferred Akka replicator this section anticipated. Adopting them in the apps is 2026-07-18-secrets-0.2.0-upgrade-and-clustering.md; remaining backlog stays in components/secrets/GAPS.md.