scadaproj/docs/plans/2026-07-16-secrets-adoption-design.md

# Secrets Adoption (G-2 … G-6) — Shared Design

> **✅ EXECUTED 2026-07-16 at lib `0.1.2` — all four apps adopted.** Version numbers and the
> Layer-A/Layer-B wiring template below are the **historical record of that adoption** and are
> left as-written; do not read `0.1.2` here as current. **Current version is `0.2.0`**, and the
> clustering guidance in §5 has been superseded — see
> [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md)
> for the version bump and the per-app clustered topology. Everything else here (API surface §1,
> the two resolution layers §2, the wiring template §3, the gotchas §8) remains accurate and is
> still the reference for any new consumer.

> **For Claude:** This is the shared design that the three per-repo implementation plans
> depend on. It is NOT itself executed. The executable plans are:
> - [`2026-07-16-secrets-adoption-otopcua.md`](2026-07-16-secrets-adoption-otopcua.md) — G-2, G-4, G-5, G-6
> - [`2026-07-16-secrets-adoption-scadabridge.md`](2026-07-16-secrets-adoption-scadabridge.md) — G-3, G-4, G-5, G-6
> - [`2026-07-16-secrets-adoption-mxaccessgw.md`](2026-07-16-secrets-adoption-mxaccessgw.md) — G-4, G-5, G-6

**Goal:** Adopt the already-built, already-published `ZB.MOM.WW.Secrets` library
(envelope-encrypted secrets manager + `${secret:}` config expander + runtime `ISecretResolver`
+ `/admin/secrets` Blazor UI) into the three remaining family apps — OtOpcUa, ScadaBridge,
MxAccessGateway — retiring plaintext credentials at rest.

**Architecture:** No library construction — the lib is built, published to the Gitea feed at
`0.1.2`, and **reference-consumer-proven** in HistorianGateway (live wonder end-to-end,
2026-07-16). Every task below is *per-app wiring* of an existing package, following the
HistorianGateway template verbatim. The gaps map to two distinct resolution layers (below).

**Tech Stack:** .NET 10, C#, `ZB.MOM.WW.Secrets{,.Abstractions,.Ui}` @ `0.1.2` from the Gitea
NuGet feed, AES-256-GCM envelope crypto, SQLite store, ASP.NET Core authorization + Blazor RCL.

**Backlog source:** [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md). Per-app
baselines: [`components/secrets/current-state/`](../../components/secrets/current-state/).

---

## 1. What is being adopted — the library API surface

Verified against `scadaproj/ZB.MOM.WW.Secrets/` (the source of truth). These are the exact
identifiers every plan calls; do not guess variants.

| Concern | Type / member | Notes |
|---|---|---|
| DI registration | `SecretsServiceCollectionExtensions.AddZbSecrets(this IServiceCollection, IConfiguration config, string sectionPath)` | **One overload only.** Both the pre-host throwaway provider and the runtime registration call this identically, section `"Secrets"`. |
| Pre-host expander | `SecretReferenceExpander(ISecretResolver)` → `Task ExpandConfigurationAsync(IConfigurationRoot config, CancellationToken ct)` | Rewrites `${secret:name}` values **in place** via the `IConfigurationRoot` indexer. Fail-closed: unknown ref throws `SecretNotFoundException`. **Skips** any key whose leaf segment (after the last `:`) starts with `_` (the `_comment` convention). |
| Store migrator | `SqliteSecretsStoreMigrator.MigrateAsync(CancellationToken)` | Idempotent (`CREATE … IF NOT EXISTS`). `AddZbSecrets` also registers `SecretsMigrationHostedService` to run it at startup, but the **pre-host** path must call `MigrateAsync` explicitly before the first resolve. |
| Runtime resolver | `ISecretResolver.GetAsync(SecretName name, CancellationToken ct)` → `Task<string?>` | **`GetAsync`, not `ResolveAsync`.** Returns decrypted plaintext, or `null` if absent/tombstoned. This is the seam G-2/G-3 driver-secret resolution calls. |
| Secret key type | `readonly record struct SecretName` — `new SecretName("some/name")` | Normalizes `Trim().ToLowerInvariant()`; validates allow-list `[a-z0-9._/-]`, no `..`, no leading/trailing/`//`. Implicit conversion **to** `string` only. |
| Authorization | `SecretsAuthorization.AddSecretsAuthorization(this AuthorizationOptions)` — **in the `.Ui` package** | Adds policies `secrets:manage` (`ManagePolicy`) + `secrets:reveal` (`RevealPolicy`). `AdminRole = nameof(CanonicalRole.Administrator)` == `"Administrator"`, so an existing Administrator satisfies both. Additive — composes with existing `Configure<AuthorizationOptions>` callbacks. |
| UI page | `ZB.MOM.WW.Secrets.Ui.SecretsPage` — `@page "/admin/secrets"`, `[Authorize(Policy = ManagePolicy)]` | Mount handle: `typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly`. RCL depends on `ZB.MOM.WW.Theme` + `ZB.MOM.WW.Auth.AspNetCore` + `ZB.MOM.WW.Audit`. |
| Master key | `IMasterKeyProvider` via `MasterKeyProviderFactory.Create(MasterKeyOptions)` | `MasterKeySource { Environment, File, Dpapi }`; default `Environment`, env var `ZB_SECRETS_MASTER_KEY` (base64 32 bytes). File/DPAPI use `FilePath`; optional explicit `KekId`. |
| Options | `SecretsOptions` bound from the `"Secrets"` section | `SqlitePath` (default `secrets.db`), `MasterKey`, `RunMigrationsOnStartup` (default true), `ResolveCacheTtl` (default 30 s). |

The `Secrets` appsettings block (matches HistorianGateway):

```json
"Secrets": {
  "SqlitePath": "<app>-secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true,
  "ResolveCacheTtl": "00:00:30"
}
```

---

## 2. The two resolution layers — the load-bearing distinction

The gaps split cleanly across **two different layers** that must not be conflated. This is the
single most important design point, surfaced by the OtOpcUa recon (driver secrets are *not*
`IConfiguration`).

### Layer A — pre-host config expansion (`${secret:}`) — covers G-4 (all apps) + parts of G-2/G-3

Secrets that live in `IConfiguration` (appsettings / env): LDAP passwords, SQL connection
strings, JWT signing keys, deploy/API keys, peppers. These are rewritten **once, before the
host is built**, by a throwaway bootstrap provider — the HistorianGateway template (§3). A
config value becomes the literal token `"${secret:sql/<app>/db-password}"`; the expander
replaces it with the decrypted value before any options validator binds.

### Layer B — runtime resolution (`ISecretResolver.GetAsync`) — covers G-2, G-3

Secrets that live **in a database row as data**, not in config: OtOpcUa's per-driver
`DriverConfig` JSON (Galaxy API key, OpcUaClient `Password`/`UserCertificatePassword`), and
ScadaBridge's MxGateway per-endpoint `ApiKey` inside the `DataConnection` JSON blob. The
pre-host expander never sees these — they are read at driver-instantiation / connection time.
The stored value becomes a `secret:<name>` reference; the **consuming code** calls
`ISecretResolver.GetAsync(new SecretName(name), ct)` at the point of use.

> **Rule of thumb:** if the secret is in appsettings/env → Layer A. If it is a column/JSON
> field in the app's own database → Layer B. G-4 is entirely Layer A. G-2 and G-3 are Layer B.
> G-6 (UI) and G-5 (master key) underpin both.

---

## 3. The proven wiring template (from HistorianGateway `Program.cs`)

Every app reuses these four pieces. Insertion points differ per app (see each plan); the code
shape is identical.

**(a) Pre-host `${secret:}` expansion** — before `builder.Build()` / before any options bind:

```csharp
#pragma warning disable ASP0000 // deliberate throwaway container, disposed here, shares no singletons
await using (var secretsProvider = new ServiceCollection()
    .AddZbSecrets(builder.Configuration, "Secrets")
    .BuildServiceProvider())
#pragma warning restore ASP0000
{
    await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>().MigrateAsync(default);
    var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
    await new SecretReferenceExpander(resolver)
        .ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, default);
}
```

Usings: `ZB.MOM.WW.Secrets.Abstractions`, `.Configuration`, `.DependencyInjection`, `.Sqlite`.
Note the cast to `IConfigurationRoot`. **ScadaBridge differs** — it assembles config in a bare
`ConfigurationBuilder` (not `WebApplication.CreateBuilder`) *before* the host exists, so its
expander runs against that `IConfigurationRoot` local, still via the same throwaway provider.

**(b) Runtime registration** — on the real host container (backs Layer B + the UI):

```csharp
builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
```

**(c) Authorization** — additive:

```csharp
builder.Services.Configure<AuthorizationOptions>(o => o.AddSecretsAuthorization());
```

**(d) UI mount** — register the RCL assembly in **both** places or `/admin/secrets` 404s:

```csharp
// Router.razor / Routes.razor:
//   AdditionalAssemblies="@(new[] { typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly })"
endpoints.MapRazorComponents<App>()
    .AddInteractiveServerRenderMode()
    .AddAdditionalAssemblies(typeof(ZB.MOM.WW.Secrets.Ui.SecretsPage).Assembly);
```

**Package plumbing (every app):** add `ZB.MOM.WW.Secrets`, `.Abstractions`, `.Ui` as
`PackageReference`s at `0.1.2`; add nuget.config `packageSourceMapping` patterns
`ZB.MOM.WW.Secrets` and `ZB.MOM.WW.Secrets.*` under the `dohertj2-gitea` source (none of the
three has a `ZB.MOM.WW.*` wildcard — the mapping is explicit, so restore fails without these).
ScadaBridge additionally needs `<PackageVersion>` rows in `Directory.Packages.props` (CPM).

---

## 4. G-4 — pre-host config-secret expansion (all three apps)

Same Layer-A mechanism (§3a); per-app insertion point and target keys:

| App | Insert expander at | Runs before | Config-secret targets |
|---|---|---|---|
| **OtOpcUa** | after `Host/Program.cs:73` (role overlay done, env/cmdline re-appended) | `AddOtOpcUaConfigDb` (`:94`) + first `ValidateOnStart` (`:102`) | `Security:Jwt:SigningKey`, `Security:Ldap:ServiceAccountPassword`, `Security:DeployApiKey`, `ConfigDb` connstr, `ServerHistorian:ApiKey` |
| **ScadaBridge** | between `Host/Program.cs:43` (`.Build()`) and `:46` (`StartupValidator.Validate`) | `StartupValidator`/`ConfigPreflight` (`:46`) | `Database:ConfigurationDb`, `Database:MachineDataDb`, `Security:Ldap:ServiceAccountPassword`, `Security:JwtSigningKey`, `InboundApi:ApiKeyPepper` — **plus** the docker-per-node committed plaintext (see its plan) |
| **mxaccessgw** | after `GatewayApplication.cs:64` (`CreateBuilder`), before `:67` (TLS) | `AddGatewayConfiguration` (`:71`), `AddZbGalaxyRepository` (`:103`), `AddZbApiKeyAuth` | `MxGateway:Ldap:ServiceAccountPassword`, `MxGateway:Galaxy:ConnectionString`, `MxGateway:ApiKeyPepper` |

Each plan also **deletes the committed dev plaintext** (and the loose `*_login.txt` files in
ScadaBridge) and, where a hardcoded default exists in an options class (mxaccessgw
`LdapOptions.cs:61`, `serviceaccount123`), removes it so a blanked appsettings can't fall back
to a leaked default.

---

## 5. G-5 — master-key provider & clustering (the real fork)

The KEK never lives in the store — it is supplied out-of-band, base64 32 bytes, and **never
committed**. The clustering posture differs sharply between the single-box gateway and the two
Akka-clustered apps.

### mxaccessgw — single box, no clustering

- **Recommended: `Environment` provider** (`ZB_SECRETS_MASTER_KEY` delivered via the NSSM
  service env, exactly how `MxGateway__Ldap__ServiceAccountPassword` is already delivered).
  Consistent with HistorianGateway; simplest operationally.
  *Best-practice fit:* container/NSSM env delivery is the standard 12-factor secret-bootstrap;
  matches the app's existing out-of-band password convention.
- **Alternative: `Dpapi` provider** (machine-bound key file). Stronger at-rest binding (the key
  is unusable off the box) but ties the store to one machine and complicates DR restore.

### OtOpcUa & ScadaBridge — Akka-clustered → every node needs the *same* KEK and the *same* rows

Both apps resolve secrets on multiple nodes (OtOpcUa driver-role nodes resolve Layer-B driver
secrets; both central nodes of a ScadaBridge pair resolve Layer-A config secrets at boot). Two
constraints follow: (1) identical KEK on every node, (2) identical store contents on every node.

> **⚠️ UPDATED 2026-07-18 — the interim posture below is superseded.** This section originally
> said the library "ships a SQLite-only store with a `NoOpSecretReplicator` — there is no built-in
> shared-SQL store and no cross-node replication," and recommended a shared-mounted-SQLite
> workaround. **That is no longer true.** G-7 shipped in `0.2.0` with two real replication
> packages. The KEK guidance below still stands verbatim; the *store* guidance does not.
> The shared-SQLite-on-a-network-share workaround should **not** be adopted — it was always a
> caveat-laden stopgap, and there is now a supported alternative.
>
> Current topology options and the per-app recommendation live in
> [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md) §2.
> In brief: **OtOpcUa → Akka peer-to-peer** (one cluster, pub/sub already in product use);
> **ScadaBridge → SQL-Server hub mode** (central and each site are *separate* Akka clusters, so
> `DistributedPubSub` cannot cross that boundary — an Akka replicator there would create secret
> islands that never reconcile while every node reports healthy).

**KEK requirement (unchanged, and independent of store choice):** every node must resolve the
**same** master key. Deliver it via `MasterKey.Source=File` with a read-only mounted key file
identical on every node, or the same `ZB_SECRETS_MASTER_KEY` env value. A node with a different
KEK stores replicated rows fine and then fails closed on every resolve with a `kek_id` mismatch —
it reads like data corruption but is a deployment error.

**Store options as of `0.2.0`** (all three preserve ciphertext-only-at-rest):

- **Shared SQL store** (`AddZbSecretsSqlServerStore`) — one database, one copy of each row, no
  replication and no reconciliation. Simplest; the trade is that a node partitioned from the DB
  falls back only to the resolver's short TTL cache.
- **Local store + SQL hub** (`AddZbSecretsSqlServerReplication`) — each node keeps a local store
  and syncs bidirectionally with a central hub. Survives partition; converges eventually.
- **Akka peer-to-peer** (`AddZbSecretsAkkaReplication`) — no shared database at all; live
  broadcast plus periodic manifest anti-entropy **within a single Akka cluster**.

**Rotation is per independent store.** G-8 `rewrap-all` runs once for a shared store; once per
node *and* against the hub in hub mode; once per node in Akka mode. Re-wraps deliberately do not
replicate (they leave `revision`/`updated_utc` untouched so LWW ignores them).

---

## 6. G-6 — mount `/admin/secrets` (all three apps)

Same mechanism (§3d) in each dashboard:

| App | Dashboard project | Register RCL assembly in |
|---|---|---|
| OtOpcUa | `…OtOpcUa.AdminUI` | `EndpointRouteBuilderExtensions.cs:31` (`MapRazorComponents<TApp>()` — add `.AddAdditionalAssemblies`) **and** thread `AdditionalAssemblies` from `App.razor:22` into `<Routes/>` (param already plumbed at `Routes.razor:8,37-38`) |
| ScadaBridge | `…ScadaBridge.CentralUI` | `EndpointExtensions.cs:28` `.AddAdditionalAssemblies(...)` **and** `Host/Components/Routes.razor:3` `AdditionalAssemblies` |
| mxaccessgw | `…MxGateway.Server` (Dashboard) | `DashboardEndpointRouteBuilderExtensions.cs:133-135` (add `.AddAdditionalAssemblies`) **and** `Dashboard/Components/Routes.razor:1` (add `AdditionalAssemblies`) |

All three already have an `Administrator` canonical role, so `AddSecretsAuthorization()` grants
secrets access to admins with no new role mapping — **except** verify the claim-type match
below.

**Claim-type verification (ScadaBridge especially):** the library's policies use
`RequireRole(...)` (reads `ClaimsIdentity.RoleClaimType` / `ZbClaimTypes.Role`). OtOpcUa and
mxaccessgw already gate on role claims, so they match. ScadaBridge gates on
`RequireClaim(JwtTokenService.RoleClaimType, …)` — its plan includes a task to confirm
`JwtTokenService.RoleClaimType` equals the principal's role-claim type the library reads, or an
Administrator won't satisfy `secrets:manage`/`secrets:reveal` even though ScadaBridge's own
`RequireAdmin` works.

---

## 7. G-2 / G-3 — Layer-B database-resident secrets

### G-2 — OtOpcUa driver secrets (highest value; its own plan, full detail)

Three call sites read `DriverConfig`/API-key JSON and must resolve `secret:` refs at runtime:
`OpcUaClientDriverFactoryExtensions.CreateInstance` (`:54`), `OpcUaClientDriverProbe`
(`:38`, AdminUI Test-Connect), and `GalaxySecretRef.ResolveApiKey` (`:43`, currently a static
method with `env:`/`file:`/`dev:`/literal arms — add a `secret:` arm). Structural gotchas: the
`Drivers` project does not reference `…Security`, so `ISecretResolver` must be threaded in as a
dependency (and `GalaxySecretRef` needs a signature change off `static`); the resolver must be
registered **unconditionally** (driver-role nodes have no auth/Data-Protection/AdminUI); the
AdminUI editors must round-trip `secret:` refs rather than resolving-then-resaving cleartext.

### G-3 — ScadaBridge MxGateway `ApiKey` (its own plan)

The `ApiKey` is a field inside the `DataConnection.PrimaryConfiguration`/`BackupConfiguration`
JSON blob (`MxGatewayEndpointConfig.ApiKey`), consumed at `MxGatewayDataConnection.cs:96`.
Because it is a field *within* a JSON column (not its own column), the existing
`EncryptedStringConverter` cannot target it directly. Two options:

- **Recommended: store a `secret:<name>` ref in the `ApiKey` field and resolve via
  `ISecretResolver.GetAsync` at consumption** (`MxGatewayDataConnection.cs:96`). Consistent with
  G-2's Layer-B pattern; leaves the JSON blob human-readable; no schema change.
  *Best-practice fit:* same indirection model as OtOpcUa's driver secrets — one mental model
  across the family.
- **Alternative: encrypt the whole `PrimaryConfiguration`/`BackupConfiguration` column** with
  `EncryptedStringConverter`. Fewer code touch-points but encrypts a mixed blob (the scrubber
  and any display/edit path must decrypt first), and the column is currently read in several
  places — higher blast radius.

---

## 8. Cross-cutting gotchas (apply to every plan)

1. **Fail-closed at boot.** A `${secret:missing}` throws `SecretNotFoundException` during the
   pre-host expansion — the app won't start. Seed every referenced secret (via the `secret` CLI
   or the UI) before switching a config value to a token. Plans stage this: add the token *and*
   seed the secret in the same task, and keep a rollback (the env-var override still works if
   the token is reverted).
2. **`_`-prefixed comment keys are skipped** — keep documentation example tokens under a
   `_secretsComment`-style key (the family already uses `_secrets`/`_nodeName`); they won't be
   resolved. Confirmed safe in every app's appsettings.
3. **Register `AddZbSecrets` on the runtime host too** (§3b) even for apps that only need
   Layer A today — the UI (G-6) and any future Layer-B use need the runtime resolver, and
   `SecretsMigrationHostedService` keeps the store schema current.
4. **Audit is automatic.** `AddZbSecrets` lazily binds `IAuditWriter` — all three apps have
   `ZB.MOM.WW.Audit` registered, so reveals/resolves audit through the app's existing writer
   with no ordering constraint (falls back to no-op otherwise). The reveal audit event was
   proven to carry **no plaintext** in HistorianGateway.
5. **Do not disturb existing Data Protection key rings.** OtOpcUa/ScadaBridge persist DP keys to
   their ConfigDb (`SetApplicationName` on OtOpcUa; none on ScadaBridge); mxaccessgw uses the
   framework-default provider for SignalR hub tokens only. Secrets envelope crypto is
   independent of Data Protection — do **not** reconfigure DP as part of this work, or you risk
   invalidating cookies/hub-tokens.
6. **KEK is never committed** and never printed. Same discipline as `$GITEA_TOKEN`.

---

## 9. Recommended rollout order

1. **mxaccessgw first** — simplest (single box, no clustering, no Layer-B). Proves the
   template a second time after HistorianGateway on the least-risky app.
2. **OtOpcUa** — highest security value (retires cleartext-in-DB driver secrets), but the
   heaviest (Layer A + Layer B + clustered KEK). Do Layer A (G-4) + UI (G-6) first, then the
   Layer-B driver work (G-2) as a separate reviewable slice.
3. **ScadaBridge** — heaviest config-secret surface (incl. docker-committed plaintext) + the
   claim-type verification + G-3. Benefits from the OtOpcUa Layer-B pattern being settled first.

Each app is an independent branch (`feat/adopt-zb-secrets` per repo), mirroring how auth /
theme / audit were adopted across the family — commit + review, then FF-merge to the repo
default and push to Gitea.

## 10. Testing strategy (per app)

- **Offline:** unit test the driver/connection resolver hook (G-2/G-3) with a fake
  `ISecretResolver`; assert `secret:` refs resolve and unknown refs fail closed. Confirm the
  app builds 0-warning with the three new package refs.
- **Boot smoke:** start the app with one config value switched to `${secret:…}` and the secret
  seeded; assert clean boot. Negative control: unseed the secret → assert
  `SecretNotFoundException` at startup (proves fail-closed, exactly as HistorianGateway did).
- **UI:** browser-verify `/admin/secrets` — unauthenticated → login redirect (policy enforced);
  Administrator → metadata-only list + reveal shows plaintext + an audit row with no plaintext.
- **Live (VPN):** for OtOpcUa/ScadaBridge, prove a real driver/connection secret (`secret:` ref)
  authenticates real traffic, mirroring the HistorianGateway live proof. Requires the shared
  KEK present on the box.

## 11. Task tracking

Each per-repo plan co-locates its `*.tasks.json` beside the `.md`. The items this design deferred
— G-7 (replication) and G-8 (`RewrapAll` KEK rotation) — were **both built and shipped in `0.2.0`
on 2026-07-18**, G-7 as *two* packages (`.Replicator.SqlServer` and `.Replicator.AkkaDotNet`)
rather than the single deferred Akka replicator this section anticipated. Adopting them in the
apps is [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md);
remaining backlog stays in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md).