- All 4 apps on 0.2.1 (local branches, unpushed).
- Records the corrected security story: the version bump closed NO advisory (all four
repos already resolved patched 2.1.12); the real live vulnerability was in ScadaBridge,
masked by a NuGetAuditSuppress, and was fixed by separate work.
- Records the upstream 0.2.0 inert-Akka-replicator defect, its root cause (DI extensions
with no container-building test - third instance of that class), and the 0.2.1 fix.
- Marks clustered topology WIRED-but-default-OFF and explicitly NOT live-validated;
Task 9 remains open and the topology is not 'adopted' until it passes.
> Task 2. This app is a single Windows/NSSM box with no Akka clustering, so **no replication
> package and no topology apply**. The bump still matters: `0.2.0` carries a high-severity
> transitive `SQLitePCLRaw` advisory fix that `0.1.2` lacks.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Shared context: docs/plans/2026-07-16-secrets-adoption-design.md.
**Goal:** Adopt the already-built, already-published `ZB.MOM.WW.Secrets` library (`0.1.2`, Gitea feed, reference-consumer-proven in HistorianGateway) into **MxAccessGateway** — retiring the plaintext LDAP bind password (and any password-bearing config) at rest behind pre-host `${secret:}` expansion, adding a runtime `ISecretResolver`, and mounting the `/admin/secrets` Blazor UI. Covers gaps **G-4** (pre-host config-secret expansion), **G-5** (master-key provider), and **G-6** (mount UI). There is **no G-2/G-3** for this app — it has no database-resident driver/connection secrets.
> (`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`): it is a single Akka cluster on 1.5.62, already uses
> `DistributedPubSub` in product code, registers its `ActorSystem` in DI via `Akka.Hosting`, and
> its driver-role nodes benefit from resolving Layer-B driver secrets while partitioned.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Shared context: docs/plans/2026-07-16-secrets-adoption-design.md.
**Goal:** Adopt the already-built, already-published `ZB.MOM.WW.Secrets` library (@ `0.1.2`, Gitea feed, reference-consumer-proven in HistorianGateway) into **OtOpcUa** — retiring plaintext credentials at rest. Covers **G-4** (pre-host `${secret:}` config-secret expansion), **G-6** (mount `/admin/secrets` UI), **G-5** (clustered master-key posture), and the highest-value gap **G-2** (retire cleartext-in-DB driver secrets via runtime `ISecretResolver`).
> Reason: central and each site are **separate Akka clusters** with their own `seed-nodes` lists,
> so `DistributedPubSub` cannot cross the central↔site boundary — an Akka replicator here would
> converge each cluster internally and leave **secret islands that never reconcile**, while every
> node reports healthy. Hub mode gives sites local reads through a WAN outage, which is the
> premise of the hub-and-spoke architecture.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Shared context: [`docs/plans/2026-07-16-secrets-adoption-design.md`](2026-07-16-secrets-adoption-design.md) — read it first; this plan **references** that design (library API §1, two layers §2, wiring template §3, G-4 §4, G-5 §5, G-6 §6 incl. the ScadaBridge claim-type caveat, G-3 §7, gotchas §8) rather than re-deriving it.
**Goal:** Adopt the already-built, already-published `ZB.MOM.WW.Secrets` library (`0.1.2`, Gitea feed, reference-consumer-proven in HistorianGateway) into **ScadaBridge** — retiring plaintext credentials at rest. Concretely: **G-3** (retire the MxGateway per-endpoint `ApiKey` plaintext-in-ConfigDb via a runtime `ISecretResolver``secret:` ref), **G-4** (pre-host `${secret:}` config-secret expansion — ScadaBridge is the heaviest config-secret surface in the family), **G-5** (clustered master-key posture for the central pair + site nodes), **G-6** (mount `/admin/secrets` in CentralUI, with a mandatory claim-type verification).
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Move all four family apps from `ZB.MOM.WW.Secrets``0.1.2` to `0.2.0`, and adopt a **clustered secrets topology** in the two Akka-clustered apps (OtOpcUa, ScadaBridge) so a secret written on one node is resolvable on every node that needs it.
**Architecture:** No library construction — `0.2.0` is built, published, and restore-verified on the Gitea feed (5 packages). Two distinct workstreams: (A) a **pure version bump** for all four apps, which is behaviour-preserving and independently valuable (it picks up the G-8 KEK-rotation surface and a **high-severity transitive security fix**); (B) **topology adoption** for the two clustered apps, which is *configuration + one DI call*, no product-code change. The topology choice is **not the same for both apps** — see §2, this is the load-bearing design point.
**Tech Stack:** .NET 10, C#, `ZB.MOM.WW.Secrets{,.Abstractions,.Ui}` @ `0.2.0`, plus `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (OtOpcUa) / `ZB.MOM.WW.Secrets.Replicator.SqlServer` (ScadaBridge), all from the `dohertj2-gitea` feed. Akka.NET **1.5.62** in both clustered apps — an exact match with the replicator package's Akka dependency.
**Predecessors:** [`2026-07-16-secrets-adoption-design.md`](2026-07-16-secrets-adoption-design.md) (the adoption template — §5 rewritten by this work), the three executed per-app adoption plans, and [`2026-07-17-secrets-g7-clustered-replication-design.md`](2026-07-17-secrets-g7-clustered-replication-design.md) (the fork analysis, now a *deployment* decision rather than a build one).
---
## 1. Why bump at all
> **⚠️ CORRECTED TWICE during execution, 2026-07-18 — read this before citing the bump's value.**
> The original text claimed the bump "closes a high-severity advisory in all four apps." A first
> correction narrowed that to "two apps." **Both were wrong.** Measured by A/B against the `0.1.2`
> baseline in every repo: **all four already resolved the patched `SQLitePCLRaw.lib.e_sqlite3
> 2.1.12` before this bump.** The Secrets `0.2.0` upgrade delivers **no security change to any
> app.** The lesson — the same one G-7's post-build review taught — is that a plausible mechanism
> is not evidence. Both wrong claims came from reasoning about what `0.2.0`'s nuspec *should*
> deliver instead of measuring what each repo *already resolved*.
`0.2.0` does declare a direct dependency on the patched `SQLitePCLRaw.lib.e_sqlite3 2.1.12`
(GHSA-2m69-gcr7-jv3q, high). That is real — it is simply **redundant everywhere**, because each
repo had already arrived at ≥`2.1.12` by another route:
| App | How it already resolved ≥ 2.1.12 before the bump | Security delta from this bump |
|---|---|---|
| HistorianGateway | Direct pin `SQLitePCLRaw.lib.e_sqlite3 3.50.3` (cites a *different* CVE, CVE-2025-6965) | **None** |
Both suppressions rested on false premises: mxaccessgw's said *"no patched e_sqlite3 release exists
yet"*; ScadaBridge's said *"the only patched native lib is the SQLitePCLRaw 3.x line."***`2.1.12`
patches this advisory within the 2.1.x line** — verified empirically (suppression removed + pin in
place → forced restore reports no NU1903). HistorianGateway's separate `3.50.3` pin cites
**CVE-2025-6965**, a *different* advisory; that is the likely source of the "only 3.x" confusion.
**Fix applied:** explicit `PackageReference` to `SQLitePCLRaw.lib.e_sqlite3 2.1.12` in every
SQLite-consuming project, suppressions deleted in both repos (ScadaBridge `2a0faeab`, mxaccessgw
`2f0cfe3`). HistorianGateway and OtOpcUa needed nothing — genuinely fixed already, no suppressions.
**Rejected alternative:**`CentralPackageTransitivePinningEnabled`, which clears it in one line but
makes every central version a ceiling for transitive resolution too — immediately demanding bumps
to `Google.Protobuf`, `Grpc.Net.Client`, `Microsoft.Data.SqlClient` and `Newtonsoft.Json`. That is
a gRPC/data-access change to a production SCADA platform and belongs in its own reviewed commit,
not smuggled in under a security fix. **Recorded as a genuine follow-up** — it is the better
long-term posture.
Verified: ScadaBridge 55 projects build 0 warn / 0 error, **1108 tests pass across the SQLite
layer**; mxaccessgw 0 warn / 0 error with test results identical to baseline.
Tasks 1–4 remain four independent, single-file, behaviour-preserving edits in four separate repos
— **fully parallelizable**.
### Build/test expectations — corrected against reality
The original text expected "0 warnings" everywhere. Measured, on the `0.1.2` baseline:
- **OtOpcUa** emits **861** pre-existing warnings (`OTOPCUA0001` / xUnit analyzers). Identical count before and after the bump. Do not expect 0.
- **HistorianGateway**'s full-solution restore **fails** on an AngleSharp `NU1902` advisory promoted to an error by `TreatWarningsAsErrors` — pre-existing, reproduced on the unmodified base branch.
- **MxAccessGateway** has no `src/MxGateway.sln`; the solutions are `src/ZB.MOM.WW.MxGateway.slnx` and `src/ZB.MOM.WW.MxGateway.NonWindows.slnx` (the macOS-safe subset omitting the x86 net48 worker). **`CLAUDE.md`'s recorded build command is stale** and should be corrected.
- Several suites have pre-existing environment-gated failures on macOS (live-historian DNS, Unix-domain-socket path length, shared Docker-fixture contention). **A/B against the baseline before attributing any failure to the bump.**
## 2. The topology fork — and why the two clustered apps resolve it differently
Verified against both repos on 2026-07-18. This is the single most important section; getting it wrong produces a cluster that looks healthy and silently fails to converge.
### OtOpcUa → **Akka peer-to-peer** (recommended)
- OtOpcUa is **one Akka cluster**. Every node shares a single `seed-nodes` list, so `DistributedPubSub` reaches every member. The app **already uses `DistributedPubSub` heavily** in product code (`DriverHostActor.cs:419` caches the mediator; driver-health, alarm, and deployment-ack topics all ride it), so the transport is proven in-app, not newly introduced.
- OtOpcUa registers its `ActorSystem`**in DI via `Akka.Hosting`** — which is exactly what `SecretReplicationActorProvider` requires.
- **Driver-role nodes resolve Layer-B driver secrets** (Galaxy API key, OpcUaClient passwords) at driver-session-open. Those nodes should keep starting drivers when central SQL is unreachable; peer-to-peer replication serves reads from local state, so a partitioned driver node keeps working on last-known-good and re-converges on heal.
- *Best-practice fit:* matches the app's existing cluster-communication idiom, adds no new infrastructure dependency, and preserves partition tolerance for the node role that most needs it.
- **Alternative:** shared SQL store against the existing ConfigDb (`AddZbSecretsSqlServerStore`) — simpler (zero distributed failure modes, one `rewrap-all`), but couples every driver node's secret resolution to ConfigDb reachability. Prefer this **only** if you decide driver nodes should hard-fail rather than run on cached secrets during a ConfigDb outage.
### ScadaBridge → **SQL-Server hub mode** (recommended) — Akka *cannot* do the job here
- ScadaBridge is **hub-and-spoke: one central cluster + N site clusters, each a SEPARATE Akka cluster** with its own `seed-nodes` list (`AkkaHostedService.BuildHocon`, `seed-nodes = [...]` per node). Central↔site traffic goes through `CentralCommunicationActor`, not cluster membership.
- **Therefore `DistributedPubSub` is cluster-scoped and does not cross the central↔site boundary.** An Akka replicator would converge the central pair among themselves and each site pair among themselves, producing **independent secret islands that never reconcile** — while every node reports healthy. This is the failure mode to avoid, and it is why "integrate with Akka" is the wrong answer for this app.
- Hub mode (`AddZbSecretsSqlServerReplication`) fits the topology exactly: each node keeps a **local** store and syncs bidirectionally with a shared central SQL hub. Site nodes keep resolving secrets **through a WAN outage** — which is the entire point of the hub-and-spoke design — and re-converge when the link returns.
- *Best-practice fit:* mirrors how ScadaBridge already treats central SQL (config, audit forwarding) — authoritative when reachable, locally buffered when not.
- **Alternative:** shared SQL store (`AddZbSecretsSqlServerStore`) — one store, no reconciliation, but a site that loses the WAN loses secret resolution past the resolver's short TTL cache. **Not recommended**: it contradicts the site-autonomy premise of the architecture.
Both are single-box, non-clustered. No replication package, no topology. Tasks 1–2.
## 3. Constraints that apply to every topology
1.**Same KEK on every node — non-negotiable.** Rows are ciphertext-only; a node whose master key differs stores replicated rows fine and then **fails closed on every resolve** with a `kek_id` mismatch. It reads like data corruption but is a deployment error. Verify with the digest check in [`clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md) §0 **before** enabling replication.
2.**KEK rotation does not replicate.** A re-wrap deliberately leaves `revision`/`updated_utc` untouched so it stays invisible to LWW. `rewrap-all` runs **once per independent store** — once for a shared store; once per node *and* against the hub in hub mode; once per node in Akka mode.
3.**`nuget.config` needs no change.** Both clustered repos already map `ZB.MOM.WW.Secrets.*` to `dohertj2-gitea` (OtOpcUa `NuGet.config:29`, ScadaBridge `nuget.config:29`), and that pattern already covers `ZB.MOM.WW.Secrets.Replicator.SqlServer` / `.Replicator.AkkaDotNet`. **Verified** — do not add redundant patterns.
4.**Do not reconfigure Data Protection.** Unchanged from the original adoption: secrets envelope crypto is independent of DP, and touching DP risks invalidating cookies/hub tokens.
cd ~/Desktop/HistorianGateway && dotnet build ZB.MOM.WW.HistorianGateway.slnx
```
**Expected:** succeeds, 0 warnings. `0.2.0` is API-compatible with `0.1.2` for everything this app uses.
3. Confirm the security fix actually landed — this is the *point* of the task, so verify it rather than assuming:
```bash
dotnet list ZB.MOM.WW.HistorianGateway.slnx package --include-transitive | grep -i sqlitepclraw
```
**Expected:** `SQLitePCLRaw.lib.e_sqlite3` resolves to `2.1.12` (not `2.1.11`).
4. Run the offline suite: `dotnet test ZB.MOM.WW.HistorianGateway.slnx`
**Expected:** green. **Known pre-existing:** an AngleSharp `NU1902` advisory warning in the test project — unrelated to this change; do not chase it, but confirm it is the *only* warning and that it predates the bump (`git stash` + rebuild to compare if unsure).
1. OtOpcUa **is** on Central Package Management — edit the three `<PackageVersion>` rows to `0.2.0`. Consuming csprojs carry bare `<PackageReference>` with no `Version`; **do not** add versions there.
**Expected:** green except the **known pre-existing** AngleSharp `NU1902` break in `CentralUI.Tests`. Confirm it predates the bump; do not fix it here.
5. Commit. Branch: `feat/secrets-0.2.0` off `origin/main`.
---
> **Checkpoint — review after Tasks 1–4.** All four apps are on `0.2.0` and the advisory is closed family-wide. This is a coherent, independently shippable slice: FF-merge and push each repo before starting Task 5. If the topology work is deferred, the plan still delivered its security value.
---
## Task 5 — OtOpcUa: add the Akka replicator package
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 7 (different repo)
**Depends on:** Task 3
**Files:**
- Modify: `~/Desktop/OtOpcUa/Directory.Packages.props` (ZB.MOM block, beside the rows edited in Task 3)
2. Add a bare reference in the **Host** csproj only (the Host owns the actor system and the secrets registration; driver projects keep referencing only `.Abstractions`):
**Expected:** resolves from `dohertj2-gitea` via the existing `ZB.MOM.WW.Secrets.*` mapping. It pulls `Akka.Cluster.Tools 1.5.62` — **the exact version OtOpcUa already pins**, so expect no downgrade/conflict. If NU1109 appears, stop: that means the Akka pins have drifted and the plan's premise needs rechecking.
> **High-risk because** it changes which `ISecretStore` implementation the container resolves on **every** node, including driver-role nodes that have no auth/AdminUI. A mistake here surfaces as drivers failing to open sessions, not as a test failure.
**Files:**
- Modify: `~/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (the existing unconditional `AddZbSecrets` call site added by the G-2 adoption)
**Leave the pre-host expander untouched** — it builds a throwaway container that must stay a plain local SQLite store. Replication is a *runtime* concern; the bootstrap provider runs before the actor system exists.
2. Merge the serializer HOCON into the actor-system config so replication messages use the package's `SerializerWithStringManifest` rather than falling back to a default serializer:
4. **Write a DI-resolution test before wiring** (this is the class of defect that shipped in the library itself — every unit test passed because nothing ever built a container):
```csharp
[Fact]
public void Secrets_store_resolves_as_the_replicating_decorator()
{
// Build the Host's service collection, then:
var store = provider.GetRequiredService<ISecretStore>();
Assert.IsType<ReplicatingSecretStore>(store);
// And the undecorated concrete store must still resolve — the decorator depends on it:
Run it first and confirm it **fails** before step 1's edit, passes after.
5. Build + full suite → **Expected:** 0 warnings, green.
6. **Verify the same KEK is on every node** before any multi-node run — per §3.1, use the digest check in `clustered-secrets.md` §0. Do not skip: a mismatch is silent until first resolve.
1. Add CPM row + bare reference for `ZB.MOM.WW.Secrets.Replicator.SqlServer` @ `0.2.0`.
2. Restore + build → **Expected:** resolves via the existing mapping; pulls `Microsoft.Data.SqlClient 6.0.2`. ScadaBridge already uses SqlClient — **if NU1109 appears, the repo pins an older SqlClient and you must reconcile the pin before continuing.** Report rather than force.
3. **Do not hand-provision the hub schema.** `AddZbSecretsSqlServerReplication` registers `SqlServerHubMigrationHostedService`, which creates it idempotently at startup. The DB and a login with `CREATE TABLE`/`CREATE SCHEMA` rights must exist — see `clustered-secrets.md` for the provisioning SQL.
> **High-risk because** it introduces a background sweep that writes to a shared central database from every site node, and because a wrong `Secrets:SqlServer:ConnectionString` on a site node fails at startup (eager `Validate()`), taking the node down.
2. Config block — the connection string is itself a secret, so deliver it the way every other ScadaBridge connstr is delivered (a `${secret:}` token resolved by the pre-host expander, which runs *before* this registration binds):
**Ordering gotcha:** seed `sql/scadabridge/secrets-hub` in the **local** store on every node first. The hub connstr cannot itself come from the hub — that is a bootstrap cycle. This is the one config value that must stay locally seeded.
3. Add the same DI-resolution test as Task 6 step 4, asserting `ReplicatingSecretStore`.
4. Build + suite → **Expected:** 0 warnings; green modulo the known AngleSharp break.
5. Verify the shared KEK across central + site nodes (§3.1).
6. Commit.
---
## Task 9 — Live convergence gate (both clustered apps)
**Classification:** high-risk
**Estimated implement time:** ~5 min (excluding rig setup)
**Parallelizable with:** none
**Depends on:** Task 6, Task 8
> This is the **G-2-class live gate** the G-7 design called for. The library's offline suite proves the algorithms (2-node in-process Akka cluster, 15 live SQL-Server tests); it cannot prove *this app's* wiring. Do not mark the topology adopted without it.
**Steps — run per app:**
1. Stand up 2 nodes (OtOpcUa: docker-dev rig; ScadaBridge: central pair, then a site node against the central hub).
2. **Write on node A**, via the `/admin/secrets` UI or the `secret` CLI.
3. **Resolve on node B.** Expected: the value appears within one sweep/announce interval (~30 s).
4. **Delete on A → confirm the tombstone propagates to B** and B no longer resolves it. (Resurrection is the classic anti-entropy bug; this is the case that catches it.)
5. **Partition test** — the reason each topology was chosen:
- OtOpcUa: stop node B's cluster link; confirm B still resolves its Layer-B driver secrets from local state; heal; confirm re-convergence.
- ScadaBridge: stop the site node's route to the central hub; confirm it still resolves; restore; confirm the sweep re-converges both directions.
6. **Negative KEK control** (proves fail-closed rather than silent-wrong): point one node at a *different* KEK, confirm it fails closed with a `kek_id` mismatch rather than returning a wrong plaintext. Revert.
7. Record results in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md) — including anything that did **not** pass.
---
## Task 10 — Docs reconcile
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** none
**Depends on:** Task 9
**Files:**
- Modify: `components/secrets/GAPS.md` (close the two Open items: version bump; topology adoption)
- Modify: `CLAUDE.md` (Secrets row: consumer pins `0.1.2` → `0.2.0`; record the adopted topology per app)
**Steps:** State what was **proven live** vs what was only built. If Task 9's partition or KEK-negative steps were skipped for lack of a rig, say so explicitly — an unproven claim in these docs is the exact failure mode [`component-status-claims-are-optimistic`](../../components/secrets/GAPS.md) already warns about.
---
## Rollback
Every task is a version/config change with a clean revert:
- Tasks 1–4: revert the version rows; `0.1.2` is still on the feed.
- Tasks 5–8: revert the `AddZbSecrets*Replication` call to plain `AddZbSecrets(config, "Secrets")` and drop the package reference. Local SQLite stores are untouched by replication — rows written while replication was on remain readable, since the KEK and schema are unchanged.
"subject":"Task 9: Live convergence gate (both clustered apps)",
"status":"pending",
"classification":"high-risk",
"blockedBy":[
6,
8
]
},
{
"id":10,
"nativeId":"16",
"subject":"Task 10: Docs reconcile",
"status":"pending",
"classification":"small",
"blockedBy":[
9
]
}
],
"checkpoints":[
{
"afterTask":4,
"note":"All four apps on 0.2.0, advisory closed family-wide. Independently shippable \u2014 FF-merge and push each repo before Task 5."
}
],
"lastUpdated":"2026-07-18",
"notes":"Tasks 1-8 complete on local feat/secrets-0.2.0 branches in 4 repos (not pushed). Apps ended on Secrets 0.2.1, not 0.2.0: adoption uncovered that 0.2.0's Akka replicator was silently inert (TryAdd order); fixed and republished. Task 9 (live gate) NOT done - needs 2-node rigs. Replication is default-OFF everywhere, so nothing ships enabled."
}
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Joseph Doherty