Compare commits

...

2 Commits

Author SHA1 Message Date
Joseph Doherty 62a96e5f22 docs(secrets): reconcile 0.2.1 adoption - what was proven vs merely built
- All 4 apps on 0.2.1 (local branches, unpushed).
- Records the corrected security story: the version bump closed NO advisory (all four
  repos already resolved patched 2.1.12); the real live vulnerability was in ScadaBridge,
  masked by a NuGetAuditSuppress, and was fixed by separate work.
- Records the upstream 0.2.0 inert-Akka-replicator defect, its root cause (DI extensions
  with no container-building test - third instance of that class), and the 0.2.1 fix.
- Marks clustered topology WIRED-but-default-OFF and explicitly NOT live-validated;
  Task 9 remains open and the topology is not 'adopted' until it passes.
2026-07-18 12:21:31 -04:00
Joseph Doherty 6e8d346670 fix(secrets): Akka replication was inert in 0.2.0 - registration order bug (0.2.1)
AddZbSecretsAkkaReplication called AddZbSecrets FIRST, which does
TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(). The package's own
TryAddSingleton<ISecretReplicator> therefore found a descriptor already present
and was silently discarded.

Consequence: ISecretReplicator resolved to the no-op sink, so every write
published into nothing; and because SecretReplicationActor is only spawned as a
side effect of constructing AkkaSecretReplicator, no actor was ever created
either. No exception, no log line - a cluster that reports healthy and silently
never converges. The worst available failure mode for a secrets store.

Fix: register ISecretReplicator BEFORE AddZbSecrets, matching what the SQL-Server
package already did. Found during OtOpcUa adoption (Task 6), which is the first
code that ever built a container around this extension.

Root cause of the gap: the SQL-Server package had a DI test asserting its
replicator type (AddZbSecretsSqlServerTests:58) and the correct order; the Akka
package had neither. Every Akka test exercised the actor, serializer and
reconciler in isolation - none built a container, so nothing could see it. This
is the third instance of the same defect class in this library (the inert
ISecretReplicator seam, the unregistered concrete SqliteSecretStore, and now
this), all of which share one cause: unit tests that never construct the DI graph.

Adds AddZbSecretsAkkaReplicationTests (5 tests) asserting registration at the
ServiceCollection level. Verified to discriminate: with the 0.2.0 order restored,
2 of the 5 fail; with the fix, all 5 pass. They assert descriptors rather than
resolving from a provider on purpose - resolving ISecretReplicator eagerly spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
which would make the test hang rather than fail.

Full suite Release-green: 175 passed, 15 skipped, no new warnings.
2026-07-18 11:36:45 -04:00
12 changed files with 789 additions and 30 deletions
+1 -1
View File
@@ -148,7 +148,7 @@ each project's **code-verified current state**, and the **gaps** between. See
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
| LocalDb (embedded cache + 2-node sync) | Built (3 pkgs `0.1.0`, **published to the Gitea feed** 2026-07-18, restore-verified from a scratch consumer; no app adoption yet) | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |
The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
+1 -1
View File
@@ -5,7 +5,7 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>latest</LangVersion>
<Version>0.2.0</Version>
<Version>0.2.1</Version>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
<PackageReadmeFile>README.md</PackageReadmeFile>
</PropertyGroup>
@@ -66,9 +66,6 @@ public static class AkkaSecretsServiceCollectionExtensions
section.Bind(eager);
eager.Validate();
// The local store + migrator come from AddZbSecrets (SQLite by default).
services.AddZbSecrets(config, secretsSectionPath);
// The actor is created lazily on first use rather than at registration: the ActorSystem is
// typically registered by the application AFTER this call, and a cluster node may also not be
// ready to join at the moment the container is built.
@@ -78,9 +75,17 @@ public static class AkkaSecretsServiceCollectionExtensions
sp.GetService<ISecretCacheInvalidator>(),
sp.GetRequiredService<IOptions<AkkaSecretsReplicationOptions>>().Value));
// MUST be registered BEFORE AddZbSecrets. That call does
// TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(), so if it ran first it would
// win and the TryAdd below would be silently discarded — leaving replication bound to the
// no-op sink, publishing nothing, with no actor ever created and no error anywhere.
// Shipped that way in 0.2.0; fixed in 0.2.1. The SqlServer package always had this order.
services.TryAddSingleton<ISecretReplicator>(sp =>
new AkkaSecretReplicator(sp.GetRequiredService<SecretReplicationActorProvider>().ActorRef));
// The local store + migrator come from AddZbSecrets (SQLite by default).
services.AddZbSecrets(config, secretsSectionPath);
// Decorate the local store so every write publishes. Resolved from the concrete SQLite store,
// not ISecretStore — asking the container for ISecretStore here would resolve this very
// decorator and recurse forever.
@@ -0,0 +1,109 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replication;
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
using ZB.MOM.WW.Secrets.Sqlite;
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.DependencyInjection;
/// <summary>
/// Registration tests for <c>AddZbSecretsAkkaReplication</c>.
/// <para>
/// These exist because 0.2.0 shipped with Akka replication silently inert and every other test in
/// this assembly still passed. The bug was pure registration ORDER: the extension called
/// <c>AddZbSecrets</c> first, which does
/// <c>TryAddSingleton&lt;ISecretReplicator, NoOpSecretReplicator&gt;()</c>, so the package's own
/// <c>TryAddSingleton&lt;ISecretReplicator&gt;</c> found a descriptor already present and was
/// silently discarded. Writes then published into a no-op sink and no replication actor was ever
/// spawned — with no exception and no log line anywhere.
/// </para>
/// <para>
/// The equivalent SQL-Server package always had a test asserting its replicator type, and always
/// had the correct order. That asymmetry is exactly why the defect landed in one package and not
/// the other, so the gap is closed here.
/// </para>
/// <para>
/// These assert against the <see cref="IServiceCollection"/> rather than a built provider on
/// purpose. Resolving <c>ISecretReplicator</c> eagerly spawns the replication actor, whose
/// <c>PreStart</c> calls <c>DistributedPubSub.Get(...)</c>; that requires a joined cluster, so a
/// provider-based test would need a real single-node cluster and would hang without one. The
/// defect was entirely about which descriptor wins, and the descriptor is where it is visible.
/// End-to-end actor behaviour is already covered by <c>TwoNodeClusterReplicationTests</c>.
/// </para>
/// </summary>
public sealed class AddZbSecretsAkkaReplicationTests
{
private static IServiceCollection Register()
{
IConfigurationRoot config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["Secrets:SqlitePath"] = Path.Combine(Path.GetTempPath(), $"zb-akkadi-{Guid.NewGuid():N}.db"),
["Secrets:Replication:AnnounceInterval"] = "00:00:30",
})
.Build();
var services = new ServiceCollection();
services.AddLogging();
services.AddZbSecretsAkkaReplication(config, "Secrets", "Secrets:Replication");
return services;
}
private static ServiceDescriptor Descriptor<T>(IServiceCollection services) =>
services.Single(d => d.ServiceType == typeof(T));
[Fact]
public void ISecretReplicator_is_not_bound_to_the_no_op_sink()
{
// THE REGRESSION TEST. Before the fix this descriptor was NoOpSecretReplicator, so every
// replicated write went nowhere. A no-op secret replicator is the worst failure mode
// available: the cluster reports healthy and silently never converges.
ServiceDescriptor descriptor = Descriptor<ISecretReplicator>(Register());
Assert.NotEqual(typeof(NoOpSecretReplicator), descriptor.ImplementationType);
}
[Fact]
public void ISecretReplicator_is_registered_by_this_package_via_a_factory()
{
// The package registers through a factory (it needs the lazily-created actor ref), whereas
// the core no-op is registered by concrete type. A factory registration is therefore proof
// that this package's descriptor won the TryAdd race rather than core's.
ServiceDescriptor descriptor = Descriptor<ISecretReplicator>(Register());
Assert.NotNull(descriptor.ImplementationFactory);
Assert.Null(descriptor.ImplementationType);
}
[Fact]
public void ISecretStore_is_decorated_so_writes_actually_publish()
{
// AddSingleton (not TryAdd) means the decorator is appended, and the LAST registration for
// a service type is what the container resolves. If this regressed to the bare SQLite
// store, writes would never publish.
ServiceDescriptor last = Register()
.Last(d => d.ServiceType == typeof(ISecretStore));
Assert.NotNull(last.ImplementationFactory);
}
[Fact]
public void The_undecorated_concrete_store_is_registered()
{
// ReplicatingSecretStore resolves SqliteSecretStore by concrete type. Core registering it
// only behind ISecretStore was the OTHER defect this library shipped, so pin it here too.
IServiceCollection services = Register();
Assert.Contains(services, d => d.ServiceType == typeof(SqliteSecretStore));
}
[Fact]
public void The_store_migrator_seam_is_registered()
{
IServiceCollection services = Register();
Assert.Contains(services, d => d.ServiceType == typeof(ISecretsStoreMigrator));
}
}
@@ -12,6 +12,7 @@
<PackageReference Include="Akka.TestKit.Xunit2" />
<PackageReference Include="Akka.Remote" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
<PackageReference Include="Microsoft.Extensions.Logging" />
</ItemGroup>
<ItemGroup>
+55 -2
View File
@@ -49,11 +49,64 @@ construction).
### Open
- ⬜ Bump the four apps `0.1.2` → `0.1.3`/`0.2.0` to pick up KEK rotation + the SQLitePCLRaw security fix.
- ✅ **All four apps bumped `0.1.2` → `0.2.1`** (2026-07-18), each on a local `feat/secrets-0.2.0`
branch, **not pushed**. HistorianGateway `bb9d923`, MxAccessGateway `b021e29`, OtOpcUa `1ccc237c`,
ScadaBridge `0083ce35`.
- **Correction:** the bump was originally justified as closing GHSA-2m69-gcr7-jv3q in all four
apps. Measured against the `0.1.2` baseline in every repo, **all four already resolved the
patched `SQLitePCLRaw.lib.e_sqlite3 2.1.12`** — two via their own pins, two transitively via
`Auth.ApiKeys 0.1.5`. The bump delivers **no security change**; its real value is version
currency, `secret rewrap-all` access, and being the precondition for the replicator packages.
- ✅ **A real SQLite vulnerability WAS found and fixed — but by separate work, not the bump.**
ScadaBridge's `NuGetAuditSuppress` was masking a **live** high-severity finding: only the Host
project resolved a patched `2.1.12`; `AuditLog`, `SiteRuntime`, `StoreAndForward`,
`SiteEventLogging` and 11 test projects still resolved the vulnerable **`2.1.11`**. Fixed with an
explicit `2.1.12` pin per SQLite-consuming project and the suppression deleted (`2a0faeab`).
MxAccessGateway's obsolete suppression removed likewise (`2f0cfe3`). Both suppressions rested on
false premises (*"no patched release exists"* / *"only the 3.x line is patched"*) — `2.1.12`
patches it within 2.1.x, verified by forced restore. HistorianGateway and OtOpcUa needed nothing.
- Rejected `CentralPackageTransitivePinningEnabled`: it clears the advisory in one line but makes
every central version a transitive ceiling, demanding `Google.Protobuf`, `Grpc.Net.Client`,
`Microsoft.Data.SqlClient` and `Newtonsoft.Json` bumps. That is a gRPC/data-access change to a
production SCADA platform and deserves its own reviewed commit. **Still open as a follow-up** —
it is the better long-term posture.
- ✅ `0.2.0` **PUBLISHED to the Gitea feed 2026-07-18** (all 5 packages; restore-verified from a
scratch consumer — resolves `SQLitePCLRaw.lib.e_sqlite3 2.1.12`, `Microsoft.Data.SqlClient 6.0.2`,
`Akka.Cluster.Tools 1.5.62`, and scans clean). Merged to `main` @ `15ef1f3`.
- ⬜ Adopt a topology in ScadaBridge / OtOpcUa (config + shared KEK; no code change needed).
- 🟡 **Topology WIRED in both clustered apps, default-OFF, NOT live-validated.**
**OtOpcUa → Akka peer-to-peer** (one cluster; `DistributedPubSub` already in product use;
`ActorSystem` in DI via `Akka.Hosting`; Akka 1.5.62 matches the package exactly).
**ScadaBridge → SQL-Server hub mode**, because central and each site are *separate* Akka clusters
— `DistributedPubSub` cannot cross that boundary, so an Akka replicator there would produce
**secret islands that never reconcile while every node reports healthy**.
- Both gate on `Secrets:Replication:Enabled`, **default false**, so current behaviour is
unchanged. ScadaBridge additionally requires a non-blank hub connection string — its options
`Validate()` eagerly, so unconditional wiring would have broken startup on every node.
- Bootstrap constraint recorded: the hub connstr **cannot come from the hub**; it must be seeded
in each node's local store or supplied via env.
- ⬜ **Task 9, the G-2-class live gate, is NOT done** — write→resolve across nodes, tombstone
propagation (no resurrection), partition survival, and the negative-KEK fail-closed control.
Needs a 2-node OtOpcUa rig and a ScadaBridge central-pair + site + SQL hub. **The topology is
not "adopted" until this passes.** Do not enable `Secrets:Replication:Enabled` anywhere first.
### Upstream defect found during adoption — `0.2.0` Akka replication was inert
- ✅ **Fixed in `0.2.1`, published to the feed 2026-07-18** (all 5 packages; all four apps consume it).
`AddZbSecretsAkkaReplication` called `AddZbSecrets` **first**, which does
`TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>()`, so the package's own `TryAdd` was
silently discarded. Replication published into a no-op sink and **no actor was ever spawned** —
no exception, no log line. The SQL-Server replicator always had the correct order.
- **Why it escaped:** the SQL-Server package had a DI test asserting its replicator type; the Akka
package had none. Every Akka test exercised the actor, serializer and reconciler in isolation —
**nothing built a container**. This is the **third** instance of this defect class in this library
(inert `ISecretReplicator` seam → unregistered concrete `SqliteSecretStore` → this), all with one
cause: DI extensions with no test that constructs the graph. `0.2.1` adds 5 such tests, verified
to discriminate (2 fail on the old order).
- **Testing note:** `Akka.TestKit` is the family convention for non-end-to-end Akka tests, but
`Akka.TestKit.Xunit2` is **xunit-v2-only** and OtOpcUa's `Host.IntegrationTests` is on xunit.v3
(`CS0433` if added). OtOpcUa's guards therefore assert on `ServiceCollection` descriptors — which
is exactly where this defect lives — while actor creation and convergence stay covered by the
library's own 2-node cluster suite.
- ⬜ Per-app live wonder gates for the G-8 rotation path.
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
drift symptom, not the forwarding filter itself (noted in the test).
@@ -1,5 +1,14 @@
# Secrets Adoption (G-2 … G-6) — Shared Design
> **✅ EXECUTED 2026-07-16 at lib `0.1.2` — all four apps adopted.** Version numbers and the
> Layer-A/Layer-B wiring template below are the **historical record of that adoption** and are
> left as-written; do not read `0.1.2` here as current. **Current version is `0.2.0`**, and the
> clustering guidance in §5 has been superseded — see
> [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md)
> for the version bump and the per-app clustered topology. Everything else here (API surface §1,
> the two resolution layers §2, the wiring template §3, the gotchas §8) remains accurate and is
> still the reference for any new consumer.
> **For Claude:** This is the shared design that the three per-repo implementation plans
> depend on. It is NOT itself executed. The executable plans are:
> - [`2026-07-16-secrets-adoption-otopcua.md`](2026-07-16-secrets-adoption-otopcua.md) — G-2, G-4, G-5, G-6
@@ -178,28 +187,40 @@ Both apps resolve secrets on multiple nodes (OtOpcUa driver-role nodes resolve L
secrets; both central nodes of a ScadaBridge pair resolve Layer-A config secrets at boot). Two
constraints follow: (1) identical KEK on every node, (2) identical store contents on every node.
The library today ships a **SQLite-only store** with a **`NoOpSecretReplicator`** — there is no
built-in shared-SQL store and no cross-node replication. So the near-term (G-5) posture is:
> **⚠️ UPDATED 2026-07-18 — the interim posture below is superseded.** This section originally
> said the library "ships a SQLite-only store with a `NoOpSecretReplicator` — there is no built-in
> shared-SQL store and no cross-node replication," and recommended a shared-mounted-SQLite
> workaround. **That is no longer true.** G-7 shipped in `0.2.0` with two real replication
> packages. The KEK guidance below still stands verbatim; the *store* guidance does not.
> The shared-SQLite-on-a-network-share workaround should **not** be adopted — it was always a
> caveat-laden stopgap, and there is now a supported alternative.
>
> Current topology options and the per-app recommendation live in
> [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md) §2.
> In brief: **OtOpcUa → Akka peer-to-peer** (one cluster, pub/sub already in product use);
> **ScadaBridge → SQL-Server hub mode** (central and each site are *separate* Akka clusters, so
> `DistributedPubSub` cannot cross that boundary — an Akka replicator there would create secret
> islands that never reconcile while every node reports healthy).
- **Recommended interim: `File` provider with a shared mounted key** (`MasterKey.Source=File`,
`FilePath` = a read-only mounted secret file identical on every node) **+ a single shared
SQLite store on a shared/replicated volume** (`SqlitePath` → the shared mount). Secrets are
written rarely, from one node's UI/CLI; every node reads the same file.
*Best-practice fit:* satisfies the library's documented hard requirement (same KEK per node)
with zero new code; acceptable because secret **writes** are rare and human-driven while
**reads** dominate. *Caveat:* SQLite over a network share has known multi-writer locking
limits — fine for read-mostly, single-writer; not a substitute for real replication.
- **Alternative (more integrated, more work): a ConfigDb-backed `ISecretStore`** mirroring each
app's existing `AddDataProtection().PersistKeysToDbContext<…>()` pattern (OtOpcUa persists its
DP key ring to ConfigDb precisely to share key material across nodes). This is the "right"
long-term shape but requires building a custom `ISecretStore` — it **overlaps G-7** and is
deferred there, not attempted in this cut.
**KEK requirement (unchanged, and independent of store choice):** every node must resolve the
**same** master key. Deliver it via `MasterKey.Source=File` with a read-only mounted key file
identical on every node, or the same `ZB_SECRETS_MASTER_KEY` env value. A node with a different
KEK stores replicated rows fine and then fails closed on every resolve with a `kek_id` mismatch —
it reads like data corruption but is a deployment error.
> **Decision to confirm at execution time:** for the two clustered apps, take the interim
> File-KEK + shared-SQLite posture now (ships G-5 with no new library code) and track the
> ConfigDb/`ISecretStore` integration under G-7 — **unless** you want the integrated store
> built first, which enlarges G-5 into library work. Each clustered plan is written for the
> interim posture and flags the G-7 hand-off.
**Store options as of `0.2.0`** (all three preserve ciphertext-only-at-rest):
- **Shared SQL store** (`AddZbSecretsSqlServerStore`) — one database, one copy of each row, no
replication and no reconciliation. Simplest; the trade is that a node partitioned from the DB
falls back only to the resolver's short TTL cache.
- **Local store + SQL hub** (`AddZbSecretsSqlServerReplication`) — each node keeps a local store
and syncs bidirectionally with a central hub. Survives partition; converges eventually.
- **Akka peer-to-peer** (`AddZbSecretsAkkaReplication`) — no shared database at all; live
broadcast plus periodic manifest anti-entropy **within a single Akka cluster**.
**Rotation is per independent store.** G-8 `rewrap-all` runs once for a shared store; once per
node *and* against the hub in hub mode; once per node in Akka mode. Re-wraps deliberately do not
replicate (they leave `revision`/`updated_utc` untouched so LWW ignores them).
---
@@ -315,6 +336,9 @@ default and push to Gitea.
## 11. Task tracking
Each per-repo plan co-locates its `*.tasks.json` beside the `.md`. Deferred items (G-7 Akka
replicator / shared-SQL `ISecretStore`, G-8 `RewrapAll` KEK-rotation) stay in
[`components/secrets/GAPS.md`](../../components/secrets/GAPS.md) and are out of scope here.
Each per-repo plan co-locates its `*.tasks.json` beside the `.md`. The items this design deferred
— G-7 (replication) and G-8 (`RewrapAll` KEK rotation) — were **both built and shipped in `0.2.0`
on 2026-07-18**, G-7 as *two* packages (`.Replicator.SqlServer` and `.Replicator.AkkaDotNet`)
rather than the single deferred Akka replicator this section anticipated. Adopting them in the
apps is [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md);
remaining backlog stays in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md).
@@ -1,4 +1,15 @@
# MxAccessGateway — Secrets Adoption (G-4, G-5, G-6) Implementation Plan
> **✅ EXECUTED 2026-07-16** — merged to `origin/main` @ `e088dfa`, box-verified fail-closed and
> encrypt-at-rest. This document is the **historical record**; the `0.1.2` pins are what was
> landed then, **not** what to use now.
>
> **Follow-on (not yet executed):** a **version bump only** —
> [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md)
> Task 2. This app is a single Windows/NSSM box with no Akka clustering, so **no replication
> package and no topology apply**. The bump still matters: `0.2.0` carries a high-severity
> transitive `SQLitePCLRaw` advisory fix that `0.1.2` lacks.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Shared context: docs/plans/2026-07-16-secrets-adoption-design.md.
**Goal:** Adopt the already-built, already-published `ZB.MOM.WW.Secrets` library (`0.1.2`, Gitea feed, reference-consumer-proven in HistorianGateway) into **MxAccessGateway** — retiring the plaintext LDAP bind password (and any password-bearing config) at rest behind pre-host `${secret:}` expansion, adding a runtime `ISecretResolver`, and mounting the `/admin/secrets` Blazor UI. Covers gaps **G-4** (pre-host config-secret expansion), **G-5** (master-key provider), and **G-6** (mount UI). There is **no G-2/G-3** for this app — it has no database-resident driver/connection secrets.
@@ -1,4 +1,16 @@
# OtOpcUa — Secrets Adoption (G-2, G-4, G-5, G-6) Implementation Plan
> **✅ EXECUTED 2026-07-16** — merged to `origin/master` @ `872cf7e3`, G-2 live-proven against the
> real production MxGateway. This document is the **historical record** of that adoption; the
> `0.1.2` pins throughout are what was landed then, **not** what to use now.
>
> **Follow-on (not yet executed):** bump to `0.2.0` and adopt clustered replication —
> [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md),
> Tasks 3, 5, 6, 9. OtOpcUa's recommended topology is **Akka peer-to-peer**
> (`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`): it is a single Akka cluster on 1.5.62, already uses
> `DistributedPubSub` in product code, registers its `ActorSystem` in DI via `Akka.Hosting`, and
> its driver-role nodes benefit from resolving Layer-B driver secrets while partitioned.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Shared context: docs/plans/2026-07-16-secrets-adoption-design.md.
**Goal:** Adopt the already-built, already-published `ZB.MOM.WW.Secrets` library (@ `0.1.2`, Gitea feed, reference-consumer-proven in HistorianGateway) into **OtOpcUa** — retiring plaintext credentials at rest. Covers **G-4** (pre-host `${secret:}` config-secret expansion), **G-6** (mount `/admin/secrets` UI), **G-5** (clustered master-key posture), and the highest-value gap **G-2** (retire cleartext-in-DB driver secrets via runtime `ISecretResolver`).
@@ -1,5 +1,19 @@
# ScadaBridge — Secrets Adoption (G-3, G-4, G-5, G-6) Implementation Plan
> **✅ EXECUTED 2026-07-16** — merged to `origin/main` @ `128f1596`, G-3 live-proven against the
> real production MxGateway gateway. This document is the **historical record**; the `0.1.2` pins
> are what was landed then, **not** what to use now.
>
> **Follow-on (not yet executed):** bump to `0.2.0` and adopt clustered replication —
> [`2026-07-18-secrets-0.2.0-upgrade-and-clustering.md`](2026-07-18-secrets-0.2.0-upgrade-and-clustering.md),
> Tasks 4, 7, 8, 9. ScadaBridge's recommended topology is **SQL-Server hub mode**
> (`ZB.MOM.WW.Secrets.Replicator.SqlServer`, `AddZbSecretsSqlServerReplication`), **not** Akka.
> Reason: central and each site are **separate Akka clusters** with their own `seed-nodes` lists,
> so `DistributedPubSub` cannot cross the central↔site boundary — an Akka replicator here would
> converge each cluster internally and leave **secret islands that never reconcile**, while every
> node reports healthy. Hub mode gives sites local reads through a WAN outage, which is the
> premise of the hub-and-spoke architecture.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Shared context: [`docs/plans/2026-07-16-secrets-adoption-design.md`](2026-07-16-secrets-adoption-design.md) — read it first; this plan **references** that design (library API §1, two layers §2, wiring template §3, G-4 §4, G-5 §5, G-6 §6 incl. the ScadaBridge claim-type caveat, G-3 §7, gotchas §8) rather than re-deriving it.
**Goal:** Adopt the already-built, already-published `ZB.MOM.WW.Secrets` library (`0.1.2`, Gitea feed, reference-consumer-proven in HistorianGateway) into **ScadaBridge** — retiring plaintext credentials at rest. Concretely: **G-3** (retire the MxGateway per-endpoint `ApiKey` plaintext-in-ConfigDb via a runtime `ISecretResolver` `secret:` ref), **G-4** (pre-host `${secret:}` config-secret expansion — ScadaBridge is the heaviest config-secret surface in the family), **G-5** (clustered master-key posture for the central pair + site nodes), **G-6** (mount `/admin/secrets` in CentralUI, with a mandatory claim-type verification).
@@ -0,0 +1,402 @@
# Secrets `0.2.0` Upgrade + Clustered Topology Adoption — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Move all four family apps from `ZB.MOM.WW.Secrets` `0.1.2` to `0.2.0`, and adopt a **clustered secrets topology** in the two Akka-clustered apps (OtOpcUa, ScadaBridge) so a secret written on one node is resolvable on every node that needs it.
**Architecture:** No library construction — `0.2.0` is built, published, and restore-verified on the Gitea feed (5 packages). Two distinct workstreams: (A) a **pure version bump** for all four apps, which is behaviour-preserving and independently valuable (it picks up the G-8 KEK-rotation surface and a **high-severity transitive security fix**); (B) **topology adoption** for the two clustered apps, which is *configuration + one DI call*, no product-code change. The topology choice is **not the same for both apps** — see §2, this is the load-bearing design point.
**Tech Stack:** .NET 10, C#, `ZB.MOM.WW.Secrets{,.Abstractions,.Ui}` @ `0.2.0`, plus `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (OtOpcUa) / `ZB.MOM.WW.Secrets.Replicator.SqlServer` (ScadaBridge), all from the `dohertj2-gitea` feed. Akka.NET **1.5.62** in both clustered apps — an exact match with the replicator package's Akka dependency.
**Predecessors:** [`2026-07-16-secrets-adoption-design.md`](2026-07-16-secrets-adoption-design.md) (the adoption template — §5 rewritten by this work), the three executed per-app adoption plans, and [`2026-07-17-secrets-g7-clustered-replication-design.md`](2026-07-17-secrets-g7-clustered-replication-design.md) (the fork analysis, now a *deployment* decision rather than a build one).
---
## 1. Why bump at all
> **⚠️ CORRECTED TWICE during execution, 2026-07-18 — read this before citing the bump's value.**
> The original text claimed the bump "closes a high-severity advisory in all four apps." A first
> correction narrowed that to "two apps." **Both were wrong.** Measured by A/B against the `0.1.2`
> baseline in every repo: **all four already resolved the patched `SQLitePCLRaw.lib.e_sqlite3
> 2.1.12` before this bump.** The Secrets `0.2.0` upgrade delivers **no security change to any
> app.** The lesson — the same one G-7's post-build review taught — is that a plausible mechanism
> is not evidence. Both wrong claims came from reasoning about what `0.2.0`'s nuspec *should*
> deliver instead of measuring what each repo *already resolved*.
`0.2.0` does declare a direct dependency on the patched `SQLitePCLRaw.lib.e_sqlite3 2.1.12`
(GHSA-2m69-gcr7-jv3q, high). That is real — it is simply **redundant everywhere**, because each
repo had already arrived at ≥`2.1.12` by another route:
| App | How it already resolved ≥ 2.1.12 before the bump | Security delta from this bump |
|---|---|---|
| HistorianGateway | Direct pin `SQLitePCLRaw.lib.e_sqlite3 3.50.3` (cites a *different* CVE, CVE-2025-6965) | **None** |
| OtOpcUa | `SQLitePCLRaw.bundle_e_sqlite3` pinned `2.1.12` in `Directory.Packages.props` | **None** |
| MxAccessGateway | Transitively from the pre-existing `ZB.MOM.WW.Auth.ApiKeys 0.1.5` reference | **None** (A/B-verified on the 0.1.2 baseline) |
| ScadaBridge | Transitively from `ZB.MOM.WW.Auth.ApiKeys 0.1.5` (commit `50d79ed1`) | **None** (A/B-verified) |
**The honest justification for Tasks 1–4 is therefore version hygiene plus a precondition:** the
replication packages in Tasks 5–9 are `0.2.0` artifacts and cannot be consumed on `0.1.2`, and
`0.2.0` exposes the G-8 KEK-rotation surface (`secret rewrap-all`) the runbooks document but no app
can currently run. That is a legitimate reason to land them — but it is **weaker than originally
stated**, and it means Tasks 1–4 are *not* independently valuable in the way this plan first
claimed. If the clustering work is abandoned, the bumps buy only currency and rotation access.
### The real SQLite finding — and it was a live vulnerability
Chasing the suppressions turned up something the version bump had nothing to do with. **Fixed
2026-07-18**, in its own commits, separate from the Secrets bump:
**ScadaBridge was shipping a live high-severity vulnerability, masked by its own suppression.**
The earlier A/B only checked the *Host* project, which resolved a patched `2.1.12` transitively via
`Auth.ApiKeys`. Every *other* SQLite consumer — `AuditLog`, `SiteRuntime`, `StoreAndForward`,
`SiteEventLogging` and 11 test projects — still resolved the vulnerable **`2.1.11`**, and the
`NuGetAuditSuppress` hid NU1903 for all of them. Proven by removing the suppression and forcing a
restore: 12+ projects immediately reported NU1903.
Both suppressions rested on false premises: mxaccessgw's said *"no patched e_sqlite3 release exists
yet"*; ScadaBridge's said *"the only patched native lib is the SQLitePCLRaw 3.x line."* **`2.1.12`
patches this advisory within the 2.1.x line** — verified empirically (suppression removed + pin in
place → forced restore reports no NU1903). HistorianGateway's separate `3.50.3` pin cites
**CVE-2025-6965**, a *different* advisory; that is the likely source of the "only 3.x" confusion.
**Fix applied:** explicit `PackageReference` to `SQLitePCLRaw.lib.e_sqlite3 2.1.12` in every
SQLite-consuming project, suppressions deleted in both repos (ScadaBridge `2a0faeab`, mxaccessgw
`2f0cfe3`). HistorianGateway and OtOpcUa needed nothing — genuinely fixed already, no suppressions.
**Rejected alternative:** `CentralPackageTransitivePinningEnabled`, which clears it in one line but
makes every central version a ceiling for transitive resolution too — immediately demanding bumps
to `Google.Protobuf`, `Grpc.Net.Client`, `Microsoft.Data.SqlClient` and `Newtonsoft.Json`. That is
a gRPC/data-access change to a production SCADA platform and belongs in its own reviewed commit,
not smuggled in under a security fix. **Recorded as a genuine follow-up** — it is the better
long-term posture.
Verified: ScadaBridge 55 projects build 0 warn / 0 error, **1108 tests pass across the SQLite
layer**; mxaccessgw 0 warn / 0 error with test results identical to baseline.
Tasks 1–4 remain four independent, single-file, behaviour-preserving edits in four separate repos
— **fully parallelizable**.
### Build/test expectations — corrected against reality
The original text expected "0 warnings" everywhere. Measured, on the `0.1.2` baseline:
- **OtOpcUa** emits **861** pre-existing warnings (`OTOPCUA0001` / xUnit analyzers). Identical count before and after the bump. Do not expect 0.
- **HistorianGateway**'s full-solution restore **fails** on an AngleSharp `NU1902` advisory promoted to an error by `TreatWarningsAsErrors` — pre-existing, reproduced on the unmodified base branch.
- **MxAccessGateway** has no `src/MxGateway.sln`; the solutions are `src/ZB.MOM.WW.MxGateway.slnx` and `src/ZB.MOM.WW.MxGateway.NonWindows.slnx` (the macOS-safe subset omitting the x86 net48 worker). **`CLAUDE.md`'s recorded build command is stale** and should be corrected.
- Several suites have pre-existing environment-gated failures on macOS (live-historian DNS, Unix-domain-socket path length, shared Docker-fixture contention). **A/B against the baseline before attributing any failure to the bump.**
## 2. The topology fork — and why the two clustered apps resolve it differently
Verified against both repos on 2026-07-18. This is the single most important section; getting it wrong produces a cluster that looks healthy and silently fails to converge.
### OtOpcUa → **Akka peer-to-peer** (recommended)
- OtOpcUa is **one Akka cluster**. Every node shares a single `seed-nodes` list, so `DistributedPubSub` reaches every member. The app **already uses `DistributedPubSub` heavily** in product code (`DriverHostActor.cs:419` caches the mediator; driver-health, alarm, and deployment-ack topics all ride it), so the transport is proven in-app, not newly introduced.
- OtOpcUa registers its `ActorSystem` **in DI via `Akka.Hosting`** — which is exactly what `SecretReplicationActorProvider` requires.
- **Driver-role nodes resolve Layer-B driver secrets** (Galaxy API key, OpcUaClient passwords) at driver-session-open. Those nodes should keep starting drivers when central SQL is unreachable; peer-to-peer replication serves reads from local state, so a partitioned driver node keeps working on last-known-good and re-converges on heal.
- *Best-practice fit:* matches the app's existing cluster-communication idiom, adds no new infrastructure dependency, and preserves partition tolerance for the node role that most needs it.
- **Alternative:** shared SQL store against the existing ConfigDb (`AddZbSecretsSqlServerStore`) — simpler (zero distributed failure modes, one `rewrap-all`), but couples every driver node's secret resolution to ConfigDb reachability. Prefer this **only** if you decide driver nodes should hard-fail rather than run on cached secrets during a ConfigDb outage.
### ScadaBridge → **SQL-Server hub mode** (recommended) — Akka *cannot* do the job here
- ScadaBridge is **hub-and-spoke: one central cluster + N site clusters, each a SEPARATE Akka cluster** with its own `seed-nodes` list (`AkkaHostedService.BuildHocon`, `seed-nodes = [...]` per node). Central↔site traffic goes through `CentralCommunicationActor`, not cluster membership.
- **Therefore `DistributedPubSub` is cluster-scoped and does not cross the central↔site boundary.** An Akka replicator would converge the central pair among themselves and each site pair among themselves, producing **independent secret islands that never reconcile** — while every node reports healthy. This is the failure mode to avoid, and it is why "integrate with Akka" is the wrong answer for this app.
- Hub mode (`AddZbSecretsSqlServerReplication`) fits the topology exactly: each node keeps a **local** store and syncs bidirectionally with a shared central SQL hub. Site nodes keep resolving secrets **through a WAN outage** — which is the entire point of the hub-and-spoke design — and re-converge when the link returns.
- *Best-practice fit:* mirrors how ScadaBridge already treats central SQL (config, audit forwarding) — authoritative when reachable, locally buffered when not.
- **Alternative:** shared SQL store (`AddZbSecretsSqlServerStore`) — one store, no reconciliation, but a site that loses the WAN loses secret resolution past the resolver's short TTL cache. **Not recommended**: it contradicts the site-autonomy premise of the architecture.
### HistorianGateway, MxAccessGateway → **version bump only**
Both are single-box, non-clustered. No replication package, no topology. Tasks 1–2.
## 3. Constraints that apply to every topology
1. **Same KEK on every node — non-negotiable.** Rows are ciphertext-only; a node whose master key differs stores replicated rows fine and then **fails closed on every resolve** with a `kek_id` mismatch. It reads like data corruption but is a deployment error. Verify with the digest check in [`clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md) §0 **before** enabling replication.
2. **KEK rotation does not replicate.** A re-wrap deliberately leaves `revision`/`updated_utc` untouched so it stays invisible to LWW. `rewrap-all` runs **once per independent store** — once for a shared store; once per node *and* against the hub in hub mode; once per node in Akka mode.
3. **`nuget.config` needs no change.** Both clustered repos already map `ZB.MOM.WW.Secrets.*` to `dohertj2-gitea` (OtOpcUa `NuGet.config:29`, ScadaBridge `nuget.config:29`), and that pattern already covers `ZB.MOM.WW.Secrets.Replicator.SqlServer` / `.Replicator.AkkaDotNet`. **Verified** — do not add redundant patterns.
4. **Do not reconfigure Data Protection.** Unchanged from the original adoption: secrets envelope crypto is independent of DP, and touching DP risks invalidating cookies/hub tokens.
---
## Task 1 — HistorianGateway: bump `0.1.2` → `0.2.0`
**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 2, Task 3, Task 4 (different repos entirely)
**Files:**
- Modify: `~/Desktop/HistorianGateway/src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:37-38`
**Steps:**
1. This repo does **not** use CPM for these — versions are inline on the `PackageReference`. Change both:
```xml
<PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.0" />
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.0" />
```
2. Restore + build:
```bash
cd ~/Desktop/HistorianGateway && dotnet build ZB.MOM.WW.HistorianGateway.slnx
```
**Expected:** succeeds, 0 warnings. `0.2.0` is API-compatible with `0.1.2` for everything this app uses.
3. Confirm the security fix actually landed — this is the *point* of the task, so verify it rather than assuming:
```bash
dotnet list ZB.MOM.WW.HistorianGateway.slnx package --include-transitive | grep -i sqlitepclraw
```
**Expected:** `SQLitePCLRaw.lib.e_sqlite3` resolves to `2.1.12` (not `2.1.11`).
4. Run the offline suite: `dotnet test ZB.MOM.WW.HistorianGateway.slnx`
**Expected:** green. **Known pre-existing:** an AngleSharp `NU1902` advisory warning in the test project — unrelated to this change; do not chase it, but confirm it is the *only* warning and that it predates the bump (`git stash` + rebuild to compare if unsure).
5. Commit:
```bash
git add -A && git commit -m "chore(secrets): bump ZB.MOM.WW.Secrets 0.1.2 -> 0.2.0 (SQLitePCLRaw advisory fix + KEK rotation)"
```
---
## Task 2 — MxAccessGateway: bump `0.1.2` → `0.2.0`
**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 3, Task 4
**Files:**
- Modify: `~/Desktop/MxAccessGateway/src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj:20-22`
**Steps:**
1. Set all three inline versions to `0.2.0` (`ZB.MOM.WW.Secrets`, `.Abstractions`, `.Ui`).
2. Build: `cd ~/Desktop/MxAccessGateway && dotnet build src/MxGateway.sln` → **Expected:** 0 warnings.
3. Verify the transitive pin as in Task 1 step 3.
4. Test: `dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj` → **Expected:** green (no MXAccess needed — fake worker).
5. Commit with the same message shape as Task 1.
---
## Task 3 — OtOpcUa: bump `0.1.2` → `0.2.0` (CPM)
**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 2, Task 4
**Files:**
- Modify: `~/Desktop/OtOpcUa/Directory.Packages.props:135-137`
**Steps:**
1. OtOpcUa **is** on Central Package Management — edit the three `<PackageVersion>` rows to `0.2.0`. Consuming csprojs carry bare `<PackageReference>` with no `Version`; **do not** add versions there.
2. Build: `cd ~/Desktop/OtOpcUa && dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **Expected:** 0 warnings (repo builds under `TreatWarningsAsErrors`).
3. Verify the transitive pin as in Task 1 step 3.
4. Test: `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → **Expected:** green.
5. Commit. Branch: `feat/secrets-0.2.0` off `origin/master` (**OtOpcUa's default is `master`**).
---
## Task 4 — ScadaBridge: bump `0.1.2` → `0.2.0` (CPM)
**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 2, Task 3
**Files:**
- Modify: `~/Desktop/ScadaBridge/Directory.Packages.props:89-91`
**Steps:**
1. Edit the three `<PackageVersion>` rows to `0.2.0`.
2. Build: `cd ~/Desktop/ScadaBridge && dotnet build ZB.MOM.WW.ScadaBridge.slnx` → **Expected:** 0 warnings.
3. Verify the transitive pin as in Task 1 step 3.
4. Test: `dotnet test ZB.MOM.WW.ScadaBridge.slnx`
**Expected:** green except the **known pre-existing** AngleSharp `NU1902` break in `CentralUI.Tests`. Confirm it predates the bump; do not fix it here.
5. Commit. Branch: `feat/secrets-0.2.0` off `origin/main`.
---
> **Checkpoint — review after Tasks 1–4.** All four apps are on `0.2.0` and the advisory is closed family-wide. This is a coherent, independently shippable slice: FF-merge and push each repo before starting Task 5. If the topology work is deferred, the plan still delivered its security value.
---
## Task 5 — OtOpcUa: add the Akka replicator package
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 7 (different repo)
**Depends on:** Task 3
**Files:**
- Modify: `~/Desktop/OtOpcUa/Directory.Packages.props` (ZB.MOM block, beside the rows edited in Task 3)
- Modify: `~/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
**Steps:**
1. Add the CPM version row:
```xml
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" Version="0.2.0" />
```
2. Add a bare reference in the **Host** csproj only (the Host owns the actor system and the secrets registration; driver projects keep referencing only `.Abstractions`):
```xml
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" />
```
3. Restore: `dotnet restore ZB.MOM.WW.OtOpcUa.slnx`
**Expected:** resolves from `dohertj2-gitea` via the existing `ZB.MOM.WW.Secrets.*` mapping. It pulls `Akka.Cluster.Tools 1.5.62` — **the exact version OtOpcUa already pins**, so expect no downgrade/conflict. If NU1109 appears, stop: that means the Akka pins have drifted and the plan's premise needs rechecking.
4. Build → **Expected:** 0 warnings, nothing consumes the package yet.
5. Commit.
---
## Task 6 — OtOpcUa: wire Akka replication + shared-KEK config
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Depends on:** Task 5
> **High-risk because** it changes which `ISecretStore` implementation the container resolves on **every** node, including driver-role nodes that have no auth/AdminUI. A mistake here surfaces as drivers failing to open sessions, not as a test failure.
**Files:**
- Modify: `~/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (the existing unconditional `AddZbSecrets` call site added by the G-2 adoption)
- Modify: `~/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`
- Test: `~/Desktop/OtOpcUa/tests/.../Host.Tests/` — DI-resolution test (see step 4)
**Steps:**
1. **Replace** the runtime registration — `AddZbSecretsAkkaReplication` calls `AddZbSecrets` internally, so calling both would double-register:
```csharp
// was: builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
builder.Services.AddZbSecretsAkkaReplication(builder.Configuration, "Secrets", "Secrets:Replication");
```
Using: `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection`.
**Leave the pre-host expander untouched** — it builds a throwaway container that must stay a plain local SQLite store. Replication is a *runtime* concern; the bootstrap provider runs before the actor system exists.
2. Merge the serializer HOCON into the actor-system config so replication messages use the package's `SerializerWithStringManifest` rather than falling back to a default serializer:
```csharp
// where the Akka HOCON is assembled:
config = config.WithFallback(AkkaSecretsReplication.SerializationConfig);
```
3. Add the config block to `appsettings.json` beside the existing `Secrets` section:
```json
"Secrets": {
"Replication": { "AnnounceInterval": "00:00:30", "ActorName": "zb-secret-replication" }
}
```
4. **Write a DI-resolution test before wiring** (this is the class of defect that shipped in the library itself — every unit test passed because nothing ever built a container):
```csharp
[Fact]
public void Secrets_store_resolves_as_the_replicating_decorator()
{
// Build the Host's service collection, then:
var store = provider.GetRequiredService<ISecretStore>();
Assert.IsType<ReplicatingSecretStore>(store);
// And the undecorated concrete store must still resolve — the decorator depends on it:
Assert.NotNull(provider.GetRequiredService<SqliteSecretStore>());
}
```
Run it first and confirm it **fails** before step 1's edit, passes after.
5. Build + full suite → **Expected:** 0 warnings, green.
6. **Verify the same KEK is on every node** before any multi-node run — per §3.1, use the digest check in `clustered-secrets.md` §0. Do not skip: a mismatch is silent until first resolve.
7. Commit.
---
## Task 7 — ScadaBridge: add the SQL-Server replicator package + hub schema
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 5 (different repo)
**Depends on:** Task 4
**Files:**
- Modify: `~/Desktop/ScadaBridge/Directory.Packages.props`
- Modify: `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj`
**Steps:**
1. Add CPM row + bare reference for `ZB.MOM.WW.Secrets.Replicator.SqlServer` @ `0.2.0`.
2. Restore + build → **Expected:** resolves via the existing mapping; pulls `Microsoft.Data.SqlClient 6.0.2`. ScadaBridge already uses SqlClient — **if NU1109 appears, the repo pins an older SqlClient and you must reconcile the pin before continuing.** Report rather than force.
3. **Do not hand-provision the hub schema.** `AddZbSecretsSqlServerReplication` registers `SqlServerHubMigrationHostedService`, which creates it idempotently at startup. The DB and a login with `CREATE TABLE`/`CREATE SCHEMA` rights must exist — see `clustered-secrets.md` for the provisioning SQL.
4. Commit.
---
## Task 8 — ScadaBridge: wire hub replication + per-node config
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Depends on:** Task 7
> **High-risk because** it introduces a background sweep that writes to a shared central database from every site node, and because a wrong `Secrets:SqlServer:ConnectionString` on a site node fails at startup (eager `Validate()`), taking the node down.
**Files:**
- Modify: `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (existing `AddZbSecrets` call site)
- Modify: `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json`
- Modify: the per-node docker appsettings under `~/Desktop/ScadaBridge/docker/`
**Steps:**
1. Replace the runtime registration (it calls `AddZbSecrets` internally — do not call both):
```csharp
builder.Services.AddZbSecretsSqlServerReplication(
builder.Configuration, "Secrets", "Secrets:SqlServer");
```
2. Config block — the connection string is itself a secret, so deliver it the way every other ScadaBridge connstr is delivered (a `${secret:}` token resolved by the pre-host expander, which runs *before* this registration binds):
```json
"Secrets": {
"SqlServer": {
"ConnectionString": "${secret:sql/scadabridge/secrets-hub}",
"SchemaName": "zbsecrets",
"SyncInterval": "00:00:30",
"SyncOnStartup": true
}
}
```
**Ordering gotcha:** seed `sql/scadabridge/secrets-hub` in the **local** store on every node first. The hub connstr cannot itself come from the hub — that is a bootstrap cycle. This is the one config value that must stay locally seeded.
3. Add the same DI-resolution test as Task 6 step 4, asserting `ReplicatingSecretStore`.
4. Build + suite → **Expected:** 0 warnings; green modulo the known AngleSharp break.
5. Verify the shared KEK across central + site nodes (§3.1).
6. Commit.
---
## Task 9 — Live convergence gate (both clustered apps)
**Classification:** high-risk
**Estimated implement time:** ~5 min (excluding rig setup)
**Parallelizable with:** none
**Depends on:** Task 6, Task 8
> This is the **G-2-class live gate** the G-7 design called for. The library's offline suite proves the algorithms (2-node in-process Akka cluster, 15 live SQL-Server tests); it cannot prove *this app's* wiring. Do not mark the topology adopted without it.
**Steps — run per app:**
1. Stand up 2 nodes (OtOpcUa: docker-dev rig; ScadaBridge: central pair, then a site node against the central hub).
2. **Write on node A**, via the `/admin/secrets` UI or the `secret` CLI.
3. **Resolve on node B.** Expected: the value appears within one sweep/announce interval (~30 s).
4. **Delete on A → confirm the tombstone propagates to B** and B no longer resolves it. (Resurrection is the classic anti-entropy bug; this is the case that catches it.)
5. **Partition test** — the reason each topology was chosen:
- OtOpcUa: stop node B's cluster link; confirm B still resolves its Layer-B driver secrets from local state; heal; confirm re-convergence.
- ScadaBridge: stop the site node's route to the central hub; confirm it still resolves; restore; confirm the sweep re-converges both directions.
6. **Negative KEK control** (proves fail-closed rather than silent-wrong): point one node at a *different* KEK, confirm it fails closed with a `kek_id` mismatch rather than returning a wrong plaintext. Revert.
7. Record results in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md) — including anything that did **not** pass.
---
## Task 10 — Docs reconcile
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** none
**Depends on:** Task 9
**Files:**
- Modify: `components/secrets/GAPS.md` (close the two Open items: version bump; topology adoption)
- Modify: `CLAUDE.md` (Secrets row: consumer pins `0.1.2` → `0.2.0`; record the adopted topology per app)
- Modify: `components/secrets/current-state/*/CURRENT-STATE.md` (per-app store/topology)
**Steps:** State what was **proven live** vs what was only built. If Task 9's partition or KEK-negative steps were skipped for lack of a rig, say so explicitly — an unproven claim in these docs is the exact failure mode [`component-status-claims-are-optimistic`](../../components/secrets/GAPS.md) already warns about.
---
## Rollback
Every task is a version/config change with a clean revert:
- Tasks 1–4: revert the version rows; `0.1.2` is still on the feed.
- Tasks 5–8: revert the `AddZbSecrets*Replication` call to plain `AddZbSecrets(config, "Secrets")` and drop the package reference. Local SQLite stores are untouched by replication — rows written while replication was on remain readable, since the KEK and schema are unchanged.
@@ -0,0 +1,128 @@
{
"planPath": "docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md",
"tasks": [
{
"id": 1,
"nativeId": "7",
"subject": "Task 1: HistorianGateway bump 0.1.2 -> 0.2.0",
"status": "completed",
"classification": "trivial",
"parallelizableWith": [
2,
3,
4
]
},
{
"id": 2,
"nativeId": "8",
"subject": "Task 2: MxAccessGateway bump 0.1.2 -> 0.2.0",
"status": "completed",
"classification": "trivial",
"parallelizableWith": [
1,
3,
4
]
},
{
"id": 3,
"nativeId": "9",
"subject": "Task 3: OtOpcUa bump 0.1.2 -> 0.2.0 (CPM)",
"status": "completed",
"classification": "trivial",
"parallelizableWith": [
1,
2,
4
]
},
{
"id": 4,
"nativeId": "10",
"subject": "Task 4: ScadaBridge bump 0.1.2 -> 0.2.0 (CPM)",
"status": "completed",
"classification": "trivial",
"parallelizableWith": [
1,
2,
3
]
},
{
"id": 5,
"nativeId": "11",
"subject": "Task 5: OtOpcUa add Akka replicator package",
"status": "completed",
"classification": "small",
"blockedBy": [
3
],
"parallelizableWith": [
7
]
},
{
"id": 6,
"nativeId": "12",
"subject": "Task 6: OtOpcUa wire Akka replication + shared KEK",
"status": "completed",
"classification": "high-risk",
"blockedBy": [
5
]
},
{
"id": 7,
"nativeId": "13",
"subject": "Task 7: ScadaBridge add SqlServer replicator package",
"status": "completed",
"classification": "standard",
"blockedBy": [
4
],
"parallelizableWith": [
5
]
},
{
"id": 8,
"nativeId": "14",
"subject": "Task 8: ScadaBridge wire hub replication + per-node config",
"status": "completed",
"classification": "high-risk",
"blockedBy": [
7
]
},
{
"id": 9,
"nativeId": "15",
"subject": "Task 9: Live convergence gate (both clustered apps)",
"status": "pending",
"classification": "high-risk",
"blockedBy": [
6,
8
]
},
{
"id": 10,
"nativeId": "16",
"subject": "Task 10: Docs reconcile",
"status": "pending",
"classification": "small",
"blockedBy": [
9
]
}
],
"checkpoints": [
{
"afterTask": 4,
"note": "All four apps on 0.2.0, advisory closed family-wide. Independently shippable \u2014 FF-merge and push each repo before Task 5."
}
],
"lastUpdated": "2026-07-18",
"notes": "Tasks 1-8 complete on local feat/secrets-0.2.0 branches in 4 repos (not pushed). Apps ended on Secrets 0.2.1, not 0.2.0: adoption uncovered that 0.2.0's Akka replicator was silently inert (TryAdd order); fixed and republished. Task 9 (live gate) NOT done - needs 2-node rigs. Replication is default-OFF everywhere, so nothing ships enabled."
}