Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.
Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.
Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
scadaproj#1 holds the defect (the library lives here); lmxopcua#482 is the
consumer-side tracker keeping Secrets:Replication:Enabled=false until it is fixed.
SQL-Server hub topology (ScadaBridge) is live-validated against a real SQL Server
2022 with two nodes: write->resolve via hub, tombstone propagation without
resurrection, bidirectional sweep, partitioned-node local resolution (site
autonomy - the reason hub mode was chosen), and wrong-KEK fail-closed via
SecretDecryptionException.
Akka peer-to-peer topology (OtOpcUa) FAILS the gate: resolving ISecretReplicator
hangs indefinitely in a real clustered process. Reproduced twice on a healthy
2-node cluster; isolated to constructing AkkaSecretReplicator -> ActorRef ->
system.ActorOf, NOT nested DI resolution. The library's own 2-node suite passes,
so the trigger is creation through DI in a hosted process. Root cause not yet
identified.
Impact: OtOpcUa would hang at startup if replication were enabled, since the
startup hook resolves ISecretStore. Harmless today only because the flag defaults
false. Do NOT enable Secrets:Replication:Enabled in OtOpcUa.
The gate did its job - it caught a blocking defect that every offline suite,
including the library's own real-cluster tests, had passed over.
- All 4 apps on 0.2.1 (local branches, unpushed).
- Records the corrected security story: the version bump closed NO advisory (all four
repos already resolved patched 2.1.12); the real live vulnerability was in ScadaBridge,
masked by a NuGetAuditSuppress, and was fixed by separate work.
- Records the upstream 0.2.0 inert-Akka-replicator defect, its root cause (DI extensions
with no container-building test - third instance of that class), and the 0.2.1 fix.
- Marks clustered topology WIRED-but-default-OFF and explicitly NOT live-validated;
Task 9 remains open and the topology is not 'adopted' until it passes.
I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.
The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.
Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.
170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.
Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.
Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.
Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Packing Secrets 0.1.3 surfaced NU1903: Microsoft.Data.Sqlite 10.0.7 pulls
SQLitePCLRaw.lib.e_sqlite3 2.1.11, which carries high-severity advisory
GHSA-2m69-gcr7-jv3q. Auth.ApiKeys had the same exposure and was already shipped
at 0.1.4 to three consumers, so both libs are fixed rather than only the one
being published.
Fixed the way ZB.MOM.WW.LocalDb already had it: CentralPackageTransitivePinningEnabled
plus a pin to the patched 2.1.12. Bumping Microsoft.Data.Sqlite does not help --
even 10.0.10 still resolves 2.1.11 -- so the pin is the actual fix. The pin reaches
consumers: both nuspecs now declare SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12 as a direct
dependency, verified by restoring the published packages into a scratch project,
which resolves 2.1.12 and scans clean.
Auth goes 0.1.4 -> 0.1.5 with no API change (0.1.4 plus the pin). Suites re-run
after the native-lib swap with identical counts, so no behavioral regression:
Secrets 97 pass/1 skip, Auth 215 pass/1 skip (both skips are Windows-only DPAPI
and opt-in LDAP).
Consumers are NOT yet bumped and remain on vulnerable versions: mxaccessgw 0.1.4,
HistorianGateway 0.1.4, ScadaBridge 0.1.3. Secrets consumers all sit at 0.1.2 and
also lack KEK rotation.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Publishes the three ZB.MOM.WW.LocalDb packages to the Gitea feed (restore-verified
from a scratch consumer) and adds the build/push.sh the other shared libs already have.
The status prose across CLAUDE.md, README.md and components/*/GAPS.md had drifted from
reality, so it was re-derived from the feed listing and the actual PackageReferences +
registration calls on each consumer's default branch rather than from prior claims.
Five claims were false: Health "not yet adopted" (all four apps wire MapZbHealth),
GalaxyRepository's mxaccessgw adoption "a follow-on" (its Server wires
AddZbGalaxyRepository), Configuration "not yet pushed", Secrets G-8 "not yet
committed", and Theme pinned at 0.2.0 (all four are on 0.3.1). Every doc also said
"three apps" while HistorianGateway is a fourth consumer of seven libs, and all
eight libraries' test counts were stale (re-ran each suite; all green).
Surfaces one previously unrecorded gap: Secrets source is at 0.1.3 with KEK rotation
committed, but the feed tops out at 0.1.2, so no app can consume rotation until it
is published.
Health and observability divergence tables are labelled historical, not re-verified —
the libraries are proven wired, but per-app probe coverage vs spec was not re-walked.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3:
- ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only
(bodies never re-encrypted; revision/timestamps preserved -> invisible to
cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed
wraps; DEK zeroed on all paths.
- ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek):
updates only the 4 wrap columns + kek_id, compare-and-swap on the current
wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a
review-caught TOCTOU).
- KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl.
tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK.
- `secret rewrap-all` CLI verb: key material only via env-var name / file path,
JSON counts report; README section + operator runbook.
Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end
CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed).
G-7 (clustered replication) — designed + planned, no code:
- Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka
replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc +
executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*.
Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
Record the OtOpcUa live wonder gate (item 2). A throwaway harness drove OtOpcUa's
real GalaxyDriverBrowser (secret: ApiKeySecretRef resolved via ISecretResolver) against
the real production MxGateway: dummy secret: ref -> MxGatewayAuthenticationException,
real metadata:read key -> CONNECTED + browsed the real Galaxy root (10 nodes). Temp key
minted via the dashboard, then revoked (functionally proven) + deleted; no prod key
touched; harness + scratch stores destroyed. All four apps now G-2..G-6 done + live-proven.
Record the OtOpcUa secrets adoption (G-2/G-4/G-5/G-6, the last app): FF-merged +
pushed to lmxopcua origin/master @872cf7e3. Update the components/secrets/GAPS.md
OtOpcUa bullet (execution record + the three verified plan deviations + the regression
the gate caught) and the CLAUDE.md secrets-component-table row. All four apps
(HistorianGateway, mxaccessgw, ScadaBridge, OtOpcUa) now adopt ZB.MOM.WW.Secrets.
The v3.0 dual-namespace rewrite (PR #472) moved anchors and reshaped the
build, so the plan is refreshed to stay executable: CPM correction (OtOpcUa
is on Central Package Management, not inline versions), OpcUaClient options
in .Contracts + GalaxyDriverBrowser in .Browser, DriverHostActor line
shifts, and appsettings/owner precision. Driver-secret flow itself was
untouched by v3.0, so the two-layer approach holds.
Add components/secrets/ (SPEC, realized shared-contract, code-verified
current-state for OtOpcUa/mxaccessgw/ScadaBridge, GAPS adoption backlog) and
register the Secrets row in CLAUDE.md + components/README.md. The
ZB.MOM.WW.Secrets lib is built + published 0.1.2 + reference-consumer-proven
(HistorianGateway, live vs the wonder historian); per-app adoption is the
tracked follow-on.
Joseph Doherty