dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(secrets): root-cause + fix the Akka replicator's hosted-process DI deadlock (0.2.2)

Browse Source 
Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.

Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.

Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-18 14:39:40 -04:00
parent 57d2d193fc
commit a1df7ef26c
8 changed files with 317 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
components/secrets/GAPS.md
+32 -18
View File
@@ -105,28 +105,42 @@ The KEK control fired spontaneously first: a re-run against a hub still holding
run (different KEK) threw `SecretDecryptionException` rather than returning garbage — unplanned but
real evidence of fail-closed.
### Live gate: Akka peer-to-peer (OtOpcUa topology) — ❌ FAILED, blocking defect
### Live gate: Akka peer-to-peer (OtOpcUa topology) — ✅ PASSES 6/6 (root-caused + fixed in `0.2.2`, 2026-07-18)
**Resolving `ISecretReplicator` hangs indefinitely** in a real clustered process. Reproduced twice
on a healthy 2-node cluster (both members confirmed `Up` before the hang). Isolated by resolving
`ISecretReplicator` *directly* rather than via `ISecretStore`, so it is **not** nested-resolution
contention — the hang is in constructing `AkkaSecretReplicator` → `SecretReplicationActorProvider.ActorRef`
→ `system.ActorOf(SecretReplicationActor.Props(...))`. Native stacks show `Monitor_Wait`.
The `0.2.1` hang is **root-caused and fixed**. It was never Akka at all: the Akka package's DI
wiring had a **circular singleton dependency** the container cannot detect through factory lambdas —
`ISecretStore` (the `ReplicatingSecretStore` decorator) → `ISecretReplicator` →
`SecretReplicationActorProvider` → `ISecretCacheInvalidator` → `DefaultSecretResolver` →
`ISecretStore` again. Resolution recurses around that loop (same-thread `Monitor` re-entry keeps it
alive) until MS.DI's `StackGuard` moves the recursion onto a fresh thread-pool thread to avoid a
stack overflow — and that thread then blocks forever on a singleton call-site lock the first thread
still holds. Managed stacks from a `dotnet-dump` of the hung process show the full cycle repeating
and both threads parked, which is the `Monitor_Wait` the native samples saw. Both earlier candidate
causes (`DistributedPubSub.Get` vs the `Lazy<IActorRef>` lock; missing `Akka.Cluster.Tools` HOCON)
are **disproven** — the actor's constructor was never even reached, and the deadlock reproduces on a
single-node rig with no clustering in play.
**Impact if enabled:** OtOpcUa would hang at startup, since the wiring resolves `ISecretStore` from a
startup hook. It is currently harmless only because `Secrets:Replication:Enabled` defaults false.
**Fix (`0.2.2`):** the one cycle-closing edge is deferred. `SecretReplicationActorProvider` now
receives a `DeferredSecretCacheInvalidator` that resolves the real invalidator on **first
eviction** — which can only happen when a replicated row is applied, strictly after the graph has
finished resolving. The SQL-Server package never had the cycle (its replicator chain never touches
the invalidator), which is exactly why the hub gate passed while Akka hung.
**Not explained by the library's own suite:** `TwoNodeClusterReplicationTests` creates the same actor
on a real 2-node cluster and passes (33 tests green). The difference is creation through DI inside a
real host process. Root cause NOT yet identified — candidates: `DistributedPubSub.Get(...)` in
`PreStart` interacting with the `Lazy<IActorRef>` (`ExecutionAndPublication`) lock, or missing
`Akka.Cluster.Tools` reference config in the composed HOCON.
**Verified:** live convergence gate re-run — **6/6 checks** on a real 2-node cluster (write→peer,
delete→tombstone, tombstoned secret no longer resolves on the peer — the check that proves the
deferred invalidator actually evicts — reverse direction, wrong-KEK fail-closed). Full Akka test
project 38/38 green.
- ⬜ **Do NOT enable `Secrets:Replication:Enabled` in OtOpcUa.** The Akka topology is not adoptable
until this deadlock is root-caused and fixed, with a regression test that creates the actor
through DI in a hosted process rather than directly from a test.
**Tracked:** [`scadaproj#1`](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) (the defect,
where the library code lives) and [`lmxopcua#482`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482)
- ✅ **Regression test now exists at the exact blind spot:** `HostedProcessResolutionTests` builds
the graph the way a host does (container-registered `ActorSystem`, hosted services started, then
resolves `ISecretStore`/`ISecretReplicator`/`ISecretResolver` under a 20 s watchdog). Verified to
discriminate: it deadlocks/fails on `0.2.1` and passes on `0.2.2`. `DeferredSecretCacheInvalidatorTests`
pins the wrapper's contract (no eager resolve; evictions forward; lookup runs once). This was the
**fourth** defect in this library visible only when the DI graph is built inside a real host.
- ⬜ OtOpcUa may now adopt the Akka topology: bump to `0.2.2` (once published), re-run its gate,
then consider enabling `Secrets:Replication:Enabled`.
**Tracked:** [`scadaproj#1`](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) (the defect —
fixed) and [`lmxopcua#482`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482)
(the consumer-side tracker: bump the package, re-run the gate, then consider enabling).
- ✅ ScadaBridge's hub topology **is** live-validated and safe to enable (config + a shared KEK).