dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Secrets: Akka replicator deadlocks on DI actor creation in a hosted process (blocks OtOpcUa clustered-secrets adoption) #1

New Issue
Closed
opened 2026-07-18 14:12:50 -04:00 by dohertj2 · 3 comments
dohertj2 commented 2026-07-18 14:12:50 -04:00
Owner
Copy Link
Copy Source

Summary

ZB.MOM.WW.Secrets.Replicator.AkkaDotNet deadlocks when its replication actor is created through DI in a real hosted process. Resolving ISecretReplicator never returns.

This blocks OtOpcUa from adopting the Akka clustered-secrets topology (Secrets:Replication:Enabled). ScadaBridge is unaffected — it uses the SQL-Server hub replicator, which passed its live gate 6/6.

Impact

If Secrets:Replication:Enabled=true were set in OtOpcUa, the host would hang at startup: the wiring registers a startup hook that resolves ISecretStore, which constructs ReplicatingSecretStore → ISecretReplicator → the actor.

It is harmless today only because the flag defaults to false in SecretsRegistration.cs.

Reproduction

Reproduced twice with a standalone 2-node harness (two ActorSystems, real dot-netty remoting, one process), both members confirmed Up before the hang:

[PASS] cluster forms (2 members Up)  — both nodes Up
 .. starting hosted services on A
 .. starting hosted services on B
 .. resolving ISecretReplicator on A FIRST
<hangs indefinitely — no timeout, no exception, no log line>

What has been ruled out

  • Not nested DI resolution. First observed resolving ISecretStore; isolated by resolving ISecretReplicator directly instead. Hangs identically, so it is not contention on the container's singleton lock for ISecretStore.
  • Not cluster formation. Both members reach Up first; the check that asserts this passes.
  • Not the 0.2.0 inert-replicator bug. That was fixed in 0.2.1 (registration order); this is 0.2.1 behaviour and is a different failure.
  • Not reproduced by the library's own tests. TwoNodeClusterReplicationTests creates the same actor on a real 2-node cluster and passes (33 tests green). The distinguishing factor is creation through DI inside a hosted process, not direct system.ActorOf from a test.

Native stacks (sample) show Monitor_Wait; managed frames were not resolvable.

Candidate causes (unverified — do not fix on this basis without confirming)

  1. SecretReplicationActor.PreStart calls DistributedPubSub.Get(Context.System).Mediator, and the extension's lazy initialisation interacts badly with the Lazy<IActorRef> (LazyThreadSafetyMode.ExecutionAndPublication) lock in SecretReplicationActorProvider.
  2. The composed HOCON lacks the Akka.Cluster.Tools reference config / akka.extensions entry that the app hosts declare explicitly, so DistributedPubSub.Get blocks rather than failing.

Note both OtOpcUa and ScadaBridge list the DistributedPubSub extension explicitly in their own HOCON — the harness (and possibly AkkaSecretsReplication.SerializationConfig) does not.

Definition of done

  • Root cause identified — not a speculative fix.
  • A regression test that creates the actor through DI in a hosted process (IHostedService + ActorSystem from the container), not directly via system.ActorOf from a test. Every existing test passes over this.
  • If it is the HOCON/extension case, AkkaSecretsReplication should supply what it needs rather than relying on the host.
  • Then re-run the Akka live gate (write→resolve, tombstone propagation, partition, negative-KEK) before enabling anywhere.

Context

  • Library lives at ZB.MOM.WW.Secrets/ in this repo; affected package ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.1 (on the Gitea feed).
  • Gate results and full detail: components/secrets/GAPS.md.
  • Adoption plan: docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md (Task 9).
  • This is the third defect in this library that offline suites passed over; all three only appear once the DI graph is built inside a real host.
## Summary `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` **deadlocks when its replication actor is created through DI in a real hosted process**. Resolving `ISecretReplicator` never returns. This **blocks OtOpcUa from adopting the Akka clustered-secrets topology** (`Secrets:Replication:Enabled`). ScadaBridge is unaffected — it uses the SQL-Server hub replicator, which passed its live gate 6/6. ## Impact If `Secrets:Replication:Enabled=true` were set in OtOpcUa, **the host would hang at startup**: the wiring registers a startup hook that resolves `ISecretStore`, which constructs `ReplicatingSecretStore` → `ISecretReplicator` → the actor. It is harmless today only because the flag **defaults to false** in `SecretsRegistration.cs`. ## Reproduction Reproduced twice with a standalone 2-node harness (two `ActorSystem`s, real dot-netty remoting, one process), both members confirmed `Up` before the hang: ``` [PASS] cluster forms (2 members Up) — both nodes Up .. starting hosted services on A .. starting hosted services on B .. resolving ISecretReplicator on A FIRST <hangs indefinitely — no timeout, no exception, no log line> ``` ## What has been ruled out - **Not nested DI resolution.** First observed resolving `ISecretStore`; isolated by resolving `ISecretReplicator` *directly* instead. Hangs identically, so it is not contention on the container's singleton lock for `ISecretStore`. - **Not cluster formation.** Both members reach `Up` first; the check that asserts this passes. - **Not the 0.2.0 inert-replicator bug.** That was fixed in 0.2.1 (registration order); this is 0.2.1 behaviour and is a *different* failure. - **Not reproduced by the library's own tests.** `TwoNodeClusterReplicationTests` creates the same actor on a real 2-node cluster and passes (33 tests green). The distinguishing factor is **creation through DI inside a hosted process**, not direct `system.ActorOf` from a test. Native stacks (`sample`) show `Monitor_Wait`; managed frames were not resolvable. ## Candidate causes (unverified — do not fix on this basis without confirming) 1. `SecretReplicationActor.PreStart` calls `DistributedPubSub.Get(Context.System).Mediator`, and the extension's lazy initialisation interacts badly with the `Lazy<IActorRef>` (`LazyThreadSafetyMode.ExecutionAndPublication`) lock in `SecretReplicationActorProvider`. 2. The composed HOCON lacks the `Akka.Cluster.Tools` reference config / `akka.extensions` entry that the app hosts declare explicitly, so `DistributedPubSub.Get` blocks rather than failing. Note both OtOpcUa and ScadaBridge list the DistributedPubSub extension explicitly in their own HOCON — the harness (and possibly `AkkaSecretsReplication.SerializationConfig`) does not. ## Definition of done - Root cause identified — **not** a speculative fix. - A regression test that creates the actor **through DI in a hosted process** (`IHostedService` + `ActorSystem` from the container), not directly via `system.ActorOf` from a test. Every existing test passes over this. - If it is the HOCON/extension case, `AkkaSecretsReplication` should supply what it needs rather than relying on the host. - Then re-run the Akka live gate (write→resolve, tombstone propagation, partition, negative-KEK) before enabling anywhere. ## Context - Library lives at `ZB.MOM.WW.Secrets/` in this repo; affected package `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` **0.2.1** (on the Gitea feed). - Gate results and full detail: `components/secrets/GAPS.md`. - Adoption plan: `docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md` (Task 9). - This is the **third** defect in this library that offline suites passed over; all three only appear once the DI graph is built inside a real host.
dohertj2 commented 2026-07-18 14:13:15 -04:00
Author
Owner
Copy Link
Copy Source

Consumer-side tracker filed: lmxopcua#482 — OtOpcUa keeps Secrets:Replication:Enabled=false until this is fixed, then bumps the package and re-runs the Akka live gate.

Consumer-side tracker filed: [`lmxopcua#482`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482) — OtOpcUa keeps `Secrets:Replication:Enabled=false` until this is fixed, then bumps the package and re-runs the Akka live gate.
dohertj2 commented 2026-07-18 14:40:18 -04:00
Author
Owner
Copy Link
Copy Source

Root cause identified and fixed — main @ a1df7ef (lib 0.2.2, not yet published to the feed)

The deadlock was never Akka. Managed stacks from a dotnet-dump of the hung harness (the native Monitor_Wait finally resolved) show a circular singleton dependency hidden entirely in DI factory lambdas:

ISecretStore (ReplicatingSecretStore decorator)
  -> ISecretReplicator
    -> SecretReplicationActorProvider
      -> ISecretCacheInvalidator
        -> DefaultSecretResolver
          -> ISecretStore        <- cycle closes (resolver reads through the store)

MS.DI cannot detect cycles through opaque sp => ... factories. At runtime the resolution recurses around the loop (same-thread Monitor re-entry keeps it alive), until MS.DI's StackGuard moves the recursion onto a fresh thread-pool thread to avoid a stack overflow — and that thread then blocks forever in Monitor.Enter on a singleton call-site lock the first thread still holds (which is itself parked in WaitOne waiting for the offloaded task). A silent, permanent, cross-thread deadlock with no exception.

Both candidate causes are disproven: the actor's constructor is never reached, so DistributedPubSub.Get and the composed HOCON are irrelevant — and the new regression test reproduces the hang on a single non-clustered node.

Confirming experiment: replacing only the eager sp.GetService<ISecretCacheInvalidator>() with null eliminated the hang and the 2-node gate ran — at 5/6, with deleted secrets still resolving on the peer. So the invalidator edge is functionally required; it must be deferred, not severed.

Fix: DeferredSecretCacheInvalidator — the provider now receives a wrapper that resolves the real invalidator on first eviction, which can only happen when a replicated row is applied, strictly after graph resolution completes. (Deferring merely to actor-creation time is NOT sufficient — that still happens inside the same resolution and deadlocks identically.) The SQL-Server package never had the cycle (its replicator chain never touches the invalidator), which is exactly why the hub gate passed 6/6 while Akka hung.

Definition of done, item by item:

  • Root cause identified from managed stack evidence, not speculation. ✅
  • Regression test creating the actor through DI in a hosted process: HostedProcessResolutionTests (container-registered ActorSystem, hosted services started, watchdogged resolves of ISecretStore/ISecretReplicator/ISecretResolver). Verified red on 0.2.1 (fails at 20s watchdog), green on 0.2.2. ✅
  • Every existing test stays green: full suite 180 passed / 0 failed / 15 skipped (env-gated live SQL). ✅
  • Live convergence gate re-run: AKKA 6/6 — including the delete-visibility check that proves the deferred invalidator actually evicts. ✅

Also pinned: DeferredSecretCacheInvalidatorTests (no eager resolve, evictions forward, lookup runs once) — this library's fourth defect visible only with the DI graph built in a real host, so the pass-through itself gets guarded against going silently inert.

Remaining before lmxopcua#482 can proceed: publish 0.2.2 to the Gitea feed, then bump OtOpcUa and re-run its gate.

(One unrelated observation: ActorContextAfterAwaitTests.Self_and_Context_are_both_unreadable_after_a_real_thread_hop flaked once under the fuller parallel suite — its assertion races on which thread-pool thread a ConfigureAwait(false) continuation lands. Passed alone and on re-run; pre-existing sensitivity, not touched by this fix.)

## Root cause identified and fixed — `main` @ `a1df7ef` (lib `0.2.2`, not yet published to the feed) **The deadlock was never Akka.** Managed stacks from a `dotnet-dump` of the hung harness (the native `Monitor_Wait` finally resolved) show a **circular singleton dependency hidden entirely in DI factory lambdas**: ``` ISecretStore (ReplicatingSecretStore decorator) -> ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator -> DefaultSecretResolver -> ISecretStore <- cycle closes (resolver reads through the store) ``` MS.DI cannot detect cycles through opaque `sp => ...` factories. At runtime the resolution recurses around the loop (same-thread `Monitor` re-entry keeps it alive), until MS.DI's `StackGuard` moves the recursion onto a fresh thread-pool thread to avoid a stack overflow — and that thread then blocks forever in `Monitor.Enter` on a singleton call-site lock the first thread still holds (which is itself parked in `WaitOne` waiting for the offloaded task). A silent, permanent, cross-thread deadlock with no exception. **Both candidate causes are disproven:** the actor's constructor is never reached, so `DistributedPubSub.Get` and the composed HOCON are irrelevant — and the new regression test reproduces the hang on a **single non-clustered node**. **Confirming experiment:** replacing only the eager `sp.GetService<ISecretCacheInvalidator>()` with `null` eliminated the hang and the 2-node gate ran — at **5/6**, with *deleted secrets still resolving on the peer*. So the invalidator edge is functionally required; it must be deferred, not severed. **Fix:** `DeferredSecretCacheInvalidator` — the provider now receives a wrapper that resolves the real invalidator on **first eviction**, which can only happen when a replicated row is applied, strictly after graph resolution completes. (Deferring merely to actor-creation time is NOT sufficient — that still happens inside the same resolution and deadlocks identically.) The SQL-Server package never had the cycle (its replicator chain never touches the invalidator), which is exactly why the hub gate passed 6/6 while Akka hung. **Definition of done, item by item:** - Root cause identified from managed stack evidence, not speculation. ✅ - Regression test creating the actor **through DI in a hosted process**: `HostedProcessResolutionTests` (container-registered `ActorSystem`, hosted services started, watchdogged resolves of `ISecretStore`/`ISecretReplicator`/`ISecretResolver`). Verified red on 0.2.1 (fails at 20s watchdog), green on 0.2.2. ✅ - Every existing test stays green: full suite **180 passed / 0 failed / 15 skipped** (env-gated live SQL). ✅ - **Live convergence gate re-run: AKKA 6/6** — including the delete-visibility check that proves the deferred invalidator actually evicts. ✅ Also pinned: `DeferredSecretCacheInvalidatorTests` (no eager resolve, evictions forward, lookup runs once) — this library's fourth defect visible only with the DI graph built in a real host, so the pass-through itself gets guarded against going silently inert. Remaining before `lmxopcua#482` can proceed: publish `0.2.2` to the Gitea feed, then bump OtOpcUa and re-run its gate. (One unrelated observation: `ActorContextAfterAwaitTests.Self_and_Context_are_both_unreadable_after_a_real_thread_hop` flaked once under the fuller parallel suite — its assertion races on which thread-pool thread a `ConfigureAwait(false)` continuation lands. Passed alone and on re-run; pre-existing sensitivity, not touched by this fix.)
dohertj2 commented 2026-07-18 14:55:39 -04:00
Author
Owner
Copy Link
Copy Source

0.2.2 is PUBLISHED to the Gitea feed (all 5 packages; restore-verified from a scratch consumer — 5 pinned refs, clean restore + build). main @ b077ca1.

Also folded in since the fix comment: the ActorContextAfterAwaitTests flake observed during verification is fixed (51bef63) — its old formulation asserted that a pool-thread continuation always throws NotSupportedException, which Akka does not guarantee (pool threads legitimately carry the ThreadStatic cell during mailbox runs/async continuations, and a cleared cell makes Context return null / Self return _clearedSelf without throwing). The illegal reads now run on a dedicated fresh thread, where the no-context state is guaranteed; 11 consecutive full-project runs green.

Every item in the definition of done is met and the package is on the feed, so closing. Consumer-side adoption continues in lmxopcua#482 (bump to 0.2.2, re-run the OtOpcUa gate, then consider enabling Secrets:Replication:Enabled).

**`0.2.2` is PUBLISHED to the Gitea feed** (all 5 packages; restore-verified from a scratch consumer — 5 pinned refs, clean restore + build). `main` @ `b077ca1`. Also folded in since the fix comment: the `ActorContextAfterAwaitTests` flake observed during verification is fixed (`51bef63`) — its old formulation asserted that a pool-thread continuation *always* throws `NotSupportedException`, which Akka does not guarantee (pool threads legitimately carry the ThreadStatic cell during mailbox runs/async continuations, and a cleared cell makes `Context` return null / `Self` return `_clearedSelf` without throwing). The illegal reads now run on a dedicated fresh thread, where the no-context state *is* guaranteed; 11 consecutive full-project runs green. Every item in the definition of done is met and the package is on the feed, so closing. Consumer-side adoption continues in lmxopcua#482 (bump to `0.2.2`, re-run the OtOpcUa gate, then consider enabling `Secrets:Replication:Enabled`).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/scadaproj#1
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?