fix(secrets): root-cause + fix the Akka replicator's hosted-process DI deadlock (0.2.2)
Closes the defect in scadaproj#1. The hang was never Akka: the package's DI wiring closed a circular singleton dependency the container cannot see through factory lambdas — ISecretStore (ReplicatingSecretStore decorator) -> ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator -> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then blocks forever on a singleton call-site lock the first thread still holds: a silent permanent hang instead of a stack overflow. Managed stacks from dotnet-dump show the repeating cycle and both parked threads; both candidate causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never reached, and the deadlock reproduces on a single non-clustered node. Fix: defer the one cycle-closing edge. The provider now gets a DeferredSecretCacheInvalidator that resolves the real invalidator on first eviction — which only happens when a replicated row is applied, strictly after graph resolution. Severing the edge instead is wrong: a null-invalidator experiment ran the live gate at 5/6, with deleted secrets still resolving on the peer. The SqlServer package never had the cycle (its replicator chain never touches the invalidator), which is why the hub gate always passed. Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including the delete-visibility check that proves the deferred invalidator really evicts. New HostedProcessResolutionTests builds the graph as a host does (container- registered ActorSystem, hosted services, watchdogged resolves) and fails on 0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full suite 180 passed / 0 failed / 15 skipped (env-gated live SQL). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
+8
-1
@@ -69,10 +69,17 @@ public static class AkkaSecretsServiceCollectionExtensions
|
||||
// The actor is created lazily on first use rather than at registration: the ActorSystem is
|
||||
// typically registered by the application AFTER this call, and a cluster node may also not be
|
||||
// ready to join at the moment the container is built.
|
||||
//
|
||||
// The invalidator MUST stay deferred (scadaproj#1): it is the resolver, the resolver reads
|
||||
// ISecretStore, and ISecretStore is the ReplicatingSecretStore decorator registered below —
|
||||
// resolving it here closes a singleton cycle the container cannot detect through factory
|
||||
// lambdas, and every hosted process then deadlocks at startup inside the container's
|
||||
// resolution locks. DeferredSecretCacheInvalidator waits until the first eviction, which
|
||||
// can only happen after the graph has finished resolving.
|
||||
services.TryAddSingleton<SecretReplicationActorProvider>(sp => new SecretReplicationActorProvider(
|
||||
sp.GetRequiredService<ActorSystem>(),
|
||||
sp.GetRequiredService<SqliteSecretStore>(),
|
||||
sp.GetService<ISecretCacheInvalidator>(),
|
||||
new DeferredSecretCacheInvalidator(sp.GetService<ISecretCacheInvalidator>),
|
||||
sp.GetRequiredService<IOptions<AkkaSecretsReplicationOptions>>().Value));
|
||||
|
||||
// MUST be registered BEFORE AddZbSecrets. That call does
|
||||
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the real <see cref="ISecretCacheInvalidator"/> on first eviction instead of at
|
||||
/// construction, so the replication graph can be wired without closing a dependency cycle.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This deferral is load-bearing, not an optimization (scadaproj#1). The invalidator is the
|
||||
/// resolver, the resolver reads through <c>ISecretStore</c>, and with replication registered
|
||||
/// <c>ISecretStore</c> is the <c>ReplicatingSecretStore</c> decorator whose replicator owns this
|
||||
/// node's actor — so resolving the invalidator eagerly inside the actor provider's factory closes
|
||||
/// a singleton cycle the container cannot see through factory lambdas. Resolution then recurses
|
||||
/// until MS.DI's <c>StackGuard</c> hops it onto another thread, which deadlocks on a call-site
|
||||
/// lock the first thread holds: any host resolving the graph hangs at startup, silently and
|
||||
/// permanently.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Deferring to first eviction is sufficient because evictions only happen when a replicated row
|
||||
/// is applied — strictly after the graph has finished resolving, at which point every singleton on
|
||||
/// the former cycle is already materialized and the lookup completes without re-entering the
|
||||
/// container's resolution locks.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="resolve">
|
||||
/// Lookup for the real invalidator; returns <c>null</c> when the application registered none.
|
||||
/// </param>
|
||||
internal sealed class DeferredSecretCacheInvalidator(Func<ISecretCacheInvalidator?> resolve)
|
||||
: ISecretCacheInvalidator
|
||||
{
|
||||
private readonly Lazy<ISecretCacheInvalidator?> _inner = new(
|
||||
resolve ?? throw new ArgumentNullException(nameof(resolve)),
|
||||
LazyThreadSafetyMode.ExecutionAndPublication);
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Invalidate(SecretName name) => _inner.Value?.Invalidate(name);
|
||||
}
|
||||
Reference in New Issue
Block a user