fix(secrets): anti-entropy never ran — Self read after await in the replication actor

I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.

The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.

Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.

170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-18 04:42:16 -04:00
parent dd0a846b64
commit 15ef1f32a8
6 changed files with 239 additions and 11 deletions
@@ -143,9 +143,11 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
}
IActorRef peer = Sender;
// Captured here, on the actor thread, because the exchange awaits and cannot read Self after.
IActorRef self = Self;
IReadOnlyList<SecretManifestEntry> remote = [.. message.Entries.Select(e => e.ToEntry())];
ReconcileWithPeerAsync(peer, remote).PipeTo(Self, failure: ex => new ExchangeFailed(ex));
ReconcileWithPeerAsync(peer, self, remote).PipeTo(self, failure: ex => new ExchangeFailed(ex));
}
private void HandlePullRequest(SecretPullRequest message)
@@ -210,8 +212,19 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
// Both halves of an anti-entropy exchange, computed from one manifest: push what the peer is
// behind on, ask for what we are behind on.
//
// `self` is a PARAMETER, captured by the caller while the actor context is still current. It
// must not be read as the `Self` property below: `Self` resolves through Context, which is
// [ThreadStatic], and every `await` here can resume on a thread-pool thread where that context
// is gone — reading it there throws NotSupportedException and faults the whole exchange.
//
// This was a live bug. It hid because a local SQLite store usually completes `await`
// synchronously, keeping the continuation on the mailbox thread and the context intact, so the
// entire cluster suite passed while anti-entropy would have broken under any slower or more
// contended store. Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous forces the
// async path and fails if this regresses.
private async Task<ExchangeDone> ReconcileWithPeerAsync(
IActorRef peer, IReadOnlyList<SecretManifestEntry> remote)
IActorRef peer, IActorRef self, IReadOnlyList<SecretManifestEntry> remote)
{
IReadOnlyList<SecretManifestEntry> localManifest =
await _store.GetManifestAsync(CancellationToken.None).ConfigureAwait(false);
@@ -226,7 +239,7 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
if (rows.Count > 0)
{
peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), Self);
peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), self);
}
}
@@ -235,7 +248,7 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
if (pull.Count > 0)
{
peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), Self);
peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), self);
}
return ExchangeDone.Instance;