fix(secrets): anti-entropy never ran — Self read after await in the replication actor

I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.

The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.

Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.

170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-18 04:42:16 -04:00
parent dd0a846b64
commit 15ef1f32a8
6 changed files with 239 additions and 11 deletions
+19 -6
View File
@@ -41,7 +41,7 @@ construction).
the UI, the CLI, and any future write path at once.
- Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`);
`AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace.
- Library suite: **182 passing, 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
- Library suite: **170 passing offline (184 with the live SQL suite enabled), 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural
divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster**
(real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions,
@@ -56,7 +56,7 @@ construction).
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
drift symptom, not the forwarding filter itself (noted in the test).
### Post-build code review (2026-07-18) — 6 defects found and fixed
### Post-build code review (2026-07-18) — 7 defects found and fixed
An independent review caught real bugs that a green suite had not:
@@ -76,10 +76,23 @@ An independent review caught real bugs that a green suite had not:
- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*).
- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic.
One reported "critical" was a **false positive**: `Self` used after `await` in the actor. Akka's
`ActorBase` caches `Self` in an instance field (unlike `Context`, which is `[ThreadStatic]`), and
three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as
written.
- **CRITICAL — anti-entropy was broken by `Self` read after `await`.** Initially dismissed as a
false positive on the reasoning that `ActorBase` caches `Self` in a field and that three passing
anti-entropy tests traverse the path. **Both premises were wrong.** A direct probe
(`ActorContextAfterAwaitTests`) shows `Self` resolves through `Context`, which is `[ThreadStatic]`,
and throws `NotSupportedException` after a real thread hop. The tests passed only because a local
SQLite store usually completes `await` **synchronously**, keeping the continuation on the mailbox
thread with the context intact — so correctness silently depended on store latency and thread-pool
timing. `GenuinelyAsyncSecretStore` forces the async path;
`Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous` fails on the unfixed code
(20s timeout: "Node A never learned about a secret held by a node whose store is genuinely async")
and passes in 3s after capturing `self` on the actor thread. Every remaining `Self`/`Sender` access
in the actor was then audited.
**Process note worth keeping.** Seven of seven review findings were real. The one dismissal was
wrong, and it was dismissed using *indirect* evidence (a green suite) over a *direct* mechanism
claim. Green tests are evidence a path works under the timing the tests happened to produce, not
evidence it is correct. When a reviewer names a mechanism, probe the mechanism.
## Execution status (2026-07-16)