diff --git a/CLAUDE.md b/CLAUDE.md index 29efeed..1dd4776 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -148,7 +148,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | -| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 164 pass / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. **`0.2.0` is packed + vulnerability-scanned clean but NOT yet published to the feed**, and no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | +| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. **`0.2.0` is packed + vulnerability-scanned clean but NOT yet published to the feed**, and no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | | LocalDb (embedded cache + 2-node sync) | Built (3 pkgs `0.1.0`, **published to the Gitea feed** 2026-07-18, restore-verified from a scratch consumer; no app adoption yet) | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationActor.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationActor.cs index 8e571cc..a281a6a 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationActor.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationActor.cs @@ -143,9 +143,11 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers } IActorRef peer = Sender; + // Captured here, on the actor thread, because the exchange awaits and cannot read Self after. + IActorRef self = Self; IReadOnlyList remote = [.. message.Entries.Select(e => e.ToEntry())]; - ReconcileWithPeerAsync(peer, remote).PipeTo(Self, failure: ex => new ExchangeFailed(ex)); + ReconcileWithPeerAsync(peer, self, remote).PipeTo(self, failure: ex => new ExchangeFailed(ex)); } private void HandlePullRequest(SecretPullRequest message) @@ -210,8 +212,19 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers // Both halves of an anti-entropy exchange, computed from one manifest: push what the peer is // behind on, ask for what we are behind on. + // + // `self` is a PARAMETER, captured by the caller while the actor context is still current. It + // must not be read as the `Self` property below: `Self` resolves through Context, which is + // [ThreadStatic], and every `await` here can resume on a thread-pool thread where that context + // is gone — reading it there throws NotSupportedException and faults the whole exchange. + // + // This was a live bug. It hid because a local SQLite store usually completes `await` + // synchronously, keeping the continuation on the mailbox thread and the context intact, so the + // entire cluster suite passed while anti-entropy would have broken under any slower or more + // contended store. Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous forces the + // async path and fails if this regresses. private async Task ReconcileWithPeerAsync( - IActorRef peer, IReadOnlyList remote) + IActorRef peer, IActorRef self, IReadOnlyList remote) { IReadOnlyList localManifest = await _store.GetManifestAsync(CancellationToken.None).ConfigureAwait(false); @@ -226,7 +239,7 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers if (rows.Count > 0) { - peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), Self); + peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), self); } } @@ -235,7 +248,7 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers if (pull.Count > 0) { - peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), Self); + peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), self); } return ExchangeDone.Instance; diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/ActorContextAfterAwaitTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/ActorContextAfterAwaitTests.cs new file mode 100644 index 0000000..0327e7a --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/ActorContextAfterAwaitTests.cs @@ -0,0 +1,89 @@ +using Akka.Actor; +using Akka.TestKit.Xunit2; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests; + +/// +/// Pins the Akka behaviour that depends on: after a genuine +/// thread hop, actor context is gone, so Self cannot be read. +/// +/// +/// +/// This is here because assuming otherwise produced a real bug. Self resolves through +/// Context, which is [ThreadStatic]; a continuation that resumes on a thread-pool +/// thread has none, and reading it throws . +/// +/// +/// What made it dangerous is that it does not reliably throw. When the awaited operation +/// completes synchronously — which a local SQLite store usually does — the continuation stays on the +/// actor's mailbox thread and the context is still intact, so the illegal read succeeds. Correctness +/// then depends on store latency and thread-pool timing. The rule the production code follows is +/// therefore unconditional: capture Self (and Sender) into locals while still on the +/// actor thread, and never touch actor context after an await. +/// +/// +public sealed class ActorContextAfterAwaitTests : TestKit +{ + private sealed record Probe; + + private sealed record Outcome(string? SelfError, string? ContextError, string? CapturedSelfPath); + + private sealed class ProbeActor : ReceiveActor + { + public ProbeActor() + { + ReceiveAsync(async probe => + { + IActorRef replyTo = Sender; + + // The correct pattern: capture while the context is still current. + IActorRef capturedSelf = Self; + + // Force a real hop off the mailbox thread. + await Task.Delay(50).ConfigureAwait(false); + await Task.Yield(); + + string? selfError = null; + string? contextError = null; + + try + { + ActorPath path = Self.Path; + GC.KeepAlive(path); + } + catch (Exception ex) + { + selfError = ex.GetType().Name; + } + + try + { + IActorRef fromContext = Context.Self; + GC.KeepAlive(fromContext); + } + catch (Exception ex) + { + contextError = ex.GetType().Name; + } + + replyTo.Tell(new Outcome(selfError, contextError, capturedSelf.Path.ToString())); + }); + } + } + + [Fact] + public void Self_and_Context_are_both_unreadable_after_a_real_thread_hop() + { + IActorRef probe = Sys.ActorOf(Props.Create(() => new ProbeActor()), "context-probe"); + probe.Tell(new Probe()); + + Outcome outcome = ExpectMsg(TimeSpan.FromSeconds(10)); + + // Both throw — Self is not a cached field, it goes through the ThreadStatic context. + Assert.Equal("NotSupportedException", outcome.SelfError); + Assert.Equal("NotSupportedException", outcome.ContextError); + + // ...but a reference captured beforehand stays valid, which is what the actor relies on. + Assert.EndsWith("/user/context-probe", outcome.CapturedSelfPath, StringComparison.Ordinal); + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/Fakes/GenuinelyAsyncSecretStore.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/Fakes/GenuinelyAsyncSecretStore.cs new file mode 100644 index 0000000..6587b72 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/Fakes/GenuinelyAsyncSecretStore.cs @@ -0,0 +1,84 @@ +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.Fakes; + +/// +/// Wraps a store so every operation genuinely yields the thread before completing. +/// +/// +/// +/// This exists to close a blind spot that let a real bug through. A local SQLite store is fast +/// enough that await on it frequently completes synchronously, which leaves the +/// caller on the actor's own mailbox thread — and therefore leaves the actor context (a +/// [ThreadStatic]) intact. Code that illegally touches actor context after an await +/// then works by accident, and the whole suite passes. +/// +/// +/// Under a slower store, a contended one, or simply a differently-timed thread pool, the same +/// await goes properly asynchronous, the continuation lands on a pool thread with no actor +/// context, and the code throws. That is the worst shape of bug: green in tests, broken in +/// production, timing-dependent either way. Forcing the async path makes it deterministic. +/// +/// +/// The store to wrap. +public sealed class GenuinelyAsyncSecretStore(ISecretStore inner) : ISecretStore +{ + private static async Task YieldRealAsync() + { + // Task.Yield alone can still be scheduled inline by some schedulers; a real (if tiny) delay + // guarantees the continuation is posted to the thread pool. + await Task.Delay(5).ConfigureAwait(false); + await Task.Yield(); + } + + /// + public async Task GetAsync(SecretName name, CancellationToken ct) + { + await YieldRealAsync().ConfigureAwait(false); + return await inner.GetAsync(name, ct).ConfigureAwait(false); + } + + /// + public async Task UpsertAsync(StoredSecret row, CancellationToken ct) + { + await YieldRealAsync().ConfigureAwait(false); + await inner.UpsertAsync(row, ct).ConfigureAwait(false); + } + + /// + public async Task DeleteAsync(SecretName name, string? actor, CancellationToken ct) + { + await YieldRealAsync().ConfigureAwait(false); + return await inner.DeleteAsync(name, actor, ct).ConfigureAwait(false); + } + + /// + public async Task> ListAsync(bool includeDeleted, CancellationToken ct) + { + await YieldRealAsync().ConfigureAwait(false); + return await inner.ListAsync(includeDeleted, ct).ConfigureAwait(false); + } + + /// + public async Task> GetManifestAsync(CancellationToken ct) + { + await YieldRealAsync().ConfigureAwait(false); + return await inner.GetManifestAsync(ct).ConfigureAwait(false); + } + + /// + public async Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) + { + await YieldRealAsync().ConfigureAwait(false); + await inner.ApplyReplicatedAsync(row, ct).ConfigureAwait(false); + } + + /// + public async Task ApplyRewrapAsync( + StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) + { + await YieldRealAsync().ConfigureAwait(false); + return await inner.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek, ct) + .ConfigureAwait(false); + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/TwoNodeClusterReplicationTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/TwoNodeClusterReplicationTests.cs index 30e7da7..68aeabd 100644 --- a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/TwoNodeClusterReplicationTests.cs +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/TwoNodeClusterReplicationTests.cs @@ -4,6 +4,7 @@ using Akka.Configuration; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.Secrets.Abstractions; using ZB.MOM.WW.Secrets.Replication; +using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.Fakes; using ZB.MOM.WW.Secrets.Sqlite; namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests; @@ -263,6 +264,34 @@ public sealed class TwoNodeClusterReplicationTests : IAsyncLifetime "Node A never learned about a secret that originated on node B."); } + [Fact] + public async Task Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous() + { + // Regression guard for a bug the rest of this suite could not see. A local SQLite store + // usually completes `await` synchronously, so continuations stayed on the actor's mailbox + // thread and code that touched actor context after an await worked by accident. Wrapping the + // store so every call really yields makes that illegal access deterministic — this test + // fails outright if anti-entropy touches actor context across an await. + SqliteSecretStore backing = CreateStore(); + var slowStore = new GenuinelyAsyncSecretStore(backing); + + _systemB.ActorOf( + SecretReplicationActor.Props(slowStore, null, AnnounceInterval), "slow-node"); + + await _writableA.UpsertAsync(Row("slow/inbound", 0x42), CancellationToken.None); + + Assert.True( + await WaitForAsync(slowStore, new SecretName("slow/inbound"), r => r.Ciphertext[0] == 0x42), + "A node whose store is genuinely async never received the secret."); + + // And the other direction, which can ONLY converge through the anti-entropy exchange. + await backing.UpsertAsync(Row("slow/outbound", 0x43), CancellationToken.None); + + Assert.True( + await WaitForAsync(_storeA, new SecretName("slow/outbound"), r => r.Ciphertext[0] == 0x43), + "Node A never learned about a secret held by a node whose store is genuinely async."); + } + [Fact] public async Task A_node_that_joins_late_catches_up_on_everything() { diff --git a/components/secrets/GAPS.md b/components/secrets/GAPS.md index 69e5ae9..40cbca8 100644 --- a/components/secrets/GAPS.md +++ b/components/secrets/GAPS.md @@ -41,7 +41,7 @@ construction). the UI, the CLI, and any future write path at once. - Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`); `AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace. -- Library suite: **182 passing, 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests +- Library suite: **170 passing offline (184 with the live SQL suite enabled), 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster** (real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions, @@ -56,7 +56,7 @@ construction). - ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the drift symptom, not the forwarding filter itself (noted in the test). -### Post-build code review (2026-07-18) — 6 defects found and fixed +### Post-build code review (2026-07-18) — 7 defects found and fixed An independent review caught real bugs that a green suite had not: @@ -76,10 +76,23 @@ An independent review caught real bugs that a green suite had not: - **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*). - **A failed pull-read replied with silence** — the peer waited forever with no diagnostic. -One reported "critical" was a **false positive**: `Self` used after `await` in the actor. Akka's -`ActorBase` caches `Self` in an instance field (unlike `Context`, which is `[ThreadStatic]`), and -three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as -written. +- **CRITICAL — anti-entropy was broken by `Self` read after `await`.** Initially dismissed as a + false positive on the reasoning that `ActorBase` caches `Self` in a field and that three passing + anti-entropy tests traverse the path. **Both premises were wrong.** A direct probe + (`ActorContextAfterAwaitTests`) shows `Self` resolves through `Context`, which is `[ThreadStatic]`, + and throws `NotSupportedException` after a real thread hop. The tests passed only because a local + SQLite store usually completes `await` **synchronously**, keeping the continuation on the mailbox + thread with the context intact — so correctness silently depended on store latency and thread-pool + timing. `GenuinelyAsyncSecretStore` forces the async path; + `Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous` fails on the unfixed code + (20s timeout: "Node A never learned about a secret held by a node whose store is genuinely async") + and passes in 3s after capturing `self` on the actor thread. Every remaining `Self`/`Sender` access + in the actor was then audited. + +**Process note worth keeping.** Seven of seven review findings were real. The one dismissal was +wrong, and it was dismissed using *indirect* evidence (a green suite) over a *direct* mechanism +claim. Green tests are evidence a path works under the timing the tests happened to produce, not +evidence it is correct. When a reviewer names a mechanism, probe the mechanism. ## Execution status (2026-07-16)