fix(secrets): anti-entropy never ran — Self read after await in the replication actor
I dismissed this finding from the code review as a false positive, reasoning that Akka's ActorBase caches Self in a field and that three passing anti-entropy tests traverse the path. Both premises were wrong. Self resolves through Context, which is [ThreadStatic], and throws NotSupportedException once a continuation resumes on a thread-pool thread. The tests passed because a local SQLite store usually completes await SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context was still intact. Correctness therefore depended on store latency and thread-pool timing: green here, broken under a slower or contended store, with the only symptom a per-peer warning every announce interval while nodes silently stopped converging. The live-broadcast fast path masked it further — only the anti-entropy repair path was dead. Captures self on the actor thread and passes it in. Adds GenuinelyAsyncSecretStore to force the async path, a regression test that fails on the unfixed code (20s timeout) and passes in 3s after, and ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong assumption cannot be made again. Audited every remaining Self/Sender access in the actor. 170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -148,7 +148,7 @@ each project's **code-verified current state**, and the **gaps** between. See
|
||||
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
|
||||
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
|
||||
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
|
||||
| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 164 pass / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. **`0.2.0` is packed + vulnerability-scanned clean but NOT yet published to the feed**, and no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
|
||||
| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. **`0.2.0` is packed + vulnerability-scanned clean but NOT yet published to the feed**, and no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
|
||||
| LocalDb (embedded cache + 2-node sync) | Built (3 pkgs `0.1.0`, **published to the Gitea feed** 2026-07-18, restore-verified from a scratch consumer; no app adoption yet) | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |
|
||||
|
||||
The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
|
||||
|
||||
+17
-4
@@ -143,9 +143,11 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
|
||||
}
|
||||
|
||||
IActorRef peer = Sender;
|
||||
// Captured here, on the actor thread, because the exchange awaits and cannot read Self after.
|
||||
IActorRef self = Self;
|
||||
IReadOnlyList<SecretManifestEntry> remote = [.. message.Entries.Select(e => e.ToEntry())];
|
||||
|
||||
ReconcileWithPeerAsync(peer, remote).PipeTo(Self, failure: ex => new ExchangeFailed(ex));
|
||||
ReconcileWithPeerAsync(peer, self, remote).PipeTo(self, failure: ex => new ExchangeFailed(ex));
|
||||
}
|
||||
|
||||
private void HandlePullRequest(SecretPullRequest message)
|
||||
@@ -210,8 +212,19 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
|
||||
|
||||
// Both halves of an anti-entropy exchange, computed from one manifest: push what the peer is
|
||||
// behind on, ask for what we are behind on.
|
||||
//
|
||||
// `self` is a PARAMETER, captured by the caller while the actor context is still current. It
|
||||
// must not be read as the `Self` property below: `Self` resolves through Context, which is
|
||||
// [ThreadStatic], and every `await` here can resume on a thread-pool thread where that context
|
||||
// is gone — reading it there throws NotSupportedException and faults the whole exchange.
|
||||
//
|
||||
// This was a live bug. It hid because a local SQLite store usually completes `await`
|
||||
// synchronously, keeping the continuation on the mailbox thread and the context intact, so the
|
||||
// entire cluster suite passed while anti-entropy would have broken under any slower or more
|
||||
// contended store. Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous forces the
|
||||
// async path and fails if this regresses.
|
||||
private async Task<ExchangeDone> ReconcileWithPeerAsync(
|
||||
IActorRef peer, IReadOnlyList<SecretManifestEntry> remote)
|
||||
IActorRef peer, IActorRef self, IReadOnlyList<SecretManifestEntry> remote)
|
||||
{
|
||||
IReadOnlyList<SecretManifestEntry> localManifest =
|
||||
await _store.GetManifestAsync(CancellationToken.None).ConfigureAwait(false);
|
||||
@@ -226,7 +239,7 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
|
||||
|
||||
if (rows.Count > 0)
|
||||
{
|
||||
peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), Self);
|
||||
peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), self);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -235,7 +248,7 @@ public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
|
||||
|
||||
if (pull.Count > 0)
|
||||
{
|
||||
peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), Self);
|
||||
peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), self);
|
||||
}
|
||||
|
||||
return ExchangeDone.Instance;
|
||||
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the Akka behaviour that <see cref="SecretReplicationActor"/> depends on: after a genuine
|
||||
/// thread hop, actor context is <b>gone</b>, so <c>Self</c> cannot be read.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is here because assuming otherwise produced a real bug. <c>Self</c> resolves through
|
||||
/// <c>Context</c>, which is <c>[ThreadStatic]</c>; a continuation that resumes on a thread-pool
|
||||
/// thread has none, and reading it throws <see cref="NotSupportedException"/>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// What made it dangerous is that it does <em>not</em> reliably throw. When the awaited operation
|
||||
/// completes synchronously — which a local SQLite store usually does — the continuation stays on the
|
||||
/// actor's mailbox thread and the context is still intact, so the illegal read succeeds. Correctness
|
||||
/// then depends on store latency and thread-pool timing. The rule the production code follows is
|
||||
/// therefore unconditional: capture <c>Self</c> (and <c>Sender</c>) into locals while still on the
|
||||
/// actor thread, and never touch actor context after an <c>await</c>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ActorContextAfterAwaitTests : TestKit
|
||||
{
|
||||
private sealed record Probe;
|
||||
|
||||
private sealed record Outcome(string? SelfError, string? ContextError, string? CapturedSelfPath);
|
||||
|
||||
private sealed class ProbeActor : ReceiveActor
|
||||
{
|
||||
public ProbeActor()
|
||||
{
|
||||
ReceiveAsync<Probe>(async probe =>
|
||||
{
|
||||
IActorRef replyTo = Sender;
|
||||
|
||||
// The correct pattern: capture while the context is still current.
|
||||
IActorRef capturedSelf = Self;
|
||||
|
||||
// Force a real hop off the mailbox thread.
|
||||
await Task.Delay(50).ConfigureAwait(false);
|
||||
await Task.Yield();
|
||||
|
||||
string? selfError = null;
|
||||
string? contextError = null;
|
||||
|
||||
try
|
||||
{
|
||||
ActorPath path = Self.Path;
|
||||
GC.KeepAlive(path);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
selfError = ex.GetType().Name;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
IActorRef fromContext = Context.Self;
|
||||
GC.KeepAlive(fromContext);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
contextError = ex.GetType().Name;
|
||||
}
|
||||
|
||||
replyTo.Tell(new Outcome(selfError, contextError, capturedSelf.Path.ToString()));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Self_and_Context_are_both_unreadable_after_a_real_thread_hop()
|
||||
{
|
||||
IActorRef probe = Sys.ActorOf(Props.Create(() => new ProbeActor()), "context-probe");
|
||||
probe.Tell(new Probe());
|
||||
|
||||
Outcome outcome = ExpectMsg<Outcome>(TimeSpan.FromSeconds(10));
|
||||
|
||||
// Both throw — Self is not a cached field, it goes through the ThreadStatic context.
|
||||
Assert.Equal("NotSupportedException", outcome.SelfError);
|
||||
Assert.Equal("NotSupportedException", outcome.ContextError);
|
||||
|
||||
// ...but a reference captured beforehand stays valid, which is what the actor relies on.
|
||||
Assert.EndsWith("/user/context-probe", outcome.CapturedSelfPath, StringComparison.Ordinal);
|
||||
}
|
||||
}
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.Fakes;
|
||||
|
||||
/// <summary>
|
||||
/// Wraps a store so every operation genuinely yields the thread before completing.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This exists to close a blind spot that let a real bug through. A local SQLite store is fast
|
||||
/// enough that <c>await</c> on it frequently completes <b>synchronously</b>, which leaves the
|
||||
/// caller on the actor's own mailbox thread — and therefore leaves the actor context (a
|
||||
/// <c>[ThreadStatic]</c>) intact. Code that illegally touches actor context after an <c>await</c>
|
||||
/// then works by accident, and the whole suite passes.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Under a slower store, a contended one, or simply a differently-timed thread pool, the same
|
||||
/// <c>await</c> goes properly asynchronous, the continuation lands on a pool thread with no actor
|
||||
/// context, and the code throws. That is the worst shape of bug: green in tests, broken in
|
||||
/// production, timing-dependent either way. Forcing the async path makes it deterministic.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="inner">The store to wrap.</param>
|
||||
public sealed class GenuinelyAsyncSecretStore(ISecretStore inner) : ISecretStore
|
||||
{
|
||||
private static async Task YieldRealAsync()
|
||||
{
|
||||
// Task.Yield alone can still be scheduled inline by some schedulers; a real (if tiny) delay
|
||||
// guarantees the continuation is posted to the thread pool.
|
||||
await Task.Delay(5).ConfigureAwait(false);
|
||||
await Task.Yield();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct)
|
||||
{
|
||||
await YieldRealAsync().ConfigureAwait(false);
|
||||
return await inner.GetAsync(name, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task UpsertAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
await YieldRealAsync().ConfigureAwait(false);
|
||||
await inner.UpsertAsync(row, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct)
|
||||
{
|
||||
await YieldRealAsync().ConfigureAwait(false);
|
||||
return await inner.DeleteAsync(name, actor, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct)
|
||||
{
|
||||
await YieldRealAsync().ConfigureAwait(false);
|
||||
return await inner.ListAsync(includeDeleted, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct)
|
||||
{
|
||||
await YieldRealAsync().ConfigureAwait(false);
|
||||
return await inner.GetManifestAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
await YieldRealAsync().ConfigureAwait(false);
|
||||
await inner.ApplyReplicatedAsync(row, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
|
||||
{
|
||||
await YieldRealAsync().ConfigureAwait(false);
|
||||
return await inner.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek, ct)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
+29
@@ -4,6 +4,7 @@ using Akka.Configuration;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.Fakes;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests;
|
||||
@@ -263,6 +264,34 @@ public sealed class TwoNodeClusterReplicationTests : IAsyncLifetime
|
||||
"Node A never learned about a secret that originated on node B.");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous()
|
||||
{
|
||||
// Regression guard for a bug the rest of this suite could not see. A local SQLite store
|
||||
// usually completes `await` synchronously, so continuations stayed on the actor's mailbox
|
||||
// thread and code that touched actor context after an await worked by accident. Wrapping the
|
||||
// store so every call really yields makes that illegal access deterministic — this test
|
||||
// fails outright if anti-entropy touches actor context across an await.
|
||||
SqliteSecretStore backing = CreateStore();
|
||||
var slowStore = new GenuinelyAsyncSecretStore(backing);
|
||||
|
||||
_systemB.ActorOf(
|
||||
SecretReplicationActor.Props(slowStore, null, AnnounceInterval), "slow-node");
|
||||
|
||||
await _writableA.UpsertAsync(Row("slow/inbound", 0x42), CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(slowStore, new SecretName("slow/inbound"), r => r.Ciphertext[0] == 0x42),
|
||||
"A node whose store is genuinely async never received the secret.");
|
||||
|
||||
// And the other direction, which can ONLY converge through the anti-entropy exchange.
|
||||
await backing.UpsertAsync(Row("slow/outbound", 0x43), CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(_storeA, new SecretName("slow/outbound"), r => r.Ciphertext[0] == 0x43),
|
||||
"Node A never learned about a secret held by a node whose store is genuinely async.");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_node_that_joins_late_catches_up_on_everything()
|
||||
{
|
||||
|
||||
@@ -41,7 +41,7 @@ construction).
|
||||
the UI, the CLI, and any future write path at once.
|
||||
- Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`);
|
||||
`AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace.
|
||||
- Library suite: **182 passing, 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
|
||||
- Library suite: **170 passing offline (184 with the live SQL suite enabled), 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
|
||||
against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural
|
||||
divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster**
|
||||
(real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions,
|
||||
@@ -56,7 +56,7 @@ construction).
|
||||
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
|
||||
drift symptom, not the forwarding filter itself (noted in the test).
|
||||
|
||||
### Post-build code review (2026-07-18) — 6 defects found and fixed
|
||||
### Post-build code review (2026-07-18) — 7 defects found and fixed
|
||||
|
||||
An independent review caught real bugs that a green suite had not:
|
||||
|
||||
@@ -76,10 +76,23 @@ An independent review caught real bugs that a green suite had not:
|
||||
- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*).
|
||||
- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic.
|
||||
|
||||
One reported "critical" was a **false positive**: `Self` used after `await` in the actor. Akka's
|
||||
`ActorBase` caches `Self` in an instance field (unlike `Context`, which is `[ThreadStatic]`), and
|
||||
three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as
|
||||
written.
|
||||
- **CRITICAL — anti-entropy was broken by `Self` read after `await`.** Initially dismissed as a
|
||||
false positive on the reasoning that `ActorBase` caches `Self` in a field and that three passing
|
||||
anti-entropy tests traverse the path. **Both premises were wrong.** A direct probe
|
||||
(`ActorContextAfterAwaitTests`) shows `Self` resolves through `Context`, which is `[ThreadStatic]`,
|
||||
and throws `NotSupportedException` after a real thread hop. The tests passed only because a local
|
||||
SQLite store usually completes `await` **synchronously**, keeping the continuation on the mailbox
|
||||
thread with the context intact — so correctness silently depended on store latency and thread-pool
|
||||
timing. `GenuinelyAsyncSecretStore` forces the async path;
|
||||
`Anti_entropy_still_works_when_the_store_is_genuinely_asynchronous` fails on the unfixed code
|
||||
(20s timeout: "Node A never learned about a secret held by a node whose store is genuinely async")
|
||||
and passes in 3s after capturing `self` on the actor thread. Every remaining `Self`/`Sender` access
|
||||
in the actor was then audited.
|
||||
|
||||
**Process note worth keeping.** Seven of seven review findings were real. The one dismissal was
|
||||
wrong, and it was dismissed using *indirect* evidence (a green suite) over a *direct* mechanism
|
||||
claim. Green tests are evidence a path works under the timing the tests happened to produce, not
|
||||
evidence it is correct. When a reviewer names a mechanism, probe the mechanism.
|
||||
|
||||
## Execution status (2026-07-16)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user