fix(localdb): close both Phase 1 live-gate follow-ups

1. Wiped-node back-fill (live gate check 4). Fixed upstream in ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly off: the oplog cap was not the gate. Snapshot detection measured the peer's gap against the oldest surviving oplog row and read an empty oplog as "no gap possible" — and an empty oplog is the steady state of a converged pair, since ack-pruning deletes everything the peer confirmed. The healthy state was the one state that could not heal a wiped peer. Now measured against last_acked_seq when the oplog is empty. Pinned at this level too, not just the library's: the pair harness grows a WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node id and a zero watermark), and the new scenario asserts the emptied oplog as its precondition before wiping, then requires the deployment artifact to come back byte-identical with no new deploy. Verified RED against the pinned 0.1.1 (times out waiting for back-fill) and green on 0.1.2. 2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source: StoreAsync now skips entirely when the pointer already names this deployment/revision, the SHA matches the bytes, and the expected chunk count is present. Re-caching an artifact the node already holds — every restart's boot-from-cache, every RestoreApplied — writes nothing and mints no oplog rows, where before it cost a delete plus an insert per chunk plus a pointer update, all identical to what was already there. Identity is over the bytes and the chunks are counted, both deliberately: a re-composed artifact can carry the same ids with different bytes, and a pointer can name a deployment whose chunks are missing, where skipping would make an unreadable cache permanent. Both guards verified RED against a naive pointer-only skip. The gate's stated bound was also wrong and is corrected in the doc: growth was never headed for the 1M row cap. AddZbLocalDbReplication is registered unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes and the 7-day MaxOplogAge cap prunes. Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a clean master too) and a flaky Roslyn race test (green 2/2 in isolation). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 01:27:02 -04:00
parent c957db52e2
commit 232bff5df7
7 changed files with 295 additions and 13 deletions
@@ -103,6 +103,54 @@ public sealed class LocalDbPairConvergenceTests
            "node B to prune dep1 (with tombstones) while keeping dep3");
    }

+    [Fact]
+    public async Task WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy()
+    {
+        // The Phase 1 live gate's check 4, as an offline scenario. A node whose LocalDb volume is
+        // lost used to come back permanently empty: the pair had already converged, so every oplog
+        // row was acked and pruned, and the healthy node had no delta left to send. Nothing short of
+        // a fresh deploy could repopulate it — which is the worst possible moment to depend on
+        // central being reachable, since the cache exists precisely for when it is not.
+        //
+        // Fixed in ZB.MOM.WW.LocalDb 0.1.2 (snapshot resync now measures the peer's gap against
+        // last_acked_seq when the oplog is empty). Pin the payoff at this level, not just the
+        // library's: what matters here is that the deployment artifact itself comes back.
+        await using var harness = new LocalDbPairHarness();
+        await harness.StartAsync();
+
+        var deploymentId = Guid.NewGuid().ToString("N");
+        var artifact = MultiChunkArtifact(seed: 20);
+        await harness.CacheA.StoreAsync(Cluster, deploymentId, "rev-1", artifact,
+            TestContext.Current.CancellationToken);
+
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () => await ReadArtifactAsync(harness.CacheB, Cluster) is { } b && b.AsSpan().SequenceEqual(artifact),
+            "the artifact to converge before the wipe");
+
+        // Convergence alone is not the precondition — an emptied oplog on A is. That is the state
+        // that used to make back-fill impossible, so assert it rather than assume it.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () => await LocalDbPairHarness.OplogDepthAsync(harness.A) == 0,
+            "node A's oplog to be fully acked and pruned");
+
+        await harness.WipePassiveAsync();
+        Assert.Null(await ReadArtifactAsync(harness.CacheB, Cluster));
+
+        await harness.RestartPairAsync();
+
+        // No new deploy, no central: the rebuilt node gets its cached configuration back from its
+        // peer alone, byte-identical, and can boot from cache again.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () =>
+            {
+                var cached = await harness.CacheB.GetCurrentAsync(Cluster, TestContext.Current.CancellationToken);
+                return cached is not null
+                       && cached.DeploymentId == deploymentId
+                       && cached.Artifact.AsSpan().SequenceEqual(artifact);
+            },
+            "the wiped node to be back-filled from its peer");
+    }
+
    [Fact]
    public async Task WritesWhileTransportDown_SurviveRejoin()
    {