fix(site-secrets): keep a live node's mirror-status row from losing a race with the peer's prune

Rig finding on the upgrade restart: both nodes' rows were older than the stale threshold
(the old build never refreshed them), so node-a's reporter pruned node-b's row while node-b
wrote a fresh one, and the replicated delete won. The reporter now prunes nothing during its
first refresh interval after start-up, and each mirror pass re-writes its own row if it no
longer holds what the mirror last wrote.
This commit is contained in:
Joseph Doherty
2026-09-29 11:06:26 -04:00
parent 55e1741075
commit 3a217b9484
7 changed files with 100 additions and 13 deletions
@@ -715,6 +715,26 @@ public class DeployedSecretsMirrorTests : IDisposable
Assert.Equal(time.Now, Assert.Single(await storage.GetSecretMirrorStatusesAsync()).CheckedAtUtc);
}
[Fact]
public async Task Status_RowDeletedByAPeerPrune_IsRewrittenOnTheNextPass()
{
// Rig finding: when both nodes restart together, the peer's reporter can prune this
// node's old row concurrently with this node's fresh write, and the replicated delete
// wins. The mirror notices its row is gone and writes it again.
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
var (mirror, storage, _, _, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b"));
await mirror.ReconcileAsync(CancellationToken.None);
var written = Assert.Single(await storage.GetSecretMirrorStatusesAsync()).CheckedAtUtc;
Assert.True(await storage.DeleteSecretMirrorStatusIfUnchangedAsync("node-b", written));
time.Now = time.Now.AddSeconds(15);
await mirror.ReconcileAsync(CancellationToken.None);
var status = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
Assert.Equal("node-b", status.NodeName);
Assert.Equal(time.Now, status.CheckedAtUtc);
}
[Fact]
public async Task Status_FailureSetChange_RewritesTheRow_AndClearsWhenHealed()
{