fix(localdb): close both Phase 1 live-gate follow-ups

1. Wiped-node back-fill (live gate check 4). Fixed upstream in
   ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
   off: the oplog cap was not the gate. Snapshot detection measured the peer's
   gap against the oldest surviving oplog row and read an empty oplog as "no
   gap possible" — and an empty oplog is the steady state of a converged pair,
   since ack-pruning deletes everything the peer confirmed. The healthy state
   was the one state that could not heal a wiped peer. Now measured against
   last_acked_seq when the oplog is empty.

   Pinned at this level too, not just the library's: the pair harness grows a
   WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
   id and a zero watermark), and the new scenario asserts the emptied oplog as
   its precondition before wiping, then requires the deployment artifact to
   come back byte-identical with no new deploy. Verified RED against the pinned
   0.1.1 (times out waiting for back-fill) and green on 0.1.2.

2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
   StoreAsync now skips entirely when the pointer already names this
   deployment/revision, the SHA matches the bytes, and the expected chunk count
   is present. Re-caching an artifact the node already holds — every restart's
   boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
   rows, where before it cost a delete plus an insert per chunk plus a pointer
   update, all identical to what was already there.

   Identity is over the bytes and the chunks are counted, both deliberately: a
   re-composed artifact can carry the same ids with different bytes, and a
   pointer can name a deployment whose chunks are missing, where skipping would
   make an unreadable cache permanent. Both guards verified RED against a naive
   pointer-only skip.

   The gate's stated bound was also wrong and is corrected in the doc: growth
   was never headed for the 1M row cap. AddZbLocalDbReplication is registered
   unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
   and the 7-day MaxOplogAge cap prunes.

Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 01:27:02 -04:00
parent c957db52e2
commit 232bff5df7
7 changed files with 295 additions and 13 deletions
@@ -63,6 +63,14 @@ public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow);
var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize;
if (await IsAlreadyCachedAsync(clusterId, deploymentId, revisionHash, sha, chunkCount, ct))
{
_logger.LogDebug(
"Deployment {DeploymentId} (revision {RevisionHash}) is already cached intact for cluster {ClusterId}; skipping the re-store.",
deploymentId, revisionHash, clusterId);
return;
}
// One transaction for the whole store. A pointer that commits without its chunks — or
// chunks that commit without the pointer — is a cache that reads back as a corrupt hit
// rather than a clean miss, which is the one outcome this type exists to prevent.
@@ -155,6 +163,74 @@ public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
await tx.CommitAsync(ct);
}
/// <summary>
/// Whether this exact artifact is already cached <b>and readable</b> for the cluster, so
/// storing it again would rewrite every row to its current value.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this exists.</b> Every write to a replicated table fires a capture trigger and
/// mints an oplog row. Re-caching an artifact the node already holds — which is what a
/// restart's boot-from-cache and every <c>RestoreApplied</c> recovery does — would
/// therefore cost a delete plus an insert per chunk plus a pointer update, all carrying
/// values identical to what is already there. On a replicating pair that is pure churn;
/// on an unpaired (default-OFF) node there is no peer to ack those rows, so they
/// accumulate in the oplog until the library's age cap prunes them. The Phase 1 live gate
/// saw exactly that (check 8).
/// </para>
/// <para>
/// <b>Identity is over the bytes, not the ids.</b> The pointer must match on
/// <c>artifact_sha256</c> as well as the deployment id and revision hash: a re-composed
/// artifact can legitimately carry the same ids with different bytes, and keeping the
/// stale copy would serve a boot-from-cache node the wrong address space while every
/// id-shaped signal claimed it was right.
/// </para>
/// <para>
/// <b>The chunk count is load-bearing.</b> A pointer can name a deployment whose chunks
/// are missing or partial — a half-replicated artifact, or a prune that raced the
/// pointer. Skipping on the pointer alone would make that state permanent, because the
/// re-store that repairs it is the call being skipped.
/// </para>
/// <para>
/// <b>Accepted consequence:</b> a skipped store leaves <c>applied_at_utc</c> at the
/// moment the artifact was <i>first</i> cached rather than last confirmed. Refreshing it
/// would mint the very oplog row this check exists to avoid. Nothing reads it as a
/// liveness signal; retention ranks distinct deployments, which are unaffected.
/// </para>
/// </remarks>
private async Task<bool> IsAlreadyCachedAsync(
string clusterId, string deploymentId, string revisionHash, string sha, int chunkCount,
CancellationToken ct)
{
var matches = await _db.QueryAsync(
"""
SELECT (
SELECT COUNT(*)
FROM deployment_artifacts
WHERE cluster_id = @ClusterId
AND deployment_id = @DeploymentId
AND chunk_count = @ChunkCount
)
FROM deployment_pointer
WHERE cluster_id = @ClusterId
AND deployment_id = @DeploymentId
AND revision_hash = @RevisionHash
AND artifact_sha256 = @Sha
""",
r => r.GetInt32(0),
new
{
ClusterId = clusterId,
DeploymentId = deploymentId,
RevisionHash = revisionHash,
Sha = sha,
ChunkCount = chunkCount,
},
ct);
return matches.Count == 1 && matches[0] == chunkCount;
}
/// <inheritdoc/>
public async Task<CachedDeploymentArtifact?> GetCurrentAsync(
string clusterId, CancellationToken ct = default)