fix(localdb): close both Phase 1 live-gate follow-ups

1. Wiped-node back-fill (live gate check 4). Fixed upstream in
   ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
   off: the oplog cap was not the gate. Snapshot detection measured the peer's
   gap against the oldest surviving oplog row and read an empty oplog as "no
   gap possible" — and an empty oplog is the steady state of a converged pair,
   since ack-pruning deletes everything the peer confirmed. The healthy state
   was the one state that could not heal a wiped peer. Now measured against
   last_acked_seq when the oplog is empty.

   Pinned at this level too, not just the library's: the pair harness grows a
   WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
   id and a zero watermark), and the new scenario asserts the emptied oplog as
   its precondition before wiping, then requires the deployment artifact to
   come back byte-identical with no new deploy. Verified RED against the pinned
   0.1.1 (times out waiting for back-fill) and green on 0.1.2.

2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
   StoreAsync now skips entirely when the pointer already names this
   deployment/revision, the SHA matches the bytes, and the expected chunk count
   is present. Re-caching an artifact the node already holds — every restart's
   boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
   rows, where before it cost a delete plus an insert per chunk plus a pointer
   update, all identical to what was already there.

   Identity is over the bytes and the chunks are counted, both deliberately: a
   re-composed artifact can carry the same ids with different bytes, and a
   pointer can name a deployment whose chunks are missing, where skipping would
   make an unreadable cache permanent. Both guards verified RED against a naive
   pointer-only skip.

   The gate's stated bound was also wrong and is corrected in the doc: growth
   was never headed for the 1M row cap. AddZbLocalDbReplication is registered
   unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
   and the 7-day MaxOplogAge cap prunes.

Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 01:27:02 -04:00
parent c957db52e2
commit 232bff5df7
7 changed files with 295 additions and 13 deletions
+2 -2
View File
@@ -127,8 +127,8 @@
package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
CVE-2025-6965-fixed native binary without needing the surgical pin below. CVE-2025-6965-fixed native binary without needing the surgical pin below.
--> -->
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" /> <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" /> <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.2" />
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
@@ -72,6 +72,15 @@ The replication library's passive endpoint verifies **no** authentication — th
If a node is offline **longer than the tombstone-retention window**, a delete that happened during If a node is offline **longer than the tombstone-retention window**, a delete that happened during
its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window. briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
cached configuration back to it, with no new deploy and no central SQL. Two things are worth
knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.2** (on `0.1.1` a converged pair has pruned every
oplog row on ack, and the wiped node was never snapshotted — it stayed empty until the next
deploy), and it heals the *cache*, so the node regains boot-from-cache for future outages. A node
wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
peer replication configured self-heals only from central on its next successful apply.
- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at - **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
is only read when the central fetch fails at startup. When central returns, the node resumes is only read when the central fetch fails at startup. When central returns, the node resumes
@@ -33,11 +33,11 @@ Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 | | 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 | | 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added | | 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 limitation documented, 1 defect fixed | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** | | 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) | | 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk | | 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart | | 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up | | 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
## Summary ## Summary
@@ -56,13 +56,32 @@ missed**, each now fixed on-branch with a regression test:
4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its 4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
served state from central but stayed cache-less, unable to boot-from-cache on the next outage. served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
**Two documented limitations (follow-ups, not blockers):** **Two documented limitations (follow-ups, not blockers) — both since CLOSED:**
- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are - **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
not healed by its peer — it self-heals from central on next boot instead. not healed by its peer — it self-heals from central on next boot instead.
**FIXED in `ZB.MOM.WW.LocalDb` 0.1.2** (scadaproj `cad3bcb`). The diagnosis in this doc was
slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
`SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
the latter verified RED against the pinned 0.1.1 and green on 0.1.2.
- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so - **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap; a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
a periodic pruner or a skip-if-unchanged cache write would close it. a periodic pruner or a skip-if-unchanged cache write would close it.
**FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
chunk count is present — so re-caching an artifact the node already holds (every restart's
boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
missing chunks), both verified RED against a naive pointer-only skip.
Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.
Not merged, per the plan. The two limitations are recorded as Phase 1 follow-ups. Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
`fix/localdb-phase1-followups` (2026-07-21).
@@ -63,6 +63,14 @@ public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow); var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow);
var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize; var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize;
if (await IsAlreadyCachedAsync(clusterId, deploymentId, revisionHash, sha, chunkCount, ct))
{
_logger.LogDebug(
"Deployment {DeploymentId} (revision {RevisionHash}) is already cached intact for cluster {ClusterId}; skipping the re-store.",
deploymentId, revisionHash, clusterId);
return;
}
// One transaction for the whole store. A pointer that commits without its chunks — or // One transaction for the whole store. A pointer that commits without its chunks — or
// chunks that commit without the pointer — is a cache that reads back as a corrupt hit // chunks that commit without the pointer — is a cache that reads back as a corrupt hit
// rather than a clean miss, which is the one outcome this type exists to prevent. // rather than a clean miss, which is the one outcome this type exists to prevent.
@@ -155,6 +163,74 @@ public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
await tx.CommitAsync(ct); await tx.CommitAsync(ct);
} }
/// <summary>
/// Whether this exact artifact is already cached <b>and readable</b> for the cluster, so
/// storing it again would rewrite every row to its current value.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this exists.</b> Every write to a replicated table fires a capture trigger and
/// mints an oplog row. Re-caching an artifact the node already holds — which is what a
/// restart's boot-from-cache and every <c>RestoreApplied</c> recovery does — would
/// therefore cost a delete plus an insert per chunk plus a pointer update, all carrying
/// values identical to what is already there. On a replicating pair that is pure churn;
/// on an unpaired (default-OFF) node there is no peer to ack those rows, so they
/// accumulate in the oplog until the library's age cap prunes them. The Phase 1 live gate
/// saw exactly that (check 8).
/// </para>
/// <para>
/// <b>Identity is over the bytes, not the ids.</b> The pointer must match on
/// <c>artifact_sha256</c> as well as the deployment id and revision hash: a re-composed
/// artifact can legitimately carry the same ids with different bytes, and keeping the
/// stale copy would serve a boot-from-cache node the wrong address space while every
/// id-shaped signal claimed it was right.
/// </para>
/// <para>
/// <b>The chunk count is load-bearing.</b> A pointer can name a deployment whose chunks
/// are missing or partial — a half-replicated artifact, or a prune that raced the
/// pointer. Skipping on the pointer alone would make that state permanent, because the
/// re-store that repairs it is the call being skipped.
/// </para>
/// <para>
/// <b>Accepted consequence:</b> a skipped store leaves <c>applied_at_utc</c> at the
/// moment the artifact was <i>first</i> cached rather than last confirmed. Refreshing it
/// would mint the very oplog row this check exists to avoid. Nothing reads it as a
/// liveness signal; retention ranks distinct deployments, which are unaffected.
/// </para>
/// </remarks>
private async Task<bool> IsAlreadyCachedAsync(
string clusterId, string deploymentId, string revisionHash, string sha, int chunkCount,
CancellationToken ct)
{
var matches = await _db.QueryAsync(
"""
SELECT (
SELECT COUNT(*)
FROM deployment_artifacts
WHERE cluster_id = @ClusterId
AND deployment_id = @DeploymentId
AND chunk_count = @ChunkCount
)
FROM deployment_pointer
WHERE cluster_id = @ClusterId
AND deployment_id = @DeploymentId
AND revision_hash = @RevisionHash
AND artifact_sha256 = @Sha
""",
r => r.GetInt32(0),
new
{
ClusterId = clusterId,
DeploymentId = deploymentId,
RevisionHash = revisionHash,
Sha = sha,
ChunkCount = chunkCount,
},
ct);
return matches.Count == 1 && matches[0] == chunkCount;
}
/// <inheritdoc/> /// <inheritdoc/>
public async Task<CachedDeploymentArtifact?> GetCurrentAsync( public async Task<CachedDeploymentArtifact?> GetCurrentAsync(
string clusterId, CancellationToken ct = default) string clusterId, CancellationToken ct = default)
@@ -103,6 +103,54 @@ public sealed class LocalDbPairConvergenceTests
"node B to prune dep1 (with tombstones) while keeping dep3"); "node B to prune dep1 (with tombstones) while keeping dep3");
} }
[Fact]
public async Task WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy()
{
// The Phase 1 live gate's check 4, as an offline scenario. A node whose LocalDb volume is
// lost used to come back permanently empty: the pair had already converged, so every oplog
// row was acked and pruned, and the healthy node had no delta left to send. Nothing short of
// a fresh deploy could repopulate it — which is the worst possible moment to depend on
// central being reachable, since the cache exists precisely for when it is not.
//
// Fixed in ZB.MOM.WW.LocalDb 0.1.2 (snapshot resync now measures the peer's gap against
// last_acked_seq when the oplog is empty). Pin the payoff at this level, not just the
// library's: what matters here is that the deployment artifact itself comes back.
await using var harness = new LocalDbPairHarness();
await harness.StartAsync();
var deploymentId = Guid.NewGuid().ToString("N");
var artifact = MultiChunkArtifact(seed: 20);
await harness.CacheA.StoreAsync(Cluster, deploymentId, "rev-1", artifact,
TestContext.Current.CancellationToken);
await LocalDbPairHarness.WaitUntilAsync(
async () => await ReadArtifactAsync(harness.CacheB, Cluster) is { } b && b.AsSpan().SequenceEqual(artifact),
"the artifact to converge before the wipe");
// Convergence alone is not the precondition — an emptied oplog on A is. That is the state
// that used to make back-fill impossible, so assert it rather than assume it.
await LocalDbPairHarness.WaitUntilAsync(
async () => await LocalDbPairHarness.OplogDepthAsync(harness.A) == 0,
"node A's oplog to be fully acked and pruned");
await harness.WipePassiveAsync();
Assert.Null(await ReadArtifactAsync(harness.CacheB, Cluster));
await harness.RestartPairAsync();
// No new deploy, no central: the rebuilt node gets its cached configuration back from its
// peer alone, byte-identical, and can boot from cache again.
await LocalDbPairHarness.WaitUntilAsync(
async () =>
{
var cached = await harness.CacheB.GetCurrentAsync(Cluster, TestContext.Current.CancellationToken);
return cached is not null
&& cached.DeploymentId == deploymentId
&& cached.Artifact.AsSpan().SequenceEqual(artifact);
},
"the wiped node to be back-filled from its peer");
}
[Fact] [Fact]
public async Task WritesWhileTransportDown_SurviveRejoin() public async Task WritesWhileTransportDown_SurviveRejoin()
{ {
@@ -69,10 +69,13 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
private readonly string _apiKeyB; private readonly string _apiKeyB;
private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"otopcua-pairA-{Guid.NewGuid():N}.db"); private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"otopcua-pairA-{Guid.NewGuid():N}.db");
private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
// Not readonly: WipePassiveAsync replaces node B's database wholesale, which is what a rebuilt
// node with a lost volume actually is — a new file, and with it a new node id.
private string _pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
private readonly ServiceProvider _dbProviderA; private readonly ServiceProvider _dbProviderA;
private readonly ServiceProvider _dbProviderB; private ServiceProvider _dbProviderB;
private IHost? _serverHost; // node B — passive private IHost? _serverHost; // node B — passive
private IHost? _initiatorHost; // node A — dials the peer private IHost? _initiatorHost; // node A — dials the peer
@@ -125,6 +128,30 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
_serverHost = null; _serverHost = null;
} }
/// <summary>
/// Destroys node B's database and replaces it with an empty one, leaving the pairing config
/// untouched — the rebuilt-node case: a lost volume, a re-imaged host, a fresh container.
/// </summary>
/// <remarks>
/// A new file, not a truncated one, because that is what the real thing is: the node id lives
/// in the database, so a rebuilt node comes back with a new identity and a zero watermark. The
/// caller follows with <see cref="RestartPairAsync"/> to bring the wiped node back online.
/// </remarks>
public async Task WipePassiveAsync()
{
await StopPassiveAsync();
await _dbProviderB.DisposeAsync();
// Pooled connections outlive the provider; without this the delete silently no-ops on
// Windows and the "wiped" node would still hold its rows.
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
DeleteDatabaseFiles(_pathB);
_pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
_dbProviderB = BuildDatabaseProvider(_pathB);
_ = B;
}
/// <summary> /// <summary>
/// Brings node B back on a NEW loopback port and re-dials from A. The initiator re-reads the /// Brings node B back on a NEW loopback port and re-dials from A. The initiator re-reads the
/// peer address on each reconnect, so this is a genuine rejoin over the same databases. /// peer address on each reconnect, so this is a genuine rejoin over the same databases.
@@ -201,6 +228,17 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
return rows[0]; return rows[0];
} }
/// <summary>
/// Counts oplog rows on a node. Zero means every local change has been acked by the peer and
/// pruned — the steady state of a converged pair, and the state from which a wiped peer can
/// only be healed by a snapshot.
/// </summary>
public static async Task<long> OplogDepthAsync(ILocalDb db)
{
var rows = await db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", static r => r.GetInt64(0));
return rows[0];
}
/// <summary>Counts tombstone rows in <c>__localdb_row_version</c> for one table on a node.</summary> /// <summary>Counts tombstone rows in <c>__localdb_row_version</c> for one table on a node.</summary>
public static async Task<long> TombstoneCountAsync(ILocalDb db, string table) public static async Task<long> TombstoneCountAsync(ILocalDb db, string table)
{ {
@@ -313,12 +351,16 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
await _dbProviderB.DisposeAsync(); await _dbProviderB.DisposeAsync();
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
foreach (var path in new[] { _pathA, _pathB }) DeleteDatabaseFiles(_pathA);
DeleteDatabaseFiles(_pathB);
}
/// <summary>Deletes a SQLite database and its WAL/shm sidecars, best effort.</summary>
private static void DeleteDatabaseFiles(string path)
{ {
foreach (var suffix in new[] { "", "-wal", "-shm" }) foreach (var suffix in new[] { "", "-wal", "-shm" })
{ {
try { File.Delete(path + suffix); } catch { /* best effort */ } try { File.Delete(path + suffix); } catch { /* best effort */ }
} }
} }
}
} }
@@ -90,6 +90,23 @@ public sealed class LocalDbDeploymentArtifactCacheTests : IDisposable
"SELECT DISTINCT deployment_id FROM deployment_artifacts ORDER BY deployment_id", "SELECT DISTINCT deployment_id FROM deployment_artifacts ORDER BY deployment_id",
r => r.GetString(0)); r => r.GetString(0));
/// <summary>
/// Total capture-trigger rows in the replication oplog — the direct measure of how much
/// replication traffic (and, on an unpaired node, how much permanent oplog growth) a store
/// costs.
/// </summary>
private async Task<long> OplogDepthAsync()
=> (await _db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", r => r.GetInt64(0)))[0];
/// <summary>
/// The LWW version stamps of every replicated row, as a stable ordered projection. Comparing
/// it across a call proves whether that call touched replicated state at all.
/// </summary>
private async Task<IReadOnlyList<string>> RowVersionsAsync()
=> await _db.QueryAsync(
"SELECT table_name, pk_json, hlc FROM __localdb_row_version ORDER BY table_name, pk_json",
r => $"{r.GetString(0)}|{r.GetString(1)}|{r.GetInt64(2)}");
[Fact] [Fact]
public async Task StoreThenGetCurrent_RoundTripsAMultiChunkArtifactByteForByte() public async Task StoreThenGetCurrent_RoundTripsAMultiChunkArtifactByteForByte()
{ {
@@ -221,6 +238,77 @@ public sealed class LocalDbDeploymentArtifactCacheTests : IDisposable
cached.Artifact.ShouldBe(second); cached.Artifact.ShouldBe(second);
} }
[Fact]
public async Task Store_WritesNothingWhenTheCachedArtifactIsAlreadyIdentical()
{
// Phase 1 live gate, check 8: every store rewrote the chunks and the pointer unconditionally,
// so re-caching an artifact the node already holds — which is what boot-from-cache and
// RestoreApplied both do on every restart — minted fresh oplog rows for a row nobody changed.
// On a replicating pair that is pure churn; on a default-OFF node there is no peer to ack
// them, so they sit in the oplog until the age cap eventually prunes them.
var cache = CreateCache();
var artifact = RandomArtifact(300 * 1024);
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
var oplogDepth = await OplogDepthAsync();
var rowVersions = await RowVersionsAsync();
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
(await OplogDepthAsync()).ShouldBe(oplogDepth);
(await RowVersionsAsync()).ShouldBe(rowVersions);
// Skipping must not cost readability: the artifact still reads back intact.
var cached = await cache.GetCurrentAsync(ClusterA);
cached.ShouldNotBeNull();
cached.DeploymentId.ShouldBe("dep-1");
cached.Artifact.ShouldBe(artifact);
}
[Fact]
public async Task Store_RewritesWhenTheCachedChunksAreIncompleteDespiteAMatchingPointer()
{
// The dangerous half of skip-if-unchanged. A pointer can name a deployment whose chunks are
// gone or partial — a half-replicated artifact, or a prune that raced the pointer. Trusting
// the pointer alone would turn a repairable cache into a permanently unreadable one, because
// the re-store that would have fixed it is exactly the call being skipped.
var cache = CreateCache();
var artifact = RandomArtifact(300 * 1024);
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
await _db.ExecuteAsync(
"DELETE FROM deployment_artifacts WHERE deployment_id = @DeploymentId AND chunk_index = 2",
new { DeploymentId = "dep-1" });
(await cache.GetCurrentAsync(ClusterA)).ShouldBeNull();
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
(await CountChunksAsync("dep-1")).ShouldBe(3);
var cached = await cache.GetCurrentAsync(ClusterA);
cached.ShouldNotBeNull();
cached.Artifact.ShouldBe(artifact);
}
[Fact]
public async Task Store_RewritesWhenTheArtifactBytesChangedUnderAnUnchangedRevision()
{
// The identity check must be over the bytes, not just the ids. A rebuilt artifact carrying
// the same deployment id and revision hash is a real case (the same revision re-composed),
// and silently keeping the stale bytes would serve a boot-from-cache node the wrong address
// space with every id-shaped signal saying it was correct.
var cache = CreateCache();
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", RandomArtifact(1024));
var rebuilt = RandomArtifact(1024);
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", rebuilt);
var cached = await cache.GetCurrentAsync(ClusterA);
cached.ShouldNotBeNull();
cached.Artifact.ShouldBe(rebuilt);
}
[Fact] [Fact]
public async Task GetCurrentUnkeyed_ReturnsTheOnlyCachedArtifact() public async Task GetCurrentUnkeyed_ReturnsTheOnlyCachedArtifact()
{ {