fix(localdb): close both Phase 1 live-gate follow-ups
1. Wiped-node back-fill (live gate check 4). Fixed upstream in ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly off: the oplog cap was not the gate. Snapshot detection measured the peer's gap against the oldest surviving oplog row and read an empty oplog as "no gap possible" — and an empty oplog is the steady state of a converged pair, since ack-pruning deletes everything the peer confirmed. The healthy state was the one state that could not heal a wiped peer. Now measured against last_acked_seq when the oplog is empty. Pinned at this level too, not just the library's: the pair harness grows a WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node id and a zero watermark), and the new scenario asserts the emptied oplog as its precondition before wiping, then requires the deployment artifact to come back byte-identical with no new deploy. Verified RED against the pinned 0.1.1 (times out waiting for back-fill) and green on 0.1.2. 2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source: StoreAsync now skips entirely when the pointer already names this deployment/revision, the SHA matches the bytes, and the expected chunk count is present. Re-caching an artifact the node already holds — every restart's boot-from-cache, every RestoreApplied — writes nothing and mints no oplog rows, where before it cost a delete plus an insert per chunk plus a pointer update, all identical to what was already there. Identity is over the bytes and the chunks are counted, both deliberately: a re-composed artifact can carry the same ids with different bytes, and a pointer can name a deployment whose chunks are missing, where skipping would make an unreadable cache permanent. Both guards verified RED against a naive pointer-only skip. The gate's stated bound was also wrong and is corrected in the doc: growth was never headed for the 1M row cap. AddZbLocalDbReplication is registered unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes and the 7-day MaxOplogAge cap prunes. Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a clean master too) and a flaky Roslyn race test (green 2/2 in isolation). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -127,8 +127,8 @@
|
|||||||
package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
|
package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
|
||||||
CVE-2025-6965-fixed native binary without needing the surgical pin below.
|
CVE-2025-6965-fixed native binary without needing the surgical pin below.
|
||||||
-->
|
-->
|
||||||
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
|
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.2" />
|
||||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
|
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.2" />
|
||||||
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
|
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
|
||||||
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
|
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
|
||||||
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
|
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
|
||||||
|
|||||||
@@ -72,6 +72,15 @@ The replication library's passive endpoint verifies **no** authentication — th
|
|||||||
If a node is offline **longer than the tombstone-retention window**, a delete that happened during
|
If a node is offline **longer than the tombstone-retention window**, a delete that happened during
|
||||||
its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
|
its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
|
||||||
briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
|
briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
|
||||||
|
- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
|
||||||
|
a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
|
||||||
|
cached configuration back to it, with no new deploy and no central SQL. Two things are worth
|
||||||
|
knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.2** (on `0.1.1` a converged pair has pruned every
|
||||||
|
oplog row on ack, and the wiped node was never snapshotted — it stayed empty until the next
|
||||||
|
deploy), and it heals the *cache*, so the node regains boot-from-cache for future outages. A node
|
||||||
|
wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
|
||||||
|
peer replication configured self-heals only from central on its next successful apply.
|
||||||
|
|
||||||
- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
|
- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
|
||||||
boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
|
boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
|
||||||
is only read when the central fetch fails at startup. When central returns, the node resumes
|
is only read when the central fetch fails at startup. When central returns, the node resumes
|
||||||
|
|||||||
@@ -33,11 +33,11 @@ Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
|
|||||||
| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
|
| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
|
||||||
| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
|
| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
|
||||||
| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
|
| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
|
||||||
| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 limitation documented, 1 defect fixed | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
|
| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
|
||||||
| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
|
| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
|
||||||
| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
|
| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
|
||||||
| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
|
| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
|
||||||
| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
|
| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
@@ -56,13 +56,32 @@ missed**, each now fixed on-branch with a regression test:
|
|||||||
4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
|
4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
|
||||||
served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
|
served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
|
||||||
|
|
||||||
**Two documented limitations (follow-ups, not blockers):**
|
**Two documented limitations (follow-ups, not blockers) — both since CLOSED:**
|
||||||
|
|
||||||
- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
|
- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
|
||||||
pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
|
pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
|
||||||
not healed by its peer — it self-heals from central on next boot instead.
|
not healed by its peer — it self-heals from central on next boot instead.
|
||||||
|
→ **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2** (scadaproj `cad3bcb`). The diagnosis in this doc was
|
||||||
|
slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
|
||||||
|
against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
|
||||||
|
empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
|
||||||
|
peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
|
||||||
|
an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
|
||||||
|
`SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
|
||||||
|
and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
|
||||||
|
the latter verified RED against the pinned 0.1.1 and green on 0.1.2.
|
||||||
- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
|
- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
|
||||||
a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
|
a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
|
||||||
a periodic pruner or a skip-if-unchanged cache write would close it.
|
a periodic pruner or a skip-if-unchanged cache write would close it.
|
||||||
|
→ **FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
|
||||||
|
the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
|
||||||
|
chunk count is present — so re-caching an artifact the node already holds (every restart's
|
||||||
|
boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
|
||||||
|
pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
|
||||||
|
missing chunks), both verified RED against a naive pointer-only skip.
|
||||||
|
Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
|
||||||
|
is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
|
||||||
|
the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.
|
||||||
|
|
||||||
Not merged, per the plan. The two limitations are recorded as Phase 1 follow-ups.
|
Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
|
||||||
|
`fix/localdb-phase1-followups` (2026-07-21).
|
||||||
|
|||||||
@@ -63,6 +63,14 @@ public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
|
|||||||
var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow);
|
var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow);
|
||||||
var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize;
|
var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize;
|
||||||
|
|
||||||
|
if (await IsAlreadyCachedAsync(clusterId, deploymentId, revisionHash, sha, chunkCount, ct))
|
||||||
|
{
|
||||||
|
_logger.LogDebug(
|
||||||
|
"Deployment {DeploymentId} (revision {RevisionHash}) is already cached intact for cluster {ClusterId}; skipping the re-store.",
|
||||||
|
deploymentId, revisionHash, clusterId);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// One transaction for the whole store. A pointer that commits without its chunks — or
|
// One transaction for the whole store. A pointer that commits without its chunks — or
|
||||||
// chunks that commit without the pointer — is a cache that reads back as a corrupt hit
|
// chunks that commit without the pointer — is a cache that reads back as a corrupt hit
|
||||||
// rather than a clean miss, which is the one outcome this type exists to prevent.
|
// rather than a clean miss, which is the one outcome this type exists to prevent.
|
||||||
@@ -155,6 +163,74 @@ public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
|
|||||||
await tx.CommitAsync(ct);
|
await tx.CommitAsync(ct);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Whether this exact artifact is already cached <b>and readable</b> for the cluster, so
|
||||||
|
/// storing it again would rewrite every row to its current value.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Why this exists.</b> Every write to a replicated table fires a capture trigger and
|
||||||
|
/// mints an oplog row. Re-caching an artifact the node already holds — which is what a
|
||||||
|
/// restart's boot-from-cache and every <c>RestoreApplied</c> recovery does — would
|
||||||
|
/// therefore cost a delete plus an insert per chunk plus a pointer update, all carrying
|
||||||
|
/// values identical to what is already there. On a replicating pair that is pure churn;
|
||||||
|
/// on an unpaired (default-OFF) node there is no peer to ack those rows, so they
|
||||||
|
/// accumulate in the oplog until the library's age cap prunes them. The Phase 1 live gate
|
||||||
|
/// saw exactly that (check 8).
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Identity is over the bytes, not the ids.</b> The pointer must match on
|
||||||
|
/// <c>artifact_sha256</c> as well as the deployment id and revision hash: a re-composed
|
||||||
|
/// artifact can legitimately carry the same ids with different bytes, and keeping the
|
||||||
|
/// stale copy would serve a boot-from-cache node the wrong address space while every
|
||||||
|
/// id-shaped signal claimed it was right.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>The chunk count is load-bearing.</b> A pointer can name a deployment whose chunks
|
||||||
|
/// are missing or partial — a half-replicated artifact, or a prune that raced the
|
||||||
|
/// pointer. Skipping on the pointer alone would make that state permanent, because the
|
||||||
|
/// re-store that repairs it is the call being skipped.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Accepted consequence:</b> a skipped store leaves <c>applied_at_utc</c> at the
|
||||||
|
/// moment the artifact was <i>first</i> cached rather than last confirmed. Refreshing it
|
||||||
|
/// would mint the very oplog row this check exists to avoid. Nothing reads it as a
|
||||||
|
/// liveness signal; retention ranks distinct deployments, which are unaffected.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
private async Task<bool> IsAlreadyCachedAsync(
|
||||||
|
string clusterId, string deploymentId, string revisionHash, string sha, int chunkCount,
|
||||||
|
CancellationToken ct)
|
||||||
|
{
|
||||||
|
var matches = await _db.QueryAsync(
|
||||||
|
"""
|
||||||
|
SELECT (
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM deployment_artifacts
|
||||||
|
WHERE cluster_id = @ClusterId
|
||||||
|
AND deployment_id = @DeploymentId
|
||||||
|
AND chunk_count = @ChunkCount
|
||||||
|
)
|
||||||
|
FROM deployment_pointer
|
||||||
|
WHERE cluster_id = @ClusterId
|
||||||
|
AND deployment_id = @DeploymentId
|
||||||
|
AND revision_hash = @RevisionHash
|
||||||
|
AND artifact_sha256 = @Sha
|
||||||
|
""",
|
||||||
|
r => r.GetInt32(0),
|
||||||
|
new
|
||||||
|
{
|
||||||
|
ClusterId = clusterId,
|
||||||
|
DeploymentId = deploymentId,
|
||||||
|
RevisionHash = revisionHash,
|
||||||
|
Sha = sha,
|
||||||
|
ChunkCount = chunkCount,
|
||||||
|
},
|
||||||
|
ct);
|
||||||
|
|
||||||
|
return matches.Count == 1 && matches[0] == chunkCount;
|
||||||
|
}
|
||||||
|
|
||||||
/// <inheritdoc/>
|
/// <inheritdoc/>
|
||||||
public async Task<CachedDeploymentArtifact?> GetCurrentAsync(
|
public async Task<CachedDeploymentArtifact?> GetCurrentAsync(
|
||||||
string clusterId, CancellationToken ct = default)
|
string clusterId, CancellationToken ct = default)
|
||||||
|
|||||||
+48
@@ -103,6 +103,54 @@ public sealed class LocalDbPairConvergenceTests
|
|||||||
"node B to prune dep1 (with tombstones) while keeping dep3");
|
"node B to prune dep1 (with tombstones) while keeping dep3");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy()
|
||||||
|
{
|
||||||
|
// The Phase 1 live gate's check 4, as an offline scenario. A node whose LocalDb volume is
|
||||||
|
// lost used to come back permanently empty: the pair had already converged, so every oplog
|
||||||
|
// row was acked and pruned, and the healthy node had no delta left to send. Nothing short of
|
||||||
|
// a fresh deploy could repopulate it — which is the worst possible moment to depend on
|
||||||
|
// central being reachable, since the cache exists precisely for when it is not.
|
||||||
|
//
|
||||||
|
// Fixed in ZB.MOM.WW.LocalDb 0.1.2 (snapshot resync now measures the peer's gap against
|
||||||
|
// last_acked_seq when the oplog is empty). Pin the payoff at this level, not just the
|
||||||
|
// library's: what matters here is that the deployment artifact itself comes back.
|
||||||
|
await using var harness = new LocalDbPairHarness();
|
||||||
|
await harness.StartAsync();
|
||||||
|
|
||||||
|
var deploymentId = Guid.NewGuid().ToString("N");
|
||||||
|
var artifact = MultiChunkArtifact(seed: 20);
|
||||||
|
await harness.CacheA.StoreAsync(Cluster, deploymentId, "rev-1", artifact,
|
||||||
|
TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
await LocalDbPairHarness.WaitUntilAsync(
|
||||||
|
async () => await ReadArtifactAsync(harness.CacheB, Cluster) is { } b && b.AsSpan().SequenceEqual(artifact),
|
||||||
|
"the artifact to converge before the wipe");
|
||||||
|
|
||||||
|
// Convergence alone is not the precondition — an emptied oplog on A is. That is the state
|
||||||
|
// that used to make back-fill impossible, so assert it rather than assume it.
|
||||||
|
await LocalDbPairHarness.WaitUntilAsync(
|
||||||
|
async () => await LocalDbPairHarness.OplogDepthAsync(harness.A) == 0,
|
||||||
|
"node A's oplog to be fully acked and pruned");
|
||||||
|
|
||||||
|
await harness.WipePassiveAsync();
|
||||||
|
Assert.Null(await ReadArtifactAsync(harness.CacheB, Cluster));
|
||||||
|
|
||||||
|
await harness.RestartPairAsync();
|
||||||
|
|
||||||
|
// No new deploy, no central: the rebuilt node gets its cached configuration back from its
|
||||||
|
// peer alone, byte-identical, and can boot from cache again.
|
||||||
|
await LocalDbPairHarness.WaitUntilAsync(
|
||||||
|
async () =>
|
||||||
|
{
|
||||||
|
var cached = await harness.CacheB.GetCurrentAsync(Cluster, TestContext.Current.CancellationToken);
|
||||||
|
return cached is not null
|
||||||
|
&& cached.DeploymentId == deploymentId
|
||||||
|
&& cached.Artifact.AsSpan().SequenceEqual(artifact);
|
||||||
|
},
|
||||||
|
"the wiped node to be back-filled from its peer");
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task WritesWhileTransportDown_SurviveRejoin()
|
public async Task WritesWhileTransportDown_SurviveRejoin()
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -69,10 +69,13 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
|
|||||||
private readonly string _apiKeyB;
|
private readonly string _apiKeyB;
|
||||||
|
|
||||||
private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"otopcua-pairA-{Guid.NewGuid():N}.db");
|
private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"otopcua-pairA-{Guid.NewGuid():N}.db");
|
||||||
private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
|
|
||||||
|
// Not readonly: WipePassiveAsync replaces node B's database wholesale, which is what a rebuilt
|
||||||
|
// node with a lost volume actually is — a new file, and with it a new node id.
|
||||||
|
private string _pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
|
||||||
|
|
||||||
private readonly ServiceProvider _dbProviderA;
|
private readonly ServiceProvider _dbProviderA;
|
||||||
private readonly ServiceProvider _dbProviderB;
|
private ServiceProvider _dbProviderB;
|
||||||
|
|
||||||
private IHost? _serverHost; // node B — passive
|
private IHost? _serverHost; // node B — passive
|
||||||
private IHost? _initiatorHost; // node A — dials the peer
|
private IHost? _initiatorHost; // node A — dials the peer
|
||||||
@@ -125,6 +128,30 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
|
|||||||
_serverHost = null;
|
_serverHost = null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Destroys node B's database and replaces it with an empty one, leaving the pairing config
|
||||||
|
/// untouched — the rebuilt-node case: a lost volume, a re-imaged host, a fresh container.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// A new file, not a truncated one, because that is what the real thing is: the node id lives
|
||||||
|
/// in the database, so a rebuilt node comes back with a new identity and a zero watermark. The
|
||||||
|
/// caller follows with <see cref="RestartPairAsync"/> to bring the wiped node back online.
|
||||||
|
/// </remarks>
|
||||||
|
public async Task WipePassiveAsync()
|
||||||
|
{
|
||||||
|
await StopPassiveAsync();
|
||||||
|
await _dbProviderB.DisposeAsync();
|
||||||
|
|
||||||
|
// Pooled connections outlive the provider; without this the delete silently no-ops on
|
||||||
|
// Windows and the "wiped" node would still hold its rows.
|
||||||
|
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||||
|
DeleteDatabaseFiles(_pathB);
|
||||||
|
|
||||||
|
_pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
|
||||||
|
_dbProviderB = BuildDatabaseProvider(_pathB);
|
||||||
|
_ = B;
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Brings node B back on a NEW loopback port and re-dials from A. The initiator re-reads the
|
/// Brings node B back on a NEW loopback port and re-dials from A. The initiator re-reads the
|
||||||
/// peer address on each reconnect, so this is a genuine rejoin over the same databases.
|
/// peer address on each reconnect, so this is a genuine rejoin over the same databases.
|
||||||
@@ -201,6 +228,17 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
|
|||||||
return rows[0];
|
return rows[0];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Counts oplog rows on a node. Zero means every local change has been acked by the peer and
|
||||||
|
/// pruned — the steady state of a converged pair, and the state from which a wiped peer can
|
||||||
|
/// only be healed by a snapshot.
|
||||||
|
/// </summary>
|
||||||
|
public static async Task<long> OplogDepthAsync(ILocalDb db)
|
||||||
|
{
|
||||||
|
var rows = await db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", static r => r.GetInt64(0));
|
||||||
|
return rows[0];
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Counts tombstone rows in <c>__localdb_row_version</c> for one table on a node.</summary>
|
/// <summary>Counts tombstone rows in <c>__localdb_row_version</c> for one table on a node.</summary>
|
||||||
public static async Task<long> TombstoneCountAsync(ILocalDb db, string table)
|
public static async Task<long> TombstoneCountAsync(ILocalDb db, string table)
|
||||||
{
|
{
|
||||||
@@ -313,12 +351,16 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
|
|||||||
await _dbProviderB.DisposeAsync();
|
await _dbProviderB.DisposeAsync();
|
||||||
|
|
||||||
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||||
foreach (var path in new[] { _pathA, _pathB })
|
DeleteDatabaseFiles(_pathA);
|
||||||
|
DeleteDatabaseFiles(_pathB);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Deletes a SQLite database and its WAL/shm sidecars, best effort.</summary>
|
||||||
|
private static void DeleteDatabaseFiles(string path)
|
||||||
|
{
|
||||||
|
foreach (var suffix in new[] { "", "-wal", "-shm" })
|
||||||
{
|
{
|
||||||
foreach (var suffix in new[] { "", "-wal", "-shm" })
|
try { File.Delete(path + suffix); } catch { /* best effort */ }
|
||||||
{
|
|
||||||
try { File.Delete(path + suffix); } catch { /* best effort */ }
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+88
@@ -90,6 +90,23 @@ public sealed class LocalDbDeploymentArtifactCacheTests : IDisposable
|
|||||||
"SELECT DISTINCT deployment_id FROM deployment_artifacts ORDER BY deployment_id",
|
"SELECT DISTINCT deployment_id FROM deployment_artifacts ORDER BY deployment_id",
|
||||||
r => r.GetString(0));
|
r => r.GetString(0));
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Total capture-trigger rows in the replication oplog — the direct measure of how much
|
||||||
|
/// replication traffic (and, on an unpaired node, how much permanent oplog growth) a store
|
||||||
|
/// costs.
|
||||||
|
/// </summary>
|
||||||
|
private async Task<long> OplogDepthAsync()
|
||||||
|
=> (await _db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", r => r.GetInt64(0)))[0];
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The LWW version stamps of every replicated row, as a stable ordered projection. Comparing
|
||||||
|
/// it across a call proves whether that call touched replicated state at all.
|
||||||
|
/// </summary>
|
||||||
|
private async Task<IReadOnlyList<string>> RowVersionsAsync()
|
||||||
|
=> await _db.QueryAsync(
|
||||||
|
"SELECT table_name, pk_json, hlc FROM __localdb_row_version ORDER BY table_name, pk_json",
|
||||||
|
r => $"{r.GetString(0)}|{r.GetString(1)}|{r.GetInt64(2)}");
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task StoreThenGetCurrent_RoundTripsAMultiChunkArtifactByteForByte()
|
public async Task StoreThenGetCurrent_RoundTripsAMultiChunkArtifactByteForByte()
|
||||||
{
|
{
|
||||||
@@ -221,6 +238,77 @@ public sealed class LocalDbDeploymentArtifactCacheTests : IDisposable
|
|||||||
cached.Artifact.ShouldBe(second);
|
cached.Artifact.ShouldBe(second);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Store_WritesNothingWhenTheCachedArtifactIsAlreadyIdentical()
|
||||||
|
{
|
||||||
|
// Phase 1 live gate, check 8: every store rewrote the chunks and the pointer unconditionally,
|
||||||
|
// so re-caching an artifact the node already holds — which is what boot-from-cache and
|
||||||
|
// RestoreApplied both do on every restart — minted fresh oplog rows for a row nobody changed.
|
||||||
|
// On a replicating pair that is pure churn; on a default-OFF node there is no peer to ack
|
||||||
|
// them, so they sit in the oplog until the age cap eventually prunes them.
|
||||||
|
var cache = CreateCache();
|
||||||
|
var artifact = RandomArtifact(300 * 1024);
|
||||||
|
|
||||||
|
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
|
||||||
|
|
||||||
|
var oplogDepth = await OplogDepthAsync();
|
||||||
|
var rowVersions = await RowVersionsAsync();
|
||||||
|
|
||||||
|
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
|
||||||
|
|
||||||
|
(await OplogDepthAsync()).ShouldBe(oplogDepth);
|
||||||
|
(await RowVersionsAsync()).ShouldBe(rowVersions);
|
||||||
|
|
||||||
|
// Skipping must not cost readability: the artifact still reads back intact.
|
||||||
|
var cached = await cache.GetCurrentAsync(ClusterA);
|
||||||
|
cached.ShouldNotBeNull();
|
||||||
|
cached.DeploymentId.ShouldBe("dep-1");
|
||||||
|
cached.Artifact.ShouldBe(artifact);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Store_RewritesWhenTheCachedChunksAreIncompleteDespiteAMatchingPointer()
|
||||||
|
{
|
||||||
|
// The dangerous half of skip-if-unchanged. A pointer can name a deployment whose chunks are
|
||||||
|
// gone or partial — a half-replicated artifact, or a prune that raced the pointer. Trusting
|
||||||
|
// the pointer alone would turn a repairable cache into a permanently unreadable one, because
|
||||||
|
// the re-store that would have fixed it is exactly the call being skipped.
|
||||||
|
var cache = CreateCache();
|
||||||
|
var artifact = RandomArtifact(300 * 1024);
|
||||||
|
|
||||||
|
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
|
||||||
|
await _db.ExecuteAsync(
|
||||||
|
"DELETE FROM deployment_artifacts WHERE deployment_id = @DeploymentId AND chunk_index = 2",
|
||||||
|
new { DeploymentId = "dep-1" });
|
||||||
|
(await cache.GetCurrentAsync(ClusterA)).ShouldBeNull();
|
||||||
|
|
||||||
|
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", artifact);
|
||||||
|
|
||||||
|
(await CountChunksAsync("dep-1")).ShouldBe(3);
|
||||||
|
var cached = await cache.GetCurrentAsync(ClusterA);
|
||||||
|
cached.ShouldNotBeNull();
|
||||||
|
cached.Artifact.ShouldBe(artifact);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Store_RewritesWhenTheArtifactBytesChangedUnderAnUnchangedRevision()
|
||||||
|
{
|
||||||
|
// The identity check must be over the bytes, not just the ids. A rebuilt artifact carrying
|
||||||
|
// the same deployment id and revision hash is a real case (the same revision re-composed),
|
||||||
|
// and silently keeping the stale bytes would serve a boot-from-cache node the wrong address
|
||||||
|
// space with every id-shaped signal saying it was correct.
|
||||||
|
var cache = CreateCache();
|
||||||
|
|
||||||
|
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", RandomArtifact(1024));
|
||||||
|
|
||||||
|
var rebuilt = RandomArtifact(1024);
|
||||||
|
await cache.StoreAsync(ClusterA, "dep-1", "rev-1", rebuilt);
|
||||||
|
|
||||||
|
var cached = await cache.GetCurrentAsync(ClusterA);
|
||||||
|
cached.ShouldNotBeNull();
|
||||||
|
cached.Artifact.ShouldBe(rebuilt);
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task GetCurrentUnkeyed_ReturnsTheOnlyCachedArtifact()
|
public async Task GetCurrentUnkeyed_ReturnsTheOnlyCachedArtifact()
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user