Add LMDB oplog migration path with dual-write cutover support
All checks were successful
NuGet Package Publish / nuget (push) Successful in 1m16s
All checks were successful
NuGet Package Publish / nuget (push) Successful in 1m16s
Introduce LMDB oplog store, migration flags, telemetry/backfill tooling, and parity tests to enable staged Surreal-to-LMDB rollout with rollback coverage.
This commit is contained in:
Joseph Doherty
55
docs/adr/0002-lmdb-oplog-migration.md
Normal file
55
docs/adr/0002-lmdb-oplog-migration.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# ADR 0002: LMDB Oplog Migration
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The existing oplog persistence layer is Surreal-backed and tightly coupled to local CDC transaction boundaries. We need an LMDB-backed oplog path to improve prune efficiency and reduce query latency while preserving existing `IOplogStore` semantics and low-risk rollback.
|
||||||
|
|
||||||
|
Key constraints:
|
||||||
|
- Dedupe by hash.
|
||||||
|
- Hash lookup, node/time scans, and chain reconstruction.
|
||||||
|
- Cutoff-based prune (not strict FIFO) with interleaved late arrivals.
|
||||||
|
- Dataset isolation in all indexes.
|
||||||
|
- Explicit handling for cross-engine atomicity when document metadata remains Surreal-backed.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Adopt an LMDB oplog provider (`LmdbOplogStore`) with feature-flag controlled migration:
|
||||||
|
- `UseLmdbOplog`
|
||||||
|
- `DualWriteOplog`
|
||||||
|
- `PreferLmdbReads`
|
||||||
|
|
||||||
|
### LMDB index schema
|
||||||
|
|
||||||
|
Single environment with named DBIs:
|
||||||
|
- `oplog_by_hash`: `{datasetId}|{hash}` -> serialized `OplogEntry`
|
||||||
|
- `oplog_by_hlc`: `{datasetId}|{wall}|{logic}|{nodeId}|{hash}` -> marker
|
||||||
|
- `oplog_by_node_hlc`: `{datasetId}|{nodeId}|{wall}|{logic}|{hash}` -> marker
|
||||||
|
- `oplog_prev_to_hash` (`DUPSORT`): `{datasetId}|{previousHash}` -> `{hash}`
|
||||||
|
- `oplog_node_head`: `{datasetId}|{nodeId}` -> `{wall,logic,hash}`
|
||||||
|
- `oplog_meta`: schema/version markers + dataset prune watermark
|
||||||
|
|
||||||
|
Composite keys use deterministic byte encodings with dataset prefixes on every index.
|
||||||
|
|
||||||
|
### Prune algorithm
|
||||||
|
|
||||||
|
Prune scans `oplog_by_hlc` up to cutoff and removes each candidate from all indexes, then recomputes touched node-head entries. Deletes run in bounded batches (`PruneBatchSize`) inside write transactions.
|
||||||
|
|
||||||
|
### Consistency model
|
||||||
|
|
||||||
|
Phase-1 consistency model is Option A (eventual cross-engine atomicity):
|
||||||
|
- Surreal local CDC writes remain authoritative for atomic document+metadata+checkpoint transactions.
|
||||||
|
- LMDB is backfilled/reconciled from Surreal when LMDB reads are preferred and gaps are detected.
|
||||||
|
- Dual-write is available for sync-path writes to accelerate cutover confidence.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Enables staged rollout (dual-write and read shadow validation before cutover).
|
||||||
|
- Improves prune/query performance characteristics via ordered LMDB indexes.
|
||||||
|
- Keeps rollback low-risk by retaining Surreal source-of-truth during migration windows.
|
||||||
|
- Requires reconciliation logic and operational monitoring of mismatch counters/logs during migration.
|
||||||
|
- Includes a dedicated backfill utility (`LmdbOplogBackfillTool`) with parity report output.
|
||||||
|
- Exposes migration telemetry counters (`OplogMigrationTelemetry`) for mismatch/reconciliation tracking.
|
||||||
@@ -237,6 +237,83 @@ Surreal persistence now stores `datasetId` on oplog, metadata, snapshot metadata
|
|||||||
4. **Delete durability**: deletes persist as oplog delete operations plus tombstone metadata.
|
4. **Delete durability**: deletes persist as oplog delete operations plus tombstone metadata.
|
||||||
5. **Remote apply behavior**: remote sync applies documents without generating local loopback CDC entries.
|
5. **Remote apply behavior**: remote sync applies documents without generating local loopback CDC entries.
|
||||||
|
|
||||||
|
## LMDB Oplog Migration Mode
|
||||||
|
|
||||||
|
CBDDC now supports an LMDB-backed oplog provider for staged cutover from Surreal oplog tables.
|
||||||
|
|
||||||
|
### Registration
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
services.AddCBDDCCore()
|
||||||
|
.AddCBDDCSurrealEmbedded<SampleDocumentStore>(optionsFactory)
|
||||||
|
.AddCBDDCLmdbOplog(
|
||||||
|
_ => new LmdbOplogOptions
|
||||||
|
{
|
||||||
|
EnvironmentPath = "/var/lib/cbddc/oplog-lmdb",
|
||||||
|
MapSizeBytes = 256L * 1024 * 1024,
|
||||||
|
MaxDatabases = 16,
|
||||||
|
PruneBatchSize = 512
|
||||||
|
},
|
||||||
|
flags =>
|
||||||
|
{
|
||||||
|
flags.UseLmdbOplog = true;
|
||||||
|
flags.DualWriteOplog = true;
|
||||||
|
flags.PreferLmdbReads = false;
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Feature Flags
|
||||||
|
|
||||||
|
- `UseLmdbOplog`: enables LMDB migration path.
|
||||||
|
- `DualWriteOplog`: mirrors writes to Surreal + LMDB.
|
||||||
|
- `PreferLmdbReads`: cuts reads over to LMDB.
|
||||||
|
- `EnableReadShadowValidation`: compares Surreal/LMDB read results and logs mismatches.
|
||||||
|
|
||||||
|
### Consistency Model
|
||||||
|
|
||||||
|
The initial migration model is eventual cross-engine atomicity (Option A):
|
||||||
|
|
||||||
|
- Surreal local CDC transactions remain authoritative for atomic document + metadata persistence.
|
||||||
|
- LMDB is backfilled/reconciled when LMDB reads are preferred and LMDB is missing recent Surreal writes.
|
||||||
|
- During rollout, keep dual-write enabled until mismatch logs remain stable.
|
||||||
|
|
||||||
|
### Backfill Utility
|
||||||
|
|
||||||
|
`LmdbOplogBackfillTool` performs Surreal -> LMDB oplog backfill and parity validation per dataset:
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
var backfill = provider.GetRequiredService<LmdbOplogBackfillTool>();
|
||||||
|
LmdbOplogBackfillReport report = await backfill.BackfillOrThrowAsync(DatasetId.Primary);
|
||||||
|
```
|
||||||
|
|
||||||
|
Validation includes:
|
||||||
|
- total entry counts
|
||||||
|
- per-node entry counts
|
||||||
|
- latest hash per node
|
||||||
|
- hash spot checks
|
||||||
|
- chain-range spot checks
|
||||||
|
|
||||||
|
### Migration Telemetry
|
||||||
|
|
||||||
|
`FeatureFlagOplogStore` records migration counters through `OplogMigrationTelemetry`:
|
||||||
|
|
||||||
|
- shadow comparisons
|
||||||
|
- shadow mismatches
|
||||||
|
- LMDB preferred-read fallbacks to Surreal
|
||||||
|
- reconciliation runs and reconciled entry counts (global + per dataset)
|
||||||
|
|
||||||
|
You can resolve `OplogMigrationTelemetry` from DI or call `GetTelemetrySnapshot()` on `FeatureFlagOplogStore`.
|
||||||
|
|
||||||
|
### Rollback Path
|
||||||
|
|
||||||
|
To roll back read/write behavior to Surreal during migration:
|
||||||
|
|
||||||
|
- set `PreferLmdbReads = false`
|
||||||
|
- set `DualWriteOplog = false`
|
||||||
|
|
||||||
|
With `UseLmdbOplog = true`, this keeps LMDB services available while routing reads/writes to Surreal only.
|
||||||
|
If LMDB should be fully disabled, set `UseLmdbOplog = false`.
|
||||||
|
|
||||||
## Feature Comparison
|
## Feature Comparison
|
||||||
|
|
||||||
| Feature | SQLite (Direct) | EF Core | PostgreSQL | Surreal Embedded |
|
| Feature | SQLite (Direct) | EF Core | PostgreSQL | Surreal Embedded |
|
||||||
|
|||||||
307
lmdbop.md
Normal file
307
lmdbop.md
Normal file
@@ -0,0 +1,307 @@
|
|||||||
|
# LMDB Oplog Migration Plan
|
||||||
|
|
||||||
|
## 1. Goal
|
||||||
|
|
||||||
|
Move `IOplogStore` persistence from Surreal-backed oplog tables to an LMDB-backed store while preserving current sync behavior and improving prune efficiency.
|
||||||
|
|
||||||
|
Primary outcomes:
|
||||||
|
- Keep existing `IOplogStore` contract semantics.
|
||||||
|
- Make `PruneOplogAsync` efficient and safe under current timestamp-based cutoff behavior.
|
||||||
|
- Keep roll-forward and rollback low risk via feature flags and verification steps.
|
||||||
|
|
||||||
|
## 2. Current Constraints That Must Be Preserved
|
||||||
|
|
||||||
|
The oplog is not just a queue; the implementation must support:
|
||||||
|
- append + dedupe by hash
|
||||||
|
- lookup by hash
|
||||||
|
- node/time range scans
|
||||||
|
- chain-range reconstruction by hash linkage
|
||||||
|
- prune by cutoff timestamp
|
||||||
|
- per-dataset isolation
|
||||||
|
|
||||||
|
Key references:
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/src/ZB.MOM.WW.CBDDC.Core/Storage/IOplogStore.cs`
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/src/ZB.MOM.WW.CBDDC.Network/SyncOrchestrator.cs`
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/src/ZB.MOM.WW.CBDDC.Network/TcpSyncServer.cs`
|
||||||
|
|
||||||
|
Important behavior notes:
|
||||||
|
- Prune is cutoff-based, not pure FIFO dequeue.
|
||||||
|
- Late-arriving remote entries can have older timestamps than recent local writes, so prune can be non-contiguous in append order.
|
||||||
|
- Current Surreal local write path performs atomic oplog+metadata(+checkpoint) persistence inside one transaction; cross-engine behavior must be handled intentionally.
|
||||||
|
|
||||||
|
## 3. Target Design (LMDB)
|
||||||
|
|
||||||
|
## 3.1 New Provider
|
||||||
|
|
||||||
|
Create an LMDB oplog provider in persistence:
|
||||||
|
- New class: `LmdbOplogStore : OplogStore`
|
||||||
|
- New options class: `LmdbOplogOptions`
|
||||||
|
- New DI extension, e.g. `AddCBDDCLmdbOplog(...)`
|
||||||
|
|
||||||
|
Suggested options:
|
||||||
|
- `EnvironmentPath`
|
||||||
|
- `MapSizeBytes`
|
||||||
|
- `MaxDatabases`
|
||||||
|
- `SyncMode` (durability/perf)
|
||||||
|
- `PruneBatchSize`
|
||||||
|
- `EnableCompactionCopy` (for optional file shrink operation)
|
||||||
|
|
||||||
|
## 3.2 LMDB Data Layout
|
||||||
|
|
||||||
|
Use multiple named DBIs (single environment):
|
||||||
|
|
||||||
|
1. `oplog_by_hash`
|
||||||
|
Key: `{datasetId}|{hash}`
|
||||||
|
Value: serialized `OplogEntry` (compact binary or UTF-8 JSON)
|
||||||
|
|
||||||
|
2. `oplog_by_hlc`
|
||||||
|
Key: `{datasetId}|{wall:big-endian}|{logic:big-endian}|{nodeId}|{hash}`
|
||||||
|
Value: empty or small marker
|
||||||
|
Purpose: `GetOplogAfterAsync`, prune range scan
|
||||||
|
|
||||||
|
3. `oplog_by_node_hlc`
|
||||||
|
Key: `{datasetId}|{nodeId}|{wall}|{logic}|{hash}`
|
||||||
|
Value: empty or marker
|
||||||
|
Purpose: `GetOplogForNodeAfterAsync`, fast node head updates
|
||||||
|
|
||||||
|
4. `oplog_prev_to_hash` (duplicate-allowed)
|
||||||
|
Key: `{datasetId}|{previousHash}`
|
||||||
|
Value: `{hash}`
|
||||||
|
Purpose: chain traversal support for `GetChainRangeAsync`
|
||||||
|
|
||||||
|
5. `oplog_node_head`
|
||||||
|
Key: `{datasetId}|{nodeId}`
|
||||||
|
Value: `{wall, logic, hash}`
|
||||||
|
Purpose: O(1) `GetLastEntryHashAsync`
|
||||||
|
|
||||||
|
6. `oplog_meta`
|
||||||
|
Stores schema version, migration markers, and optional prune watermark per dataset.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
- Use deterministic byte encoding for composite keys to preserve lexical order.
|
||||||
|
- Keep dataset prefix in every index key to guarantee dataset isolation.
|
||||||
|
|
||||||
|
## 3.3 Write Transaction Rules
|
||||||
|
|
||||||
|
`AppendOplogEntryAsync` transaction:
|
||||||
|
1. Check dedupe in `oplog_by_hash`.
|
||||||
|
2. If absent: insert `oplog_by_hash` + all secondary indexes.
|
||||||
|
3. Update `oplog_node_head` only if incoming timestamp > current head timestamp for that node.
|
||||||
|
4. Commit once.
|
||||||
|
|
||||||
|
`MergeAsync`/`ImportAsync`:
|
||||||
|
- Reuse same insert routine in loops with write batching.
|
||||||
|
- Dedupe strictly by hash.
|
||||||
|
|
||||||
|
## 3.4 Prune Strategy
|
||||||
|
|
||||||
|
Base prune operation (must-have):
|
||||||
|
1. Cursor-scan `oplog_by_hlc` up to cutoff key for target dataset.
|
||||||
|
2. For each candidate hash:
|
||||||
|
- delete from `oplog_by_hash`
|
||||||
|
- delete node index key
|
||||||
|
- delete prev->hash duplicate mapping
|
||||||
|
- delete hlc index key
|
||||||
|
3. Recompute affected `oplog_node_head` entries lazily (on read) or eagerly for touched nodes.
|
||||||
|
|
||||||
|
Efficiency enhancements (recommended):
|
||||||
|
- Process deletes in batches (`PruneBatchSize`) inside bounded write txns.
|
||||||
|
- Keep optional per-node dirty set during prune to limit head recomputation.
|
||||||
|
- Optional periodic LMDB compact copy if physical file shrink is needed (LMDB naturally reuses freed pages, but does not always shrink file immediately).
|
||||||
|
|
||||||
|
## 3.5 Atomicity with Document Metadata
|
||||||
|
|
||||||
|
Decision required (explicit in implementation review):
|
||||||
|
|
||||||
|
Option A (phase 1 recommended):
|
||||||
|
- Accept cross-engine eventual atomicity.
|
||||||
|
- Keep current document write flow.
|
||||||
|
- Add reconciliation/repair on startup:
|
||||||
|
- detect metadata entries missing oplog hash for recent writes
|
||||||
|
- rebuild node-head and index consistency from `oplog_by_hash`.
|
||||||
|
|
||||||
|
Option B (hard mode):
|
||||||
|
- Introduce durable outbox pattern to guarantee atomic handoff across engines.
|
||||||
|
- Higher complexity; schedule after functional cutover.
|
||||||
|
|
||||||
|
Plan uses Option A first for lower migration risk.
|
||||||
|
|
||||||
|
## 4. Phased Execution Plan
|
||||||
|
|
||||||
|
## Phase 0: Prep and Design Freeze
|
||||||
|
- Add ADR documenting:
|
||||||
|
- key encoding format
|
||||||
|
- index schema
|
||||||
|
- prune algorithm
|
||||||
|
- consistency model (Option A above)
|
||||||
|
- Add config model and feature flags:
|
||||||
|
- `UseLmdbOplog`
|
||||||
|
- `DualWriteOplog`
|
||||||
|
- `PreferLmdbReads`
|
||||||
|
|
||||||
|
Exit criteria:
|
||||||
|
- ADR approved.
|
||||||
|
- Configuration contract approved.
|
||||||
|
|
||||||
|
## Phase 1: LMDB Store Skeleton
|
||||||
|
- Add package reference `LightningDB`.
|
||||||
|
- Implement `LmdbOplogStore` with:
|
||||||
|
- `AppendOplogEntryAsync`
|
||||||
|
- `GetEntryByHashAsync`
|
||||||
|
- `GetLastEntryHashAsync`
|
||||||
|
- `GetOplogAfterAsync`
|
||||||
|
- `GetOplogForNodeAfterAsync`
|
||||||
|
- `GetChainRangeAsync`
|
||||||
|
- `PruneOplogAsync`
|
||||||
|
- snapshot import/export/drop/merge APIs.
|
||||||
|
- Implement startup/open/close lifecycle and map-size handling.
|
||||||
|
|
||||||
|
Exit criteria:
|
||||||
|
- Local contract tests pass for LMDB store.
|
||||||
|
|
||||||
|
## Phase 2: Dual-Write + Read Shadow Validation
|
||||||
|
- Keep Surreal oplog as source of truth.
|
||||||
|
- Write every oplog mutation to both stores (`DualWriteOplog=true`).
|
||||||
|
- Read-path comparison mode in non-prod:
|
||||||
|
- query both stores
|
||||||
|
- assert same hashes/order for key APIs
|
||||||
|
- log mismatches.
|
||||||
|
|
||||||
|
Exit criteria:
|
||||||
|
- Zero mismatches in soak tests.
|
||||||
|
|
||||||
|
## Phase 3: Cutover
|
||||||
|
- Set `PreferLmdbReads=true` in staging first.
|
||||||
|
- Keep dual-write enabled for one release window.
|
||||||
|
- Monitor:
|
||||||
|
- prune duration
|
||||||
|
- oplog query latency
|
||||||
|
- mismatch counters
|
||||||
|
- restart recovery behavior.
|
||||||
|
|
||||||
|
Exit criteria:
|
||||||
|
- Stable staging and production canary.
|
||||||
|
|
||||||
|
## Phase 4: Cleanup
|
||||||
|
- Disable Surreal oplog writes.
|
||||||
|
- Keep migration utility for rollback window.
|
||||||
|
- Remove dual-compare instrumentation after confidence period.
|
||||||
|
|
||||||
|
## 5. Data Migration / Backfill
|
||||||
|
|
||||||
|
Backfill tool steps:
|
||||||
|
1. Read dataset-scoped Surreal oplog export.
|
||||||
|
2. Bulk import into LMDB by HLC order.
|
||||||
|
3. Rebuild node-head table.
|
||||||
|
4. Validate:
|
||||||
|
- counts per dataset
|
||||||
|
- counts per node
|
||||||
|
- latest hash per node
|
||||||
|
- random hash spot checks
|
||||||
|
- chain-range spot checks.
|
||||||
|
|
||||||
|
Rollback:
|
||||||
|
- Keep Surreal oplog untouched during dual-write window.
|
||||||
|
- Flip feature flags back to Surreal reads.
|
||||||
|
|
||||||
|
## 6. Unit Test Update Instructions
|
||||||
|
|
||||||
|
## 6.1 Reuse Existing Oplog Contract Tests
|
||||||
|
|
||||||
|
Use these as baseline parity requirements:
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.Sample.Console.Tests/SurrealStoreContractTests.cs`
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.Sample.Console.Tests/BLiteStoreExportImportTests.cs` (class `SurrealStoreExportImportTests`)
|
||||||
|
|
||||||
|
Actions:
|
||||||
|
1. Extract oplog contract cases into shared test base (provider-agnostic).
|
||||||
|
2. Run same suite against:
|
||||||
|
- Surreal store
|
||||||
|
- new LMDB store
|
||||||
|
|
||||||
|
Minimum parity cases:
|
||||||
|
- append/query/merge/drop
|
||||||
|
- dataset isolation
|
||||||
|
- legacy/default dataset behavior (if supported)
|
||||||
|
- `GetChainRangeAsync` correctness
|
||||||
|
- `GetLastEntryHashAsync` persistence across restart
|
||||||
|
|
||||||
|
## 6.2 Add LMDB-Specific Unit Tests
|
||||||
|
|
||||||
|
Create new file:
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.Sample.Console.Tests/LmdbOplogStoreContractTests.cs`
|
||||||
|
|
||||||
|
Add tests for:
|
||||||
|
1. Index consistency:
|
||||||
|
- inserting one entry populates all indexes
|
||||||
|
- deleting/pruning removes all index records
|
||||||
|
2. Prune correctness:
|
||||||
|
- removes `<= cutoff`
|
||||||
|
- does not remove `> cutoff`
|
||||||
|
- handles interleaved node timestamps
|
||||||
|
- handles late-arriving old timestamp entry safely
|
||||||
|
3. Node-head maintenance:
|
||||||
|
- head advances on newer entry
|
||||||
|
- prune invalidates/recomputes correctly
|
||||||
|
4. Restart durability:
|
||||||
|
- reopen LMDB env and verify last-hash + scans
|
||||||
|
5. Dedupe:
|
||||||
|
- duplicate hash append is idempotent
|
||||||
|
|
||||||
|
## 6.3 Update Integration/E2E Coverage
|
||||||
|
|
||||||
|
Files to touch:
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.E2E.Tests/ClusterCrudSyncE2ETests.cs`
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.Sample.Console.Tests/SurrealCdcDurabilityTests.cs` (or LMDB-focused equivalent)
|
||||||
|
|
||||||
|
Add/adjust scenarios:
|
||||||
|
1. Gap recovery still works with LMDB oplog backend.
|
||||||
|
2. Peer-confirmed prune still blocks/allows correctly.
|
||||||
|
3. Crash between document commit and oplog write (Option A behavior) is detected/repaired by startup reconciliation.
|
||||||
|
4. Prune performance smoke test (large synthetic oplog, bounded runtime threshold with generous margin).
|
||||||
|
|
||||||
|
## 6.4 Keep Existing Network Unit Tests Intact
|
||||||
|
|
||||||
|
Most network tests mock `IOplogStore` and should remain unchanged:
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.Network.Tests/SyncOrchestratorMaintenancePruningTests.cs`
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.Network.Tests/SyncOrchestratorConfirmationTests.cs`
|
||||||
|
- `/Users/dohertj2/Desktop/CBDDC/tests/ZB.MOM.WW.CBDDC.Network.Tests/SnapshotReconnectRegressionTests.cs`
|
||||||
|
|
||||||
|
Only update if method behavior/ordering contracts are intentionally changed.
|
||||||
|
|
||||||
|
## 7. Performance and Observability Plan
|
||||||
|
|
||||||
|
Track and compare (Surreal vs LMDB):
|
||||||
|
- `AppendOplogEntryAsync` latency p50/p95/p99
|
||||||
|
- `GetOplogForNodeAfterAsync` latency
|
||||||
|
- prune duration and entries/sec deleted
|
||||||
|
- LMDB env file size and reclaimed free-page ratio
|
||||||
|
- mismatch counters in dual-read compare mode
|
||||||
|
|
||||||
|
Add logs/metrics:
|
||||||
|
- prune batches processed
|
||||||
|
- dirty nodes recomputed
|
||||||
|
- startup repair actions and counts
|
||||||
|
|
||||||
|
## 8. Risks and Mitigations
|
||||||
|
|
||||||
|
1. Cross-engine consistency gaps (document metadata vs oplog)
|
||||||
|
- Mitigation: startup reconciliation + dual-write shadow period.
|
||||||
|
|
||||||
|
2. Incorrect composite key encoding
|
||||||
|
- Mitigation: explicit encoding helper + property tests for sort/order invariants.
|
||||||
|
|
||||||
|
3. Prune causing stale node-head values
|
||||||
|
- Mitigation: touched-node tracking and lazy/eager recompute tests.
|
||||||
|
|
||||||
|
4. LMDB map-size exhaustion
|
||||||
|
- Mitigation: configurable mapsize, monitoring, and operational runbook for resize.
|
||||||
|
|
||||||
|
## 9. Review Checklist
|
||||||
|
|
||||||
|
- [x] ADR approved for LMDB key/index schema.
|
||||||
|
- [x] Feature flags merged (`UseLmdbOplog`, `DualWriteOplog`, `PreferLmdbReads`).
|
||||||
|
- [x] LMDB contract tests passing.
|
||||||
|
- [x] Dual-write mismatch telemetry in place.
|
||||||
|
- [x] Backfill tool implemented and validated in automated tests (staging execution ready).
|
||||||
|
- [x] Prune correctness + efficiency tests passing.
|
||||||
|
- [x] Rollback path documented and tested.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Storage;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Surreal;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Extension methods for adding the LMDB oplog provider and migration feature flags.
|
||||||
|
/// </summary>
|
||||||
|
public static class CBDDCLmdbOplogExtensions
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Registers LMDB oplog services and replaces <see cref="IOplogStore" /> with a feature-flag migration router.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="services">The service collection.</param>
|
||||||
|
/// <param name="optionsFactory">Factory creating LMDB environment options.</param>
|
||||||
|
/// <param name="configureFlags">Optional migration feature-flag configuration.</param>
|
||||||
|
/// <returns>The service collection.</returns>
|
||||||
|
public static IServiceCollection AddCBDDCLmdbOplog(
|
||||||
|
this IServiceCollection services,
|
||||||
|
Func<IServiceProvider, LmdbOplogOptions> optionsFactory,
|
||||||
|
Action<LmdbOplogFeatureFlags>? configureFlags = null)
|
||||||
|
{
|
||||||
|
if (services == null) throw new ArgumentNullException(nameof(services));
|
||||||
|
if (optionsFactory == null) throw new ArgumentNullException(nameof(optionsFactory));
|
||||||
|
|
||||||
|
services.TryAddSingleton(optionsFactory);
|
||||||
|
|
||||||
|
var flags = new LmdbOplogFeatureFlags
|
||||||
|
{
|
||||||
|
UseLmdbOplog = true,
|
||||||
|
DualWriteOplog = true,
|
||||||
|
PreferLmdbReads = false,
|
||||||
|
EnableReadShadowValidation = false
|
||||||
|
};
|
||||||
|
configureFlags?.Invoke(flags);
|
||||||
|
services.TryAddSingleton(flags);
|
||||||
|
|
||||||
|
services.TryAddSingleton<OplogMigrationTelemetry>();
|
||||||
|
services.TryAddSingleton<LmdbOplogStore>();
|
||||||
|
|
||||||
|
bool surrealRegistered = services.Any(descriptor => descriptor.ServiceType == typeof(SurrealOplogStore));
|
||||||
|
if (surrealRegistered)
|
||||||
|
{
|
||||||
|
services.TryAddSingleton<LmdbOplogBackfillTool>();
|
||||||
|
services.Replace(ServiceDescriptor.Singleton<IOplogStore, FeatureFlagOplogStore>());
|
||||||
|
}
|
||||||
|
else
|
||||||
|
services.Replace(ServiceDescriptor.Singleton<IOplogStore>(sp => sp.GetRequiredService<LmdbOplogStore>()));
|
||||||
|
|
||||||
|
return services;
|
||||||
|
}
|
||||||
|
}
|
||||||
497
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/FeatureFlagOplogStore.cs
Normal file
497
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/FeatureFlagOplogStore.cs
Normal file
@@ -0,0 +1,497 @@
|
|||||||
|
using System.Collections.Concurrent;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Storage;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Surreal;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Feature-flag controlled oplog router supporting Surreal source-of-truth, LMDB dual-write,
|
||||||
|
/// cutover reads, and shadow validation.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class FeatureFlagOplogStore : IOplogStore
|
||||||
|
{
|
||||||
|
private readonly SemaphoreSlim _reconcileGate = new(1, 1);
|
||||||
|
private readonly ConcurrentDictionary<string, DateTimeOffset> _reconcileWatermarks = new(StringComparer.Ordinal);
|
||||||
|
private readonly LmdbOplogFeatureFlags _flags;
|
||||||
|
private readonly LmdbOplogStore _lmdb;
|
||||||
|
private readonly ILogger<FeatureFlagOplogStore> _logger;
|
||||||
|
private readonly OplogMigrationTelemetry _telemetry;
|
||||||
|
private readonly SurrealOplogStore _surreal;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Initializes a new instance of the <see cref="FeatureFlagOplogStore" /> class.
|
||||||
|
/// </summary>
|
||||||
|
public FeatureFlagOplogStore(
|
||||||
|
SurrealOplogStore surreal,
|
||||||
|
LmdbOplogStore lmdb,
|
||||||
|
LmdbOplogFeatureFlags flags,
|
||||||
|
OplogMigrationTelemetry? telemetry = null,
|
||||||
|
ILogger<FeatureFlagOplogStore>? logger = null)
|
||||||
|
{
|
||||||
|
_surreal = surreal ?? throw new ArgumentNullException(nameof(surreal));
|
||||||
|
_lmdb = lmdb ?? throw new ArgumentNullException(nameof(lmdb));
|
||||||
|
_flags = flags ?? throw new ArgumentNullException(nameof(flags));
|
||||||
|
_telemetry = telemetry ?? new OplogMigrationTelemetry();
|
||||||
|
_logger = logger ?? NullLogger<FeatureFlagOplogStore>.Instance;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public event EventHandler<ChangesAppliedEventArgs>? ChangesApplied;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task AppendOplogEntryAsync(OplogEntry entry, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
await WriteAsync(
|
||||||
|
s => s.AppendOplogEntryAsync(entry, cancellationToken),
|
||||||
|
l => l.AppendOplogEntryAsync(entry, cancellationToken));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task AppendOplogEntryAsync(
|
||||||
|
OplogEntry entry,
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
await WriteAsync(
|
||||||
|
s => s.AppendOplogEntryAsync(entry, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.AppendOplogEntryAsync(entry, normalizedDatasetId, cancellationToken));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> GetOplogAfterAsync(
|
||||||
|
HlcTimestamp timestamp,
|
||||||
|
IEnumerable<string>? collections = null,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return GetOplogAfterAsync(timestamp, DatasetId.Primary, collections, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> GetOplogAfterAsync(
|
||||||
|
HlcTimestamp timestamp,
|
||||||
|
string datasetId,
|
||||||
|
IEnumerable<string>? collections = null,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.GetOplogAfterAsync(timestamp, normalizedDatasetId, collections, cancellationToken),
|
||||||
|
l => l.GetOplogAfterAsync(timestamp, normalizedDatasetId, collections, cancellationToken),
|
||||||
|
SequencesEquivalent,
|
||||||
|
"GetOplogAfterAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<HlcTimestamp> GetLatestTimestampAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return GetLatestTimestampAsync(DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<HlcTimestamp> GetLatestTimestampAsync(string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.GetLatestTimestampAsync(normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.GetLatestTimestampAsync(normalizedDatasetId, cancellationToken),
|
||||||
|
(a, b) => a.Equals(b),
|
||||||
|
"GetLatestTimestampAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<VectorClock> GetVectorClockAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return GetVectorClockAsync(DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<VectorClock> GetVectorClockAsync(string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.GetVectorClockAsync(normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.GetVectorClockAsync(normalizedDatasetId, cancellationToken),
|
||||||
|
VectorClocksEquivalent,
|
||||||
|
"GetVectorClockAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> GetOplogForNodeAfterAsync(
|
||||||
|
string nodeId,
|
||||||
|
HlcTimestamp since,
|
||||||
|
IEnumerable<string>? collections = null,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return GetOplogForNodeAfterAsync(nodeId, since, DatasetId.Primary, collections, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> GetOplogForNodeAfterAsync(
|
||||||
|
string nodeId,
|
||||||
|
HlcTimestamp since,
|
||||||
|
string datasetId,
|
||||||
|
IEnumerable<string>? collections = null,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.GetOplogForNodeAfterAsync(nodeId, since, normalizedDatasetId, collections, cancellationToken),
|
||||||
|
l => l.GetOplogForNodeAfterAsync(nodeId, since, normalizedDatasetId, collections, cancellationToken),
|
||||||
|
SequencesEquivalent,
|
||||||
|
"GetOplogForNodeAfterAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<string?> GetLastEntryHashAsync(string nodeId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return GetLastEntryHashAsync(nodeId, DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<string?> GetLastEntryHashAsync(
|
||||||
|
string nodeId,
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.GetLastEntryHashAsync(nodeId, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.GetLastEntryHashAsync(nodeId, normalizedDatasetId, cancellationToken),
|
||||||
|
(a, b) => string.Equals(a, b, StringComparison.Ordinal),
|
||||||
|
"GetLastEntryHashAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> GetChainRangeAsync(
|
||||||
|
string startHash,
|
||||||
|
string endHash,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return GetChainRangeAsync(startHash, endHash, DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> GetChainRangeAsync(
|
||||||
|
string startHash,
|
||||||
|
string endHash,
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.GetChainRangeAsync(startHash, endHash, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.GetChainRangeAsync(startHash, endHash, normalizedDatasetId, cancellationToken),
|
||||||
|
SequencesEquivalent,
|
||||||
|
"GetChainRangeAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<OplogEntry?> GetEntryByHashAsync(string hash, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return GetEntryByHashAsync(hash, DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<OplogEntry?> GetEntryByHashAsync(
|
||||||
|
string hash,
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.GetEntryByHashAsync(hash, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.GetEntryByHashAsync(hash, normalizedDatasetId, cancellationToken),
|
||||||
|
EntriesEquivalent,
|
||||||
|
"GetEntryByHashAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task ApplyBatchAsync(IEnumerable<OplogEntry> oplogEntries, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
var entries = oplogEntries.ToList();
|
||||||
|
await WriteAsync(
|
||||||
|
s => s.ApplyBatchAsync(entries, cancellationToken),
|
||||||
|
l => l.ApplyBatchAsync(entries, cancellationToken));
|
||||||
|
|
||||||
|
ChangesApplied?.Invoke(this, new ChangesAppliedEventArgs(entries));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task ApplyBatchAsync(
|
||||||
|
IEnumerable<OplogEntry> oplogEntries,
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
var entries = oplogEntries.ToList();
|
||||||
|
|
||||||
|
await WriteAsync(
|
||||||
|
s => s.ApplyBatchAsync(entries, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.ApplyBatchAsync(entries, normalizedDatasetId, cancellationToken));
|
||||||
|
|
||||||
|
ChangesApplied?.Invoke(this, new ChangesAppliedEventArgs(entries));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task PruneOplogAsync(HlcTimestamp cutoff, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return PruneOplogAsync(cutoff, DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task PruneOplogAsync(HlcTimestamp cutoff, string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return WriteAsync(
|
||||||
|
s => s.PruneOplogAsync(cutoff, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.PruneOplogAsync(cutoff, normalizedDatasetId, cancellationToken));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task DropAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return DropAsync(DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task DropAsync(string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return WriteAsync(
|
||||||
|
s => s.DropAsync(normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.DropAsync(normalizedDatasetId, cancellationToken));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> ExportAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return ExportAsync(DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<IEnumerable<OplogEntry>> ExportAsync(string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
return ReadWithOptionalShadowCompareAsync(
|
||||||
|
normalizedDatasetId,
|
||||||
|
s => s.ExportAsync(normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.ExportAsync(normalizedDatasetId, cancellationToken),
|
||||||
|
SequencesEquivalent,
|
||||||
|
"ExportAsync");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task ImportAsync(IEnumerable<OplogEntry> items, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return ImportAsync(items, DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task ImportAsync(
|
||||||
|
IEnumerable<OplogEntry> items,
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
var entries = items.ToList();
|
||||||
|
return WriteAsync(
|
||||||
|
s => s.ImportAsync(entries, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.ImportAsync(entries, normalizedDatasetId, cancellationToken));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task MergeAsync(IEnumerable<OplogEntry> items, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return MergeAsync(items, DatasetId.Primary, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task MergeAsync(
|
||||||
|
IEnumerable<OplogEntry> items,
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
var entries = items.ToList();
|
||||||
|
return WriteAsync(
|
||||||
|
s => s.MergeAsync(entries, normalizedDatasetId, cancellationToken),
|
||||||
|
l => l.MergeAsync(entries, normalizedDatasetId, cancellationToken));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Returns current migration telemetry counters.
|
||||||
|
/// </summary>
|
||||||
|
public OplogMigrationTelemetrySnapshot GetTelemetrySnapshot()
|
||||||
|
{
|
||||||
|
return _telemetry.GetSnapshot();
|
||||||
|
}
|
||||||
|
|
||||||
|
private bool IsLmdbEnabled => _flags.UseLmdbOplog;
|
||||||
|
|
||||||
|
private bool ShouldShadowCompare =>
|
||||||
|
_flags.UseLmdbOplog && _flags.DualWriteOplog && _flags.EnableReadShadowValidation;
|
||||||
|
|
||||||
|
private bool PreferLmdbReads => _flags.UseLmdbOplog && _flags.PreferLmdbReads;
|
||||||
|
|
||||||
|
private static string NormalizeDatasetId(string? datasetId)
|
||||||
|
{
|
||||||
|
return DatasetId.Normalize(datasetId);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<T> ReadWithOptionalShadowCompareAsync<T>(
|
||||||
|
string datasetId,
|
||||||
|
Func<SurrealOplogStore, Task<T>> surrealRead,
|
||||||
|
Func<LmdbOplogStore, Task<T>> lmdbRead,
|
||||||
|
Func<T, T, bool> equals,
|
||||||
|
string operationName)
|
||||||
|
{
|
||||||
|
bool preferLmdb = PreferLmdbReads;
|
||||||
|
|
||||||
|
T primary;
|
||||||
|
if (preferLmdb)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await EnsureReconciledAsync(datasetId);
|
||||||
|
primary = await lmdbRead(_lmdb);
|
||||||
|
}
|
||||||
|
catch (Exception exception)
|
||||||
|
{
|
||||||
|
_telemetry.RecordPreferredReadFallback();
|
||||||
|
_logger.LogWarning(
|
||||||
|
exception,
|
||||||
|
"LMDB preferred read fallback to Surreal for {Operation} dataset {DatasetId}.",
|
||||||
|
operationName,
|
||||||
|
datasetId);
|
||||||
|
primary = await surrealRead(_surreal);
|
||||||
|
preferLmdb = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
primary = await surrealRead(_surreal);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!ShouldShadowCompare) return primary;
|
||||||
|
|
||||||
|
T secondary;
|
||||||
|
if (preferLmdb)
|
||||||
|
{
|
||||||
|
secondary = await surrealRead(_surreal);
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
await EnsureReconciledAsync(datasetId);
|
||||||
|
secondary = await lmdbRead(_lmdb);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool isMatch = equals(primary, secondary);
|
||||||
|
_telemetry.RecordShadowComparison(isMatch);
|
||||||
|
|
||||||
|
if (!isMatch)
|
||||||
|
_logger.LogWarning(
|
||||||
|
"Oplog read shadow mismatch in {Operation} for dataset {DatasetId}. PreferLmdbReads={PreferLmdbReads}",
|
||||||
|
operationName,
|
||||||
|
datasetId,
|
||||||
|
PreferLmdbReads);
|
||||||
|
|
||||||
|
return primary;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task WriteAsync(
|
||||||
|
Func<SurrealOplogStore, Task> surrealWrite,
|
||||||
|
Func<LmdbOplogStore, Task> lmdbWrite)
|
||||||
|
{
|
||||||
|
if (!IsLmdbEnabled)
|
||||||
|
{
|
||||||
|
await surrealWrite(_surreal);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (_flags.DualWriteOplog)
|
||||||
|
{
|
||||||
|
await surrealWrite(_surreal);
|
||||||
|
await lmdbWrite(_lmdb);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (_flags.PreferLmdbReads)
|
||||||
|
await lmdbWrite(_lmdb);
|
||||||
|
else
|
||||||
|
await surrealWrite(_surreal);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task EnsureReconciledAsync(string datasetId)
|
||||||
|
{
|
||||||
|
if (!PreferLmdbReads) return;
|
||||||
|
|
||||||
|
string normalizedDatasetId = NormalizeDatasetId(datasetId);
|
||||||
|
DateTimeOffset now = DateTimeOffset.UtcNow;
|
||||||
|
|
||||||
|
if (_reconcileWatermarks.TryGetValue(normalizedDatasetId, out DateTimeOffset watermark) &&
|
||||||
|
now - watermark < _flags.ReconciliationInterval)
|
||||||
|
return;
|
||||||
|
|
||||||
|
await _reconcileGate.WaitAsync();
|
||||||
|
try
|
||||||
|
{
|
||||||
|
now = DateTimeOffset.UtcNow;
|
||||||
|
if (_reconcileWatermarks.TryGetValue(normalizedDatasetId, out watermark) &&
|
||||||
|
now - watermark < _flags.ReconciliationInterval)
|
||||||
|
return;
|
||||||
|
|
||||||
|
var source = (await _surreal.ExportAsync(normalizedDatasetId)).ToList();
|
||||||
|
await _lmdb.MergeAsync(source, normalizedDatasetId);
|
||||||
|
_telemetry.RecordReconciliation(normalizedDatasetId, source.Count);
|
||||||
|
_reconcileWatermarks[normalizedDatasetId] = DateTimeOffset.UtcNow;
|
||||||
|
|
||||||
|
_logger.LogInformation(
|
||||||
|
"Reconciled LMDB oplog for dataset {DatasetId} by merging {MergedCount} Surreal entries.",
|
||||||
|
normalizedDatasetId,
|
||||||
|
source.Count);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_reconcileGate.Release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool EntriesEquivalent(OplogEntry? left, OplogEntry? right)
|
||||||
|
{
|
||||||
|
if (left is null && right is null) return true;
|
||||||
|
if (left is null || right is null) return false;
|
||||||
|
|
||||||
|
return string.Equals(left.Hash, right.Hash, StringComparison.Ordinal) &&
|
||||||
|
string.Equals(left.DatasetId, right.DatasetId, StringComparison.Ordinal);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool SequencesEquivalent(IEnumerable<OplogEntry> left, IEnumerable<OplogEntry> right)
|
||||||
|
{
|
||||||
|
string[] leftHashes = left.Select(e => e.Hash).ToArray();
|
||||||
|
string[] rightHashes = right.Select(e => e.Hash).ToArray();
|
||||||
|
return leftHashes.SequenceEqual(rightHashes, StringComparer.Ordinal);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool VectorClocksEquivalent(VectorClock left, VectorClock right)
|
||||||
|
{
|
||||||
|
var nodeIds = new HashSet<string>(left.NodeIds, StringComparer.Ordinal);
|
||||||
|
foreach (string nodeId in right.NodeIds) nodeIds.Add(nodeId);
|
||||||
|
|
||||||
|
foreach (string nodeId in nodeIds)
|
||||||
|
if (!left.GetTimestamp(nodeId).Equals(right.GetTimestamp(nodeId)))
|
||||||
|
return false;
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
277
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/LmdbOplogBackfillTool.cs
Normal file
277
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/LmdbOplogBackfillTool.cs
Normal file
@@ -0,0 +1,277 @@
|
|||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Storage;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Surreal;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Backfills LMDB oplog content from Surreal and validates parity.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class LmdbOplogBackfillTool
|
||||||
|
{
|
||||||
|
private readonly LmdbOplogStore _destination;
|
||||||
|
private readonly ILogger<LmdbOplogBackfillTool> _logger;
|
||||||
|
private readonly SurrealOplogStore _source;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Initializes a new instance of the <see cref="LmdbOplogBackfillTool" /> class.
|
||||||
|
/// </summary>
|
||||||
|
public LmdbOplogBackfillTool(
|
||||||
|
SurrealOplogStore source,
|
||||||
|
LmdbOplogStore destination,
|
||||||
|
ILogger<LmdbOplogBackfillTool>? logger = null)
|
||||||
|
{
|
||||||
|
_source = source ?? throw new ArgumentNullException(nameof(source));
|
||||||
|
_destination = destination ?? throw new ArgumentNullException(nameof(destination));
|
||||||
|
_logger = logger ?? NullLogger<LmdbOplogBackfillTool>.Instance;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Backfills one dataset from Surreal to LMDB and validates parity.
|
||||||
|
/// </summary>
|
||||||
|
public async Task<LmdbOplogBackfillReport> BackfillAsync(
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = DatasetId.Normalize(datasetId);
|
||||||
|
var sourceEntries = (await _source.ExportAsync(normalizedDatasetId, cancellationToken))
|
||||||
|
.OrderBy(entry => entry.Timestamp.PhysicalTime)
|
||||||
|
.ThenBy(entry => entry.Timestamp.LogicalCounter)
|
||||||
|
.ThenBy(entry => entry.Timestamp.NodeId, StringComparer.Ordinal)
|
||||||
|
.ThenBy(entry => entry.Hash, StringComparer.Ordinal)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
await _destination.MergeAsync(sourceEntries, normalizedDatasetId, cancellationToken);
|
||||||
|
LmdbOplogBackfillReport report = await ValidateParityAsync(normalizedDatasetId, sourceEntries, cancellationToken);
|
||||||
|
|
||||||
|
_logger.LogInformation(
|
||||||
|
"LMDB oplog backfill {Result} for dataset {DatasetId}. Source={SourceCount}, Destination={DestinationCount}, HashSpotChecks={HashSpotChecks}, ChainSpotChecks={ChainSpotChecks}.",
|
||||||
|
report.IsSuccess ? "succeeded" : "failed",
|
||||||
|
report.DatasetId,
|
||||||
|
report.SourceCount,
|
||||||
|
report.DestinationCount,
|
||||||
|
report.HashSpotCheckCount,
|
||||||
|
report.ChainSpotCheckCount);
|
||||||
|
|
||||||
|
return report;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Validates parity only without running a backfill merge.
|
||||||
|
/// </summary>
|
||||||
|
public async Task<LmdbOplogBackfillReport> ValidateParityAsync(
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = DatasetId.Normalize(datasetId);
|
||||||
|
var sourceEntries = (await _source.ExportAsync(normalizedDatasetId, cancellationToken)).ToList();
|
||||||
|
return await ValidateParityAsync(normalizedDatasetId, sourceEntries, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Backfills and throws when parity validation fails.
|
||||||
|
/// </summary>
|
||||||
|
public async Task<LmdbOplogBackfillReport> BackfillOrThrowAsync(
|
||||||
|
string datasetId,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
LmdbOplogBackfillReport report = await BackfillAsync(datasetId, cancellationToken);
|
||||||
|
if (report.IsSuccess) return report;
|
||||||
|
|
||||||
|
throw new InvalidOperationException(
|
||||||
|
$"LMDB oplog backfill parity failed for dataset '{report.DatasetId}'. " +
|
||||||
|
$"Source={report.SourceCount}, Destination={report.DestinationCount}, " +
|
||||||
|
$"CountsMatch={report.CountsMatch}, CountsPerNodeMatch={report.CountsPerNodeMatch}, " +
|
||||||
|
$"LatestHashPerNodeMatch={report.LatestHashPerNodeMatch}, HashSpotChecksPassed={report.HashSpotChecksPassed}, " +
|
||||||
|
$"ChainSpotChecksPassed={report.ChainSpotChecksPassed}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<LmdbOplogBackfillReport> ValidateParityAsync(
|
||||||
|
string datasetId,
|
||||||
|
List<OplogEntry> sourceEntries,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var sourceOrdered = sourceEntries
|
||||||
|
.OrderBy(entry => entry.Timestamp.PhysicalTime)
|
||||||
|
.ThenBy(entry => entry.Timestamp.LogicalCounter)
|
||||||
|
.ThenBy(entry => entry.Timestamp.NodeId, StringComparer.Ordinal)
|
||||||
|
.ThenBy(entry => entry.Hash, StringComparer.Ordinal)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
var destinationOrdered = (await _destination.ExportAsync(datasetId, cancellationToken))
|
||||||
|
.OrderBy(entry => entry.Timestamp.PhysicalTime)
|
||||||
|
.ThenBy(entry => entry.Timestamp.LogicalCounter)
|
||||||
|
.ThenBy(entry => entry.Timestamp.NodeId, StringComparer.Ordinal)
|
||||||
|
.ThenBy(entry => entry.Hash, StringComparer.Ordinal)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
bool countsMatch = sourceOrdered.Count == destinationOrdered.Count;
|
||||||
|
IReadOnlyDictionary<string, int> sourceCountByNode = CountByNode(sourceOrdered);
|
||||||
|
IReadOnlyDictionary<string, int> destinationCountByNode = CountByNode(destinationOrdered);
|
||||||
|
bool countsPerNodeMatch = DictionaryEqual(sourceCountByNode, destinationCountByNode);
|
||||||
|
|
||||||
|
IReadOnlyDictionary<string, string> sourceLatestHashByNode = LatestHashByNode(sourceOrdered);
|
||||||
|
IReadOnlyDictionary<string, string> destinationLatestHashByNode = LatestHashByNode(destinationOrdered);
|
||||||
|
bool latestHashPerNodeMatch = DictionaryEqual(sourceLatestHashByNode, destinationLatestHashByNode);
|
||||||
|
|
||||||
|
(bool hashSpotChecksPassed, int hashSpotCheckCount) = await RunHashSpotChecksAsync(
|
||||||
|
datasetId,
|
||||||
|
sourceOrdered,
|
||||||
|
cancellationToken);
|
||||||
|
(bool chainSpotChecksPassed, int chainSpotCheckCount) = await RunChainSpotChecksAsync(
|
||||||
|
datasetId,
|
||||||
|
sourceOrdered,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
|
return new LmdbOplogBackfillReport(
|
||||||
|
datasetId,
|
||||||
|
sourceOrdered.Count,
|
||||||
|
destinationOrdered.Count,
|
||||||
|
sourceCountByNode,
|
||||||
|
destinationCountByNode,
|
||||||
|
sourceLatestHashByNode,
|
||||||
|
destinationLatestHashByNode,
|
||||||
|
hashSpotCheckCount,
|
||||||
|
chainSpotCheckCount,
|
||||||
|
countsMatch,
|
||||||
|
countsPerNodeMatch,
|
||||||
|
latestHashPerNodeMatch,
|
||||||
|
hashSpotChecksPassed,
|
||||||
|
chainSpotChecksPassed);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<(bool Passed, int Count)> RunHashSpotChecksAsync(
|
||||||
|
string datasetId,
|
||||||
|
IReadOnlyList<OplogEntry> sourceEntries,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
if (sourceEntries.Count == 0) return (true, 0);
|
||||||
|
|
||||||
|
var sampleIndexes = BuildSampleIndexes(sourceEntries.Count, Math.Min(10, sourceEntries.Count));
|
||||||
|
foreach (int index in sampleIndexes)
|
||||||
|
{
|
||||||
|
string hash = sourceEntries[index].Hash;
|
||||||
|
OplogEntry? destinationEntry = await _destination.GetEntryByHashAsync(hash, datasetId, cancellationToken);
|
||||||
|
if (destinationEntry == null) return (false, sampleIndexes.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (true, sampleIndexes.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<(bool Passed, int Count)> RunChainSpotChecksAsync(
|
||||||
|
string datasetId,
|
||||||
|
IReadOnlyList<OplogEntry> sourceEntries,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
if (sourceEntries.Count < 2) return (true, 0);
|
||||||
|
|
||||||
|
var sourceByHash = sourceEntries.ToDictionary(entry => entry.Hash, StringComparer.Ordinal);
|
||||||
|
var checks = sourceEntries
|
||||||
|
.Where(entry => !string.IsNullOrWhiteSpace(entry.PreviousHash) &&
|
||||||
|
sourceByHash.ContainsKey(entry.PreviousHash))
|
||||||
|
.Take(5)
|
||||||
|
.Select(entry => (StartHash: entry.PreviousHash, EndHash: entry.Hash))
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
foreach (var check in checks)
|
||||||
|
{
|
||||||
|
string[] sourceChain = (await _source.GetChainRangeAsync(check.StartHash, check.EndHash, datasetId, cancellationToken))
|
||||||
|
.Select(entry => entry.Hash)
|
||||||
|
.ToArray();
|
||||||
|
string[] destinationChain =
|
||||||
|
(await _destination.GetChainRangeAsync(check.StartHash, check.EndHash, datasetId, cancellationToken))
|
||||||
|
.Select(entry => entry.Hash)
|
||||||
|
.ToArray();
|
||||||
|
|
||||||
|
if (!sourceChain.SequenceEqual(destinationChain, StringComparer.Ordinal))
|
||||||
|
return (false, checks.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (true, checks.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyDictionary<string, int> CountByNode(IEnumerable<OplogEntry> entries)
|
||||||
|
{
|
||||||
|
return entries
|
||||||
|
.Where(entry => !string.IsNullOrWhiteSpace(entry.Timestamp.NodeId))
|
||||||
|
.GroupBy(entry => entry.Timestamp.NodeId, StringComparer.Ordinal)
|
||||||
|
.ToDictionary(group => group.Key, group => group.Count(), StringComparer.Ordinal);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyDictionary<string, string> LatestHashByNode(IEnumerable<OplogEntry> entries)
|
||||||
|
{
|
||||||
|
return entries
|
||||||
|
.Where(entry => !string.IsNullOrWhiteSpace(entry.Timestamp.NodeId))
|
||||||
|
.GroupBy(entry => entry.Timestamp.NodeId, StringComparer.Ordinal)
|
||||||
|
.ToDictionary(
|
||||||
|
group => group.Key,
|
||||||
|
group => group
|
||||||
|
.OrderByDescending(entry => entry.Timestamp.PhysicalTime)
|
||||||
|
.ThenByDescending(entry => entry.Timestamp.LogicalCounter)
|
||||||
|
.ThenByDescending(entry => entry.Hash, StringComparer.Ordinal)
|
||||||
|
.First()
|
||||||
|
.Hash,
|
||||||
|
StringComparer.Ordinal);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool DictionaryEqual<T>(
|
||||||
|
IReadOnlyDictionary<string, T> left,
|
||||||
|
IReadOnlyDictionary<string, T> right)
|
||||||
|
{
|
||||||
|
if (left.Count != right.Count) return false;
|
||||||
|
foreach (var pair in left)
|
||||||
|
{
|
||||||
|
if (!right.TryGetValue(pair.Key, out T? rightValue)) return false;
|
||||||
|
if (!EqualityComparer<T>.Default.Equals(pair.Value, rightValue)) return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static List<int> BuildSampleIndexes(int totalCount, int sampleCount)
|
||||||
|
{
|
||||||
|
if (sampleCount <= 0 || totalCount <= 0) return [];
|
||||||
|
if (sampleCount >= totalCount) return Enumerable.Range(0, totalCount).ToList();
|
||||||
|
|
||||||
|
var indexes = new HashSet<int>();
|
||||||
|
for (var i = 0; i < sampleCount; i++)
|
||||||
|
{
|
||||||
|
int index = (int)Math.Round(i * (totalCount - 1d) / (sampleCount - 1d));
|
||||||
|
indexes.Add(Math.Clamp(index, 0, totalCount - 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
return indexes.OrderBy(value => value).ToList();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Parity report produced by the LMDB backfill tool.
|
||||||
|
/// </summary>
|
||||||
|
public sealed record LmdbOplogBackfillReport(
|
||||||
|
string DatasetId,
|
||||||
|
int SourceCount,
|
||||||
|
int DestinationCount,
|
||||||
|
IReadOnlyDictionary<string, int> SourceCountByNode,
|
||||||
|
IReadOnlyDictionary<string, int> DestinationCountByNode,
|
||||||
|
IReadOnlyDictionary<string, string> SourceLatestHashByNode,
|
||||||
|
IReadOnlyDictionary<string, string> DestinationLatestHashByNode,
|
||||||
|
int HashSpotCheckCount,
|
||||||
|
int ChainSpotCheckCount,
|
||||||
|
bool CountsMatch,
|
||||||
|
bool CountsPerNodeMatch,
|
||||||
|
bool LatestHashPerNodeMatch,
|
||||||
|
bool HashSpotChecksPassed,
|
||||||
|
bool ChainSpotChecksPassed)
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Gets a value indicating whether parity validation passed all checks.
|
||||||
|
/// </summary>
|
||||||
|
public bool IsSuccess =>
|
||||||
|
CountsMatch &&
|
||||||
|
CountsPerNodeMatch &&
|
||||||
|
LatestHashPerNodeMatch &&
|
||||||
|
HashSpotChecksPassed &&
|
||||||
|
ChainSpotChecksPassed;
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
namespace ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Runtime feature flags controlling Surreal/LMDB oplog migration behavior.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class LmdbOplogFeatureFlags
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets a value indicating whether LMDB oplog support is enabled.
|
||||||
|
/// </summary>
|
||||||
|
public bool UseLmdbOplog { get; set; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets a value indicating whether writes should be mirrored to both Surreal and LMDB oplogs.
|
||||||
|
/// </summary>
|
||||||
|
public bool DualWriteOplog { get; set; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets a value indicating whether reads should prefer LMDB when LMDB is enabled.
|
||||||
|
/// </summary>
|
||||||
|
public bool PreferLmdbReads { get; set; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets a value indicating whether read results should be shadow-compared between stores.
|
||||||
|
/// </summary>
|
||||||
|
public bool EnableReadShadowValidation { get; set; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets the minimum interval between reconciliation backfills when LMDB reads are preferred.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan ReconciliationInterval { get; set; } = TimeSpan.FromSeconds(2);
|
||||||
|
}
|
||||||
68
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/LmdbOplogOptions.cs
Normal file
68
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/LmdbOplogOptions.cs
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
namespace ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Configuration for the LMDB-backed oplog environment.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class LmdbOplogOptions
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets the LMDB environment directory path.
|
||||||
|
/// </summary>
|
||||||
|
public string EnvironmentPath { get; set; } = Path.Combine(AppContext.BaseDirectory, "cbddc-oplog-lmdb");
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets the LMDB map size in bytes.
|
||||||
|
/// </summary>
|
||||||
|
public long MapSizeBytes { get; set; } = 256L * 1024 * 1024;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets the maximum number of named databases in the LMDB environment.
|
||||||
|
/// </summary>
|
||||||
|
public int MaxDatabases { get; set; } = 16;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets the maximum concurrent reader slots.
|
||||||
|
/// </summary>
|
||||||
|
public int MaxReaders { get; set; } = 256;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets sync mode balancing durability and write throughput.
|
||||||
|
/// </summary>
|
||||||
|
public LmdbSyncMode SyncMode { get; set; } = LmdbSyncMode.Full;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets the number of entries to process per prune batch transaction.
|
||||||
|
/// </summary>
|
||||||
|
public int PruneBatchSize { get; set; } = 512;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets or sets a value indicating whether prune operations may run compact-copy backup passes.
|
||||||
|
/// </summary>
|
||||||
|
public bool EnableCompactionCopy { get; set; }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// LMDB durability modes.
|
||||||
|
/// </summary>
|
||||||
|
public enum LmdbSyncMode
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Full durability semantics.
|
||||||
|
/// </summary>
|
||||||
|
Full,
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Skip metadata sync on each commit.
|
||||||
|
/// </summary>
|
||||||
|
NoMetaSync,
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Skip fsync on commit for highest throughput.
|
||||||
|
/// </summary>
|
||||||
|
NoSync,
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Write-mapped asynchronous flush mode.
|
||||||
|
/// </summary>
|
||||||
|
MapAsync
|
||||||
|
}
|
||||||
1385
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/LmdbOplogStore.cs
Normal file
1385
src/ZB.MOM.WW.CBDDC.Persistence/Lmdb/LmdbOplogStore.cs
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,89 @@
|
|||||||
|
using System.Collections.Concurrent;
|
||||||
|
using System.Collections.ObjectModel;
|
||||||
|
using System.Threading;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Tracks LMDB migration telemetry for shadow comparisons, reconciliation, and fallback behavior.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class OplogMigrationTelemetry
|
||||||
|
{
|
||||||
|
private readonly ConcurrentDictionary<string, long> _reconciliationRunsByDataset =
|
||||||
|
new(StringComparer.Ordinal);
|
||||||
|
|
||||||
|
private readonly ConcurrentDictionary<string, long> _reconciledEntriesByDataset =
|
||||||
|
new(StringComparer.Ordinal);
|
||||||
|
|
||||||
|
private long _shadowComparisons;
|
||||||
|
private long _shadowMismatches;
|
||||||
|
private long _preferredReadFallbacks;
|
||||||
|
private long _reconciliationRuns;
|
||||||
|
private long _reconciledEntries;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Records the outcome of one shadow comparison.
|
||||||
|
/// </summary>
|
||||||
|
public void RecordShadowComparison(bool isMatch)
|
||||||
|
{
|
||||||
|
Interlocked.Increment(ref _shadowComparisons);
|
||||||
|
if (!isMatch) Interlocked.Increment(ref _shadowMismatches);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Records a fallback from preferred LMDB reads back to Surreal.
|
||||||
|
/// </summary>
|
||||||
|
public void RecordPreferredReadFallback()
|
||||||
|
{
|
||||||
|
Interlocked.Increment(ref _preferredReadFallbacks);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Records one reconciliation/backfill run for a dataset.
|
||||||
|
/// </summary>
|
||||||
|
public void RecordReconciliation(string datasetId, int entriesMerged)
|
||||||
|
{
|
||||||
|
string normalizedDatasetId = DatasetId.Normalize(datasetId);
|
||||||
|
|
||||||
|
Interlocked.Increment(ref _reconciliationRuns);
|
||||||
|
Interlocked.Add(ref _reconciledEntries, Math.Max(0, entriesMerged));
|
||||||
|
_reconciliationRunsByDataset.AddOrUpdate(normalizedDatasetId, 1, (_, current) => current + 1);
|
||||||
|
_reconciledEntriesByDataset.AddOrUpdate(
|
||||||
|
normalizedDatasetId,
|
||||||
|
Math.Max(0, entriesMerged),
|
||||||
|
(_, current) => current + Math.Max(0, entriesMerged));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Returns an immutable snapshot of current counters.
|
||||||
|
/// </summary>
|
||||||
|
public OplogMigrationTelemetrySnapshot GetSnapshot()
|
||||||
|
{
|
||||||
|
var runsByDataset = new ReadOnlyDictionary<string, long>(
|
||||||
|
_reconciliationRunsByDataset.ToDictionary(pair => pair.Key, pair => pair.Value, StringComparer.Ordinal));
|
||||||
|
var entriesByDataset = new ReadOnlyDictionary<string, long>(
|
||||||
|
_reconciledEntriesByDataset.ToDictionary(pair => pair.Key, pair => pair.Value, StringComparer.Ordinal));
|
||||||
|
|
||||||
|
return new OplogMigrationTelemetrySnapshot(
|
||||||
|
Interlocked.Read(ref _shadowComparisons),
|
||||||
|
Interlocked.Read(ref _shadowMismatches),
|
||||||
|
Interlocked.Read(ref _preferredReadFallbacks),
|
||||||
|
Interlocked.Read(ref _reconciliationRuns),
|
||||||
|
Interlocked.Read(ref _reconciledEntries),
|
||||||
|
runsByDataset,
|
||||||
|
entriesByDataset);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Immutable snapshot for LMDB migration telemetry counters.
|
||||||
|
/// </summary>
|
||||||
|
public readonly record struct OplogMigrationTelemetrySnapshot(
|
||||||
|
long ShadowComparisons,
|
||||||
|
long ShadowMismatches,
|
||||||
|
long PreferredReadFallbacks,
|
||||||
|
long ReconciliationRuns,
|
||||||
|
long ReconciledEntries,
|
||||||
|
IReadOnlyDictionary<string, long> ReconciliationRunsByDataset,
|
||||||
|
IReadOnlyDictionary<string, long> ReconciledEntriesByDataset);
|
||||||
@@ -124,7 +124,8 @@ public static class CBDDCSurrealEmbeddedExtensions
|
|||||||
services.TryAddSingleton<IPeerOplogConfirmationStore, SurrealPeerOplogConfirmationStore>();
|
services.TryAddSingleton<IPeerOplogConfirmationStore, SurrealPeerOplogConfirmationStore>();
|
||||||
services.TryAddSingleton<ISnapshotMetadataStore, SurrealSnapshotMetadataStore>();
|
services.TryAddSingleton<ISnapshotMetadataStore, SurrealSnapshotMetadataStore>();
|
||||||
services.TryAddSingleton<IDocumentMetadataStore, SurrealDocumentMetadataStore>();
|
services.TryAddSingleton<IDocumentMetadataStore, SurrealDocumentMetadataStore>();
|
||||||
services.TryAddSingleton<IOplogStore, SurrealOplogStore>();
|
services.TryAddSingleton<SurrealOplogStore>();
|
||||||
|
services.TryAddSingleton<IOplogStore>(sp => sp.GetRequiredService<SurrealOplogStore>());
|
||||||
|
|
||||||
// SnapshotStore registration matches the other provider extension patterns.
|
// SnapshotStore registration matches the other provider extension patterns.
|
||||||
services.TryAddSingleton<ISnapshotService, SnapshotStore>();
|
services.TryAddSingleton<ISnapshotService, SnapshotStore>();
|
||||||
|
|||||||
@@ -20,6 +20,7 @@
|
|||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
|
<PackageReference Include="LightningDB" Version="0.21.0" />
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="9.0.4" />
|
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="9.0.4" />
|
||||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.4" />
|
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.4" />
|
||||||
<PackageReference Include="SurrealDb.Embedded.RocksDb" Version="0.9.0" />
|
<PackageReference Include="SurrealDb.Embedded.RocksDb" Version="0.9.0" />
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ using ZB.MOM.WW.CBDDC.Core.Storage;
|
|||||||
using ZB.MOM.WW.CBDDC.Core.Sync;
|
using ZB.MOM.WW.CBDDC.Core.Sync;
|
||||||
using ZB.MOM.WW.CBDDC.Network;
|
using ZB.MOM.WW.CBDDC.Network;
|
||||||
using ZB.MOM.WW.CBDDC.Network.Security;
|
using ZB.MOM.WW.CBDDC.Network.Security;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
using ZB.MOM.WW.CBDDC.Persistence.Surreal;
|
using ZB.MOM.WW.CBDDC.Persistence.Surreal;
|
||||||
|
|
||||||
namespace ZB.MOM.WW.CBDDC.E2E.Tests;
|
namespace ZB.MOM.WW.CBDDC.E2E.Tests;
|
||||||
@@ -240,6 +241,92 @@ public class ClusterCrudSyncE2ETests
|
|||||||
}, 60, "Node B did not catch up missed reconnect mutations.", () => BuildDiagnostics(nodeA, nodeB));
|
}, 60, "Node B did not catch up missed reconnect mutations.", () => BuildDiagnostics(nodeA, nodeB));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Verifies reconnect catch-up still works when reads are cut over to LMDB with dual-write enabled.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task PeerReconnect_ShouldCatchUpMissedChanges_WithLmdbPreferredReads()
|
||||||
|
{
|
||||||
|
var clusterToken = Guid.NewGuid().ToString("N");
|
||||||
|
int nodeAPort = GetAvailableTcpPort();
|
||||||
|
int nodeBPort = GetAvailableTcpPort();
|
||||||
|
while (nodeBPort == nodeAPort) nodeBPort = GetAvailableTcpPort();
|
||||||
|
|
||||||
|
await using var nodeA = TestPeerNode.Create(
|
||||||
|
"node-a",
|
||||||
|
nodeAPort,
|
||||||
|
clusterToken,
|
||||||
|
[
|
||||||
|
new KnownPeerConfiguration
|
||||||
|
{
|
||||||
|
NodeId = "node-b",
|
||||||
|
Host = "127.0.0.1",
|
||||||
|
Port = nodeBPort
|
||||||
|
}
|
||||||
|
],
|
||||||
|
useLmdbOplog: true,
|
||||||
|
dualWriteOplog: true,
|
||||||
|
preferLmdbReads: true);
|
||||||
|
|
||||||
|
await using var nodeB = TestPeerNode.Create(
|
||||||
|
"node-b",
|
||||||
|
nodeBPort,
|
||||||
|
clusterToken,
|
||||||
|
[
|
||||||
|
new KnownPeerConfiguration
|
||||||
|
{
|
||||||
|
NodeId = "node-a",
|
||||||
|
Host = "127.0.0.1",
|
||||||
|
Port = nodeAPort
|
||||||
|
}
|
||||||
|
],
|
||||||
|
useLmdbOplog: true,
|
||||||
|
dualWriteOplog: true,
|
||||||
|
preferLmdbReads: true);
|
||||||
|
|
||||||
|
await nodeA.StartAsync();
|
||||||
|
await nodeB.StartAsync();
|
||||||
|
|
||||||
|
await nodeB.StopAsync();
|
||||||
|
|
||||||
|
const string userId = "reconnect-lmdb-user";
|
||||||
|
await nodeA.UpsertUserAsync(new User
|
||||||
|
{
|
||||||
|
Id = userId,
|
||||||
|
Name = "Offline Create",
|
||||||
|
Age = 20,
|
||||||
|
Address = new Address { City = "Rome" }
|
||||||
|
});
|
||||||
|
|
||||||
|
await nodeA.UpsertUserAsync(new User
|
||||||
|
{
|
||||||
|
Id = userId,
|
||||||
|
Name = "Offline Update",
|
||||||
|
Age = 21,
|
||||||
|
Address = new Address { City = "Milan" }
|
||||||
|
});
|
||||||
|
|
||||||
|
await nodeA.UpsertUserAsync(new User
|
||||||
|
{
|
||||||
|
Id = userId,
|
||||||
|
Name = "Offline Final",
|
||||||
|
Age = 22,
|
||||||
|
Address = new Address { City = "Turin" }
|
||||||
|
});
|
||||||
|
|
||||||
|
await nodeB.StartAsync();
|
||||||
|
|
||||||
|
await AssertEventuallyAsync(() =>
|
||||||
|
{
|
||||||
|
var replicated = nodeB.ReadUser(userId);
|
||||||
|
return replicated is not null &&
|
||||||
|
replicated.Name == "Offline Final" &&
|
||||||
|
replicated.Age == 22 &&
|
||||||
|
replicated.Address?.City == "Turin";
|
||||||
|
}, 60, "Node B did not catch up missed reconnect mutations with LMDB preferred reads.",
|
||||||
|
() => BuildDiagnostics(nodeA, nodeB));
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Verifies a burst of rapid multi-node mutations converges to a deterministic final state.
|
/// Verifies a burst of rapid multi-node mutations converges to a deterministic final state.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
@@ -572,6 +659,9 @@ public class ClusterCrudSyncE2ETests
|
|||||||
/// <param name="workDirOverride">An optional working directory override for test artifacts.</param>
|
/// <param name="workDirOverride">An optional working directory override for test artifacts.</param>
|
||||||
/// <param name="preserveWorkDirOnDispose">A value indicating whether to preserve the working directory on dispose.</param>
|
/// <param name="preserveWorkDirOnDispose">A value indicating whether to preserve the working directory on dispose.</param>
|
||||||
/// <param name="useFaultInjectedCheckpointStore">A value indicating whether to inject a checkpoint persistence that fails once.</param>
|
/// <param name="useFaultInjectedCheckpointStore">A value indicating whether to inject a checkpoint persistence that fails once.</param>
|
||||||
|
/// <param name="useLmdbOplog">A value indicating whether to enable the LMDB oplog migration path.</param>
|
||||||
|
/// <param name="dualWriteOplog">A value indicating whether oplog writes should be mirrored to Surreal + LMDB.</param>
|
||||||
|
/// <param name="preferLmdbReads">A value indicating whether reads should prefer LMDB.</param>
|
||||||
/// <returns>A configured <see cref="TestPeerNode" /> instance.</returns>
|
/// <returns>A configured <see cref="TestPeerNode" /> instance.</returns>
|
||||||
public static TestPeerNode Create(
|
public static TestPeerNode Create(
|
||||||
string nodeId,
|
string nodeId,
|
||||||
@@ -580,7 +670,10 @@ public class ClusterCrudSyncE2ETests
|
|||||||
IReadOnlyList<KnownPeerConfiguration> knownPeers,
|
IReadOnlyList<KnownPeerConfiguration> knownPeers,
|
||||||
string? workDirOverride = null,
|
string? workDirOverride = null,
|
||||||
bool preserveWorkDirOnDispose = false,
|
bool preserveWorkDirOnDispose = false,
|
||||||
bool useFaultInjectedCheckpointStore = false)
|
bool useFaultInjectedCheckpointStore = false,
|
||||||
|
bool useLmdbOplog = false,
|
||||||
|
bool dualWriteOplog = true,
|
||||||
|
bool preferLmdbReads = false)
|
||||||
{
|
{
|
||||||
string workDir = workDirOverride ?? Path.Combine(Path.GetTempPath(), $"cbddc-e2e-{nodeId}-{Guid.NewGuid():N}");
|
string workDir = workDirOverride ?? Path.Combine(Path.GetTempPath(), $"cbddc-e2e-{nodeId}-{Guid.NewGuid():N}");
|
||||||
Directory.CreateDirectory(workDir);
|
Directory.CreateDirectory(workDir);
|
||||||
@@ -620,13 +713,47 @@ public class ClusterCrudSyncE2ETests
|
|||||||
if (useFaultInjectedCheckpointStore)
|
if (useFaultInjectedCheckpointStore)
|
||||||
{
|
{
|
||||||
services.AddSingleton<ISurrealCdcCheckpointPersistence, CrashAfterFirstAdvanceCheckpointPersistence>();
|
services.AddSingleton<ISurrealCdcCheckpointPersistence, CrashAfterFirstAdvanceCheckpointPersistence>();
|
||||||
coreBuilder.AddCBDDCSurrealEmbedded<FaultInjectedSampleDocumentStore>(surrealOptionsFactory)
|
var registration = coreBuilder.AddCBDDCSurrealEmbedded<FaultInjectedSampleDocumentStore>(surrealOptionsFactory);
|
||||||
.AddCBDDCNetwork<StaticPeerNodeConfigurationProvider>(false);
|
if (useLmdbOplog)
|
||||||
|
registration.AddCBDDCLmdbOplog(
|
||||||
|
_ => new LmdbOplogOptions
|
||||||
|
{
|
||||||
|
EnvironmentPath = Path.Combine(workDir, "oplog-lmdb"),
|
||||||
|
MapSizeBytes = 128L * 1024 * 1024,
|
||||||
|
MaxDatabases = 16,
|
||||||
|
PruneBatchSize = 256
|
||||||
|
},
|
||||||
|
flags =>
|
||||||
|
{
|
||||||
|
flags.UseLmdbOplog = true;
|
||||||
|
flags.DualWriteOplog = dualWriteOplog;
|
||||||
|
flags.PreferLmdbReads = preferLmdbReads;
|
||||||
|
flags.ReconciliationInterval = TimeSpan.Zero;
|
||||||
|
});
|
||||||
|
|
||||||
|
registration.AddCBDDCNetwork<StaticPeerNodeConfigurationProvider>(false);
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
coreBuilder.AddCBDDCSurrealEmbedded<SampleDocumentStore>(surrealOptionsFactory)
|
var registration = coreBuilder.AddCBDDCSurrealEmbedded<SampleDocumentStore>(surrealOptionsFactory);
|
||||||
.AddCBDDCNetwork<StaticPeerNodeConfigurationProvider>(false);
|
if (useLmdbOplog)
|
||||||
|
registration.AddCBDDCLmdbOplog(
|
||||||
|
_ => new LmdbOplogOptions
|
||||||
|
{
|
||||||
|
EnvironmentPath = Path.Combine(workDir, "oplog-lmdb"),
|
||||||
|
MapSizeBytes = 128L * 1024 * 1024,
|
||||||
|
MaxDatabases = 16,
|
||||||
|
PruneBatchSize = 256
|
||||||
|
},
|
||||||
|
flags =>
|
||||||
|
{
|
||||||
|
flags.UseLmdbOplog = true;
|
||||||
|
flags.DualWriteOplog = dualWriteOplog;
|
||||||
|
flags.PreferLmdbReads = preferLmdbReads;
|
||||||
|
flags.ReconciliationInterval = TimeSpan.Zero;
|
||||||
|
});
|
||||||
|
|
||||||
|
registration.AddCBDDCNetwork<StaticPeerNodeConfigurationProvider>(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Deterministic tests: sync uses explicit known peers, so disable UDP discovery.
|
// Deterministic tests: sync uses explicit known peers, so disable UDP discovery.
|
||||||
|
|||||||
@@ -0,0 +1,237 @@
|
|||||||
|
using System.Text.Json;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using NSubstitute;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Storage;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Sync;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Surreal;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.CBDDC.Sample.Console.Tests;
|
||||||
|
|
||||||
|
public class LmdbOplogMigrationTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public async Task FeatureFlags_DualWrite_WritesToBothStores()
|
||||||
|
{
|
||||||
|
await using var surrealHarness = new SurrealTestHarness();
|
||||||
|
var surrealStore = surrealHarness.CreateOplogStore();
|
||||||
|
using var lmdbStore = CreateLmdbStore();
|
||||||
|
|
||||||
|
var flags = new LmdbOplogFeatureFlags
|
||||||
|
{
|
||||||
|
UseLmdbOplog = true,
|
||||||
|
DualWriteOplog = true,
|
||||||
|
PreferLmdbReads = false
|
||||||
|
};
|
||||||
|
|
||||||
|
var store = new FeatureFlagOplogStore(
|
||||||
|
surrealStore,
|
||||||
|
lmdbStore,
|
||||||
|
flags,
|
||||||
|
logger: NullLogger<FeatureFlagOplogStore>.Instance);
|
||||||
|
|
||||||
|
var entry = CreateEntry("Users", "dual-write", "node-a", 100, 0, "");
|
||||||
|
await store.AppendOplogEntryAsync(entry);
|
||||||
|
|
||||||
|
(await surrealStore.GetEntryByHashAsync(entry.Hash)).ShouldNotBeNull();
|
||||||
|
(await lmdbStore.GetEntryByHashAsync(entry.Hash)).ShouldNotBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task FeatureFlags_PreferLmdbReads_ReconcilesFromSurrealWhenLmdbMissingEntries()
|
||||||
|
{
|
||||||
|
await using var surrealHarness = new SurrealTestHarness();
|
||||||
|
var surrealStore = surrealHarness.CreateOplogStore();
|
||||||
|
using var lmdbStore = CreateLmdbStore();
|
||||||
|
|
||||||
|
var flags = new LmdbOplogFeatureFlags
|
||||||
|
{
|
||||||
|
UseLmdbOplog = true,
|
||||||
|
DualWriteOplog = false,
|
||||||
|
PreferLmdbReads = true,
|
||||||
|
ReconciliationInterval = TimeSpan.Zero
|
||||||
|
};
|
||||||
|
|
||||||
|
var entry = CreateEntry("Users", "reconcile-1", "node-a", 200, 0, "");
|
||||||
|
|
||||||
|
// Simulate crash window where only Surreal persisted before LMDB migration store starts.
|
||||||
|
await surrealStore.AppendOplogEntryAsync(entry);
|
||||||
|
(await lmdbStore.GetEntryByHashAsync(entry.Hash)).ShouldBeNull();
|
||||||
|
|
||||||
|
var store = new FeatureFlagOplogStore(
|
||||||
|
surrealStore,
|
||||||
|
lmdbStore,
|
||||||
|
flags,
|
||||||
|
logger: NullLogger<FeatureFlagOplogStore>.Instance);
|
||||||
|
|
||||||
|
OplogEntry? resolved = await store.GetEntryByHashAsync(entry.Hash);
|
||||||
|
resolved.ShouldNotBeNull();
|
||||||
|
resolved.Hash.ShouldBe(entry.Hash);
|
||||||
|
|
||||||
|
// Reconciliation should have backfilled LMDB.
|
||||||
|
(await lmdbStore.GetEntryByHashAsync(entry.Hash)).ShouldNotBeNull();
|
||||||
|
|
||||||
|
OplogMigrationTelemetrySnapshot telemetry = store.GetTelemetrySnapshot();
|
||||||
|
telemetry.ReconciliationRuns.ShouldBeGreaterThanOrEqualTo(1);
|
||||||
|
telemetry.ReconciledEntries.ShouldBeGreaterThanOrEqualTo(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task FeatureFlags_ShadowValidation_RecordsMismatchTelemetry()
|
||||||
|
{
|
||||||
|
await using var surrealHarness = new SurrealTestHarness();
|
||||||
|
var surrealStore = surrealHarness.CreateOplogStore();
|
||||||
|
using var lmdbStore = CreateLmdbStore();
|
||||||
|
var telemetry = new OplogMigrationTelemetry();
|
||||||
|
|
||||||
|
var flags = new LmdbOplogFeatureFlags
|
||||||
|
{
|
||||||
|
UseLmdbOplog = true,
|
||||||
|
DualWriteOplog = true,
|
||||||
|
PreferLmdbReads = false,
|
||||||
|
EnableReadShadowValidation = true
|
||||||
|
};
|
||||||
|
|
||||||
|
var store = new FeatureFlagOplogStore(
|
||||||
|
surrealStore,
|
||||||
|
lmdbStore,
|
||||||
|
flags,
|
||||||
|
telemetry,
|
||||||
|
NullLogger<FeatureFlagOplogStore>.Instance);
|
||||||
|
|
||||||
|
var entry = CreateEntry("Users", "shadow-mismatch-1", "node-a", 210, 0, "");
|
||||||
|
await surrealStore.AppendOplogEntryAsync(entry);
|
||||||
|
|
||||||
|
OplogEntry? resolved = await store.GetEntryByHashAsync(entry.Hash);
|
||||||
|
resolved.ShouldNotBeNull();
|
||||||
|
|
||||||
|
OplogMigrationTelemetrySnapshot snapshot = store.GetTelemetrySnapshot();
|
||||||
|
snapshot.ShadowComparisons.ShouldBe(1);
|
||||||
|
snapshot.ShadowMismatches.ShouldBe(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task FeatureFlags_RollbackToSurreal_UsesSurrealForWritesAndReads()
|
||||||
|
{
|
||||||
|
await using var surrealHarness = new SurrealTestHarness();
|
||||||
|
var surrealStore = surrealHarness.CreateOplogStore();
|
||||||
|
using var lmdbStore = CreateLmdbStore();
|
||||||
|
|
||||||
|
var flags = new LmdbOplogFeatureFlags
|
||||||
|
{
|
||||||
|
UseLmdbOplog = true,
|
||||||
|
DualWriteOplog = false,
|
||||||
|
PreferLmdbReads = false
|
||||||
|
};
|
||||||
|
|
||||||
|
var store = new FeatureFlagOplogStore(
|
||||||
|
surrealStore,
|
||||||
|
lmdbStore,
|
||||||
|
flags,
|
||||||
|
logger: NullLogger<FeatureFlagOplogStore>.Instance);
|
||||||
|
|
||||||
|
var entry = CreateEntry("Users", "rollback-1", "node-a", 220, 0, "");
|
||||||
|
await store.AppendOplogEntryAsync(entry);
|
||||||
|
|
||||||
|
(await surrealStore.GetEntryByHashAsync(entry.Hash)).ShouldNotBeNull();
|
||||||
|
(await lmdbStore.GetEntryByHashAsync(entry.Hash)).ShouldBeNull();
|
||||||
|
|
||||||
|
OplogEntry? routedRead = await store.GetEntryByHashAsync(entry.Hash);
|
||||||
|
routedRead.ShouldNotBeNull();
|
||||||
|
routedRead.Hash.ShouldBe(entry.Hash);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task BackfillTool_BackfillAndValidate_ReportsSuccess()
|
||||||
|
{
|
||||||
|
await using var surrealHarness = new SurrealTestHarness();
|
||||||
|
var surrealStore = surrealHarness.CreateOplogStore();
|
||||||
|
using var lmdbStore = CreateLmdbStore();
|
||||||
|
var tool = new LmdbOplogBackfillTool(surrealStore, lmdbStore, NullLogger<LmdbOplogBackfillTool>.Instance);
|
||||||
|
|
||||||
|
var first = CreateEntry("Users", "backfill-1", "node-a", 300, 0, "");
|
||||||
|
var second = CreateEntry("Users", "backfill-2", "node-a", 301, 0, first.Hash);
|
||||||
|
var third = CreateEntry("Users", "backfill-3", "node-b", 302, 0, "");
|
||||||
|
var fourth = CreateEntry("Users", "backfill-4", "node-b", 303, 0, third.Hash);
|
||||||
|
|
||||||
|
await surrealStore.AppendOplogEntryAsync(first);
|
||||||
|
await surrealStore.AppendOplogEntryAsync(second);
|
||||||
|
await surrealStore.AppendOplogEntryAsync(third);
|
||||||
|
await surrealStore.AppendOplogEntryAsync(fourth);
|
||||||
|
|
||||||
|
LmdbOplogBackfillReport report = await tool.BackfillAsync(DatasetId.Primary);
|
||||||
|
|
||||||
|
report.IsSuccess.ShouldBeTrue();
|
||||||
|
report.CountsMatch.ShouldBeTrue();
|
||||||
|
report.CountsPerNodeMatch.ShouldBeTrue();
|
||||||
|
report.LatestHashPerNodeMatch.ShouldBeTrue();
|
||||||
|
report.HashSpotChecksPassed.ShouldBeTrue();
|
||||||
|
report.ChainSpotChecksPassed.ShouldBeTrue();
|
||||||
|
report.SourceCount.ShouldBe(4);
|
||||||
|
report.DestinationCount.ShouldBe(4);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task BackfillTool_BackfillAndValidate_WorksPerDataset()
|
||||||
|
{
|
||||||
|
await using var surrealHarness = new SurrealTestHarness();
|
||||||
|
var surrealStore = surrealHarness.CreateOplogStore();
|
||||||
|
using var lmdbStore = CreateLmdbStore();
|
||||||
|
var tool = new LmdbOplogBackfillTool(surrealStore, lmdbStore, NullLogger<LmdbOplogBackfillTool>.Instance);
|
||||||
|
|
||||||
|
var logsEntryA = CreateEntry("Logs", "log-1", "node-a", 400, 0, "");
|
||||||
|
var logsEntryB = CreateEntry("Logs", "log-2", "node-a", 401, 0, logsEntryA.Hash);
|
||||||
|
var primaryEntry = CreateEntry("Users", "primary-1", "node-a", 500, 0, "");
|
||||||
|
|
||||||
|
await surrealStore.AppendOplogEntryAsync(logsEntryA, DatasetId.Logs);
|
||||||
|
await surrealStore.AppendOplogEntryAsync(logsEntryB, DatasetId.Logs);
|
||||||
|
await surrealStore.AppendOplogEntryAsync(primaryEntry, DatasetId.Primary);
|
||||||
|
|
||||||
|
LmdbOplogBackfillReport logsReport = await tool.BackfillAsync(DatasetId.Logs);
|
||||||
|
logsReport.IsSuccess.ShouldBeTrue();
|
||||||
|
logsReport.SourceCount.ShouldBe(2);
|
||||||
|
logsReport.DestinationCount.ShouldBe(2);
|
||||||
|
|
||||||
|
(await lmdbStore.GetEntryByHashAsync(logsEntryA.Hash, DatasetId.Logs)).ShouldNotBeNull();
|
||||||
|
(await lmdbStore.GetEntryByHashAsync(logsEntryB.Hash, DatasetId.Logs)).ShouldNotBeNull();
|
||||||
|
(await lmdbStore.GetEntryByHashAsync(primaryEntry.Hash, DatasetId.Primary)).ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static LmdbOplogStore CreateLmdbStore()
|
||||||
|
{
|
||||||
|
string rootPath = Path.Combine(Path.GetTempPath(), "cbddc-lmdb-migration", Guid.NewGuid().ToString("N"));
|
||||||
|
Directory.CreateDirectory(rootPath);
|
||||||
|
|
||||||
|
return new LmdbOplogStore(
|
||||||
|
Substitute.For<IDocumentStore>(),
|
||||||
|
new LastWriteWinsConflictResolver(),
|
||||||
|
new VectorClockService(),
|
||||||
|
new LmdbOplogOptions
|
||||||
|
{
|
||||||
|
EnvironmentPath = rootPath,
|
||||||
|
MapSizeBytes = 64L * 1024 * 1024,
|
||||||
|
MaxDatabases = 16
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
NullLogger<LmdbOplogStore>.Instance);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static OplogEntry CreateEntry(
|
||||||
|
string collection,
|
||||||
|
string key,
|
||||||
|
string nodeId,
|
||||||
|
long wall,
|
||||||
|
int logic,
|
||||||
|
string previousHash)
|
||||||
|
{
|
||||||
|
return new OplogEntry(
|
||||||
|
collection,
|
||||||
|
key,
|
||||||
|
OperationType.Put,
|
||||||
|
JsonSerializer.SerializeToElement(new { key }),
|
||||||
|
new HlcTimestamp(wall, logic, nodeId),
|
||||||
|
previousHash);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,267 @@
|
|||||||
|
using System.Diagnostics;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using NSubstitute;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Storage;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Sync;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Lmdb;
|
||||||
|
using ZB.MOM.WW.CBDDC.Persistence.Surreal;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.CBDDC.Sample.Console.Tests;
|
||||||
|
|
||||||
|
public class SurrealOplogStoreContractParityTests : OplogStoreContractTestBase
|
||||||
|
{
|
||||||
|
protected override Task<IOplogStoreContractHarness> CreateHarnessAsync()
|
||||||
|
{
|
||||||
|
return Task.FromResult<IOplogStoreContractHarness>(new SurrealOplogStoreContractHarness());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public class LmdbOplogStoreContractTests : OplogStoreContractTestBase
|
||||||
|
{
|
||||||
|
protected override Task<IOplogStoreContractHarness> CreateHarnessAsync()
|
||||||
|
{
|
||||||
|
return Task.FromResult<IOplogStoreContractHarness>(new LmdbOplogStoreContractHarness());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Lmdb_IndexConsistency_InsertPopulatesAndPruneRemovesIndexes()
|
||||||
|
{
|
||||||
|
await using var harness = new LmdbOplogStoreContractHarness();
|
||||||
|
var store = (LmdbOplogStore)harness.Store;
|
||||||
|
|
||||||
|
var entry1 = CreateOplogEntry("Users", "i1", "node-a", 100, 0, "");
|
||||||
|
var entry2 = CreateOplogEntry("Users", "i2", "node-a", 101, 0, entry1.Hash);
|
||||||
|
|
||||||
|
await store.AppendOplogEntryAsync(entry1);
|
||||||
|
await store.AppendOplogEntryAsync(entry2);
|
||||||
|
|
||||||
|
LmdbOplogIndexDiagnostics before = await store.GetIndexDiagnosticsAsync(DatasetId.Primary);
|
||||||
|
before.OplogByHashCount.ShouldBe(2);
|
||||||
|
before.OplogByHlcCount.ShouldBe(2);
|
||||||
|
before.OplogByNodeHlcCount.ShouldBe(2);
|
||||||
|
before.OplogPrevToHashCount.ShouldBe(1);
|
||||||
|
before.OplogNodeHeadCount.ShouldBe(1);
|
||||||
|
|
||||||
|
await store.PruneOplogAsync(new HlcTimestamp(101, 0, "node-a"));
|
||||||
|
|
||||||
|
LmdbOplogIndexDiagnostics after = await store.GetIndexDiagnosticsAsync(DatasetId.Primary);
|
||||||
|
after.OplogByHashCount.ShouldBe(0);
|
||||||
|
after.OplogByHlcCount.ShouldBe(0);
|
||||||
|
after.OplogByNodeHlcCount.ShouldBe(0);
|
||||||
|
after.OplogPrevToHashCount.ShouldBe(0);
|
||||||
|
after.OplogNodeHeadCount.ShouldBe(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Lmdb_Prune_RemovesAtOrBeforeCutoff_AndKeepsNewerInterleavedEntries()
|
||||||
|
{
|
||||||
|
await using var harness = new LmdbOplogStoreContractHarness();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
var nodeAOld = CreateOplogEntry("Users", "a-old", "node-a", 100, 0, "");
|
||||||
|
var nodeBKeep = CreateOplogEntry("Users", "b-keep", "node-b", 105, 0, "");
|
||||||
|
var nodeANew = CreateOplogEntry("Users", "a-new", "node-a", 110, 0, nodeAOld.Hash);
|
||||||
|
var lateOld = CreateOplogEntry("Users", "late-old", "node-c", 90, 0, "");
|
||||||
|
|
||||||
|
await store.AppendOplogEntryAsync(nodeAOld);
|
||||||
|
await store.AppendOplogEntryAsync(nodeBKeep);
|
||||||
|
await store.AppendOplogEntryAsync(nodeANew);
|
||||||
|
await store.AppendOplogEntryAsync(lateOld);
|
||||||
|
|
||||||
|
await store.PruneOplogAsync(new HlcTimestamp(100, 0, "node-a"));
|
||||||
|
|
||||||
|
var remaining = (await store.ExportAsync()).Select(e => e.Hash).ToHashSet(StringComparer.Ordinal);
|
||||||
|
remaining.Contains(nodeAOld.Hash).ShouldBeFalse();
|
||||||
|
remaining.Contains(lateOld.Hash).ShouldBeFalse();
|
||||||
|
remaining.Contains(nodeBKeep.Hash).ShouldBeTrue();
|
||||||
|
remaining.Contains(nodeANew.Hash).ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Lmdb_NodeHead_AdvancesAndRecomputesAcrossPrune()
|
||||||
|
{
|
||||||
|
await using var harness = new LmdbOplogStoreContractHarness();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
var older = CreateOplogEntry("Users", "n1", "node-a", 100, 0, "");
|
||||||
|
var newer = CreateOplogEntry("Users", "n2", "node-a", 120, 0, older.Hash);
|
||||||
|
|
||||||
|
await store.AppendOplogEntryAsync(older);
|
||||||
|
await store.AppendOplogEntryAsync(newer);
|
||||||
|
|
||||||
|
(await store.GetLastEntryHashAsync("node-a")).ShouldBe(newer.Hash);
|
||||||
|
|
||||||
|
await store.PruneOplogAsync(new HlcTimestamp(110, 0, "node-a"));
|
||||||
|
(await store.GetLastEntryHashAsync("node-a")).ShouldBe(newer.Hash);
|
||||||
|
|
||||||
|
await store.PruneOplogAsync(new HlcTimestamp(130, 0, "node-a"));
|
||||||
|
(await store.GetLastEntryHashAsync("node-a")).ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Lmdb_RestartDurability_PreservesHeadAndScans()
|
||||||
|
{
|
||||||
|
await using var harness = new LmdbOplogStoreContractHarness();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
var entry1 = CreateOplogEntry("Users", "r1", "node-a", 100, 0, "");
|
||||||
|
var entry2 = CreateOplogEntry("Users", "r2", "node-a", 101, 0, entry1.Hash);
|
||||||
|
|
||||||
|
await store.AppendOplogEntryAsync(entry1);
|
||||||
|
await store.AppendOplogEntryAsync(entry2);
|
||||||
|
|
||||||
|
IOplogStore reopened = harness.ReopenStore();
|
||||||
|
(await reopened.GetLastEntryHashAsync("node-a")).ShouldBe(entry2.Hash);
|
||||||
|
|
||||||
|
var after = (await reopened.GetOplogAfterAsync(new HlcTimestamp(0, 0, string.Empty))).ToList();
|
||||||
|
after.Count.ShouldBe(2);
|
||||||
|
after[0].Hash.ShouldBe(entry1.Hash);
|
||||||
|
after[1].Hash.ShouldBe(entry2.Hash);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Lmdb_Dedupe_DuplicateHashAppendIsIdempotent()
|
||||||
|
{
|
||||||
|
await using var harness = new LmdbOplogStoreContractHarness();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
var entry = CreateOplogEntry("Users", "d1", "node-a", 100, 0, "");
|
||||||
|
|
||||||
|
await store.AppendOplogEntryAsync(entry);
|
||||||
|
await store.AppendOplogEntryAsync(entry);
|
||||||
|
|
||||||
|
var exported = (await store.ExportAsync()).ToList();
|
||||||
|
exported.Count.ShouldBe(1);
|
||||||
|
exported[0].Hash.ShouldBe(entry.Hash);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Lmdb_PrunePerformanceSmoke_LargeSyntheticWindow_CompletesWithinGenerousBudget()
|
||||||
|
{
|
||||||
|
await using var harness = new LmdbOplogStoreContractHarness();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
string previousHash = string.Empty;
|
||||||
|
for (var i = 0; i < 5000; i++)
|
||||||
|
{
|
||||||
|
var entry = CreateOplogEntry("Users", $"p-{i:D5}", "node-a", 1_000 + i, 0, previousHash);
|
||||||
|
await store.AppendOplogEntryAsync(entry);
|
||||||
|
previousHash = entry.Hash;
|
||||||
|
}
|
||||||
|
|
||||||
|
var sw = Stopwatch.StartNew();
|
||||||
|
await store.PruneOplogAsync(new HlcTimestamp(6_000, 0, "node-a"));
|
||||||
|
sw.Stop();
|
||||||
|
|
||||||
|
sw.ElapsedMilliseconds.ShouldBeLessThan(30_000L);
|
||||||
|
(await store.ExportAsync()).ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class SurrealOplogStoreContractHarness : IOplogStoreContractHarness
|
||||||
|
{
|
||||||
|
private readonly SurrealTestHarness _harness;
|
||||||
|
|
||||||
|
public SurrealOplogStoreContractHarness()
|
||||||
|
{
|
||||||
|
_harness = new SurrealTestHarness();
|
||||||
|
Store = _harness.CreateOplogStore();
|
||||||
|
}
|
||||||
|
|
||||||
|
public IOplogStore Store { get; private set; }
|
||||||
|
|
||||||
|
public IOplogStore ReopenStore()
|
||||||
|
{
|
||||||
|
Store = _harness.CreateOplogStore();
|
||||||
|
return Store;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task AppendOplogEntryAsync(OplogEntry entry, string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return ((SurrealOplogStore)Store).AppendOplogEntryAsync(entry, datasetId, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<IEnumerable<OplogEntry>> ExportAsync(string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return ((SurrealOplogStore)Store).ExportAsync(datasetId, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
public ValueTask DisposeAsync()
|
||||||
|
{
|
||||||
|
return _harness.DisposeAsync();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class LmdbOplogStoreContractHarness : IOplogStoreContractHarness
|
||||||
|
{
|
||||||
|
private readonly string _rootPath;
|
||||||
|
private LmdbOplogStore? _store;
|
||||||
|
|
||||||
|
public LmdbOplogStoreContractHarness()
|
||||||
|
{
|
||||||
|
_rootPath = Path.Combine(Path.GetTempPath(), "cbddc-lmdb-tests", Guid.NewGuid().ToString("N"));
|
||||||
|
Directory.CreateDirectory(_rootPath);
|
||||||
|
_store = CreateStore();
|
||||||
|
}
|
||||||
|
|
||||||
|
public IOplogStore Store => _store ?? throw new ObjectDisposedException(nameof(LmdbOplogStoreContractHarness));
|
||||||
|
|
||||||
|
public IOplogStore ReopenStore()
|
||||||
|
{
|
||||||
|
_store?.Dispose();
|
||||||
|
_store = CreateStore();
|
||||||
|
return _store;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task AppendOplogEntryAsync(OplogEntry entry, string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return (_store ?? throw new ObjectDisposedException(nameof(LmdbOplogStoreContractHarness)))
|
||||||
|
.AppendOplogEntryAsync(entry, datasetId, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<IEnumerable<OplogEntry>> ExportAsync(string datasetId, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
return (_store ?? throw new ObjectDisposedException(nameof(LmdbOplogStoreContractHarness)))
|
||||||
|
.ExportAsync(datasetId, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async ValueTask DisposeAsync()
|
||||||
|
{
|
||||||
|
_store?.Dispose();
|
||||||
|
_store = null;
|
||||||
|
|
||||||
|
for (var attempt = 0; attempt < 5; attempt++)
|
||||||
|
try
|
||||||
|
{
|
||||||
|
if (Directory.Exists(_rootPath))
|
||||||
|
Directory.Delete(_rootPath, true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
catch when (attempt < 4)
|
||||||
|
{
|
||||||
|
await Task.Delay(50);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private LmdbOplogStore CreateStore()
|
||||||
|
{
|
||||||
|
string lmdbPath = Path.Combine(_rootPath, "lmdb-oplog");
|
||||||
|
Directory.CreateDirectory(lmdbPath);
|
||||||
|
|
||||||
|
return new LmdbOplogStore(
|
||||||
|
Substitute.For<IDocumentStore>(),
|
||||||
|
new LastWriteWinsConflictResolver(),
|
||||||
|
new VectorClockService(),
|
||||||
|
new LmdbOplogOptions
|
||||||
|
{
|
||||||
|
EnvironmentPath = lmdbPath,
|
||||||
|
MapSizeBytes = 64L * 1024 * 1024,
|
||||||
|
MaxDatabases = 16,
|
||||||
|
PruneBatchSize = 128
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
NullLogger<LmdbOplogStore>.Instance);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,121 @@
|
|||||||
|
using System.Text.Json;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core;
|
||||||
|
using ZB.MOM.WW.CBDDC.Core.Storage;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.CBDDC.Sample.Console.Tests;
|
||||||
|
|
||||||
|
public abstract class OplogStoreContractTestBase
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public async Task OplogStore_AppendQueryMergeDrop_AndLastHash_Works()
|
||||||
|
{
|
||||||
|
await using var harness = await CreateHarnessAsync();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
var entry1 = CreateOplogEntry("Users", "u1", "node-a", 100, 0, "");
|
||||||
|
var entry2 = CreateOplogEntry("Users", "u2", "node-a", 110, 0, entry1.Hash);
|
||||||
|
var entry3 = CreateOplogEntry("Users", "u3", "node-a", 120, 1, entry2.Hash);
|
||||||
|
var otherNode = CreateOplogEntry("Users", "u4", "node-b", 115, 0, "");
|
||||||
|
|
||||||
|
await store.AppendOplogEntryAsync(entry1);
|
||||||
|
await store.AppendOplogEntryAsync(entry2);
|
||||||
|
await store.AppendOplogEntryAsync(entry3);
|
||||||
|
await store.AppendOplogEntryAsync(otherNode);
|
||||||
|
|
||||||
|
var chainRange = (await store.GetChainRangeAsync(entry1.Hash, entry3.Hash)).ToList();
|
||||||
|
chainRange.Select(x => x.Hash).ToList().ShouldBe(new[] { entry2.Hash, entry3.Hash });
|
||||||
|
|
||||||
|
var after = (await store.GetOplogAfterAsync(new HlcTimestamp(100, 0, "node-a"))).ToList();
|
||||||
|
after.Select(x => x.Hash).ToList().ShouldBe(new[] { entry2.Hash, otherNode.Hash, entry3.Hash });
|
||||||
|
|
||||||
|
var mergedEntry = CreateOplogEntry("Users", "u5", "node-a", 130, 0, entry3.Hash);
|
||||||
|
await store.MergeAsync(new[] { entry2, mergedEntry });
|
||||||
|
|
||||||
|
var exported = (await store.ExportAsync()).ToList();
|
||||||
|
exported.Count.ShouldBe(5);
|
||||||
|
exported.Count(x => x.Hash == entry2.Hash).ShouldBe(1);
|
||||||
|
|
||||||
|
string? cachedLastNodeAHash = await store.GetLastEntryHashAsync("node-a");
|
||||||
|
cachedLastNodeAHash.ShouldBe(entry3.Hash);
|
||||||
|
|
||||||
|
IOplogStore rehydratedStore = harness.ReopenStore();
|
||||||
|
string? persistedLastNodeAHash = await rehydratedStore.GetLastEntryHashAsync("node-a");
|
||||||
|
persistedLastNodeAHash.ShouldBe(mergedEntry.Hash);
|
||||||
|
|
||||||
|
await rehydratedStore.DropAsync();
|
||||||
|
(await rehydratedStore.ExportAsync()).ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task OplogStore_DatasetIsolation_Works()
|
||||||
|
{
|
||||||
|
await using var harness = await CreateHarnessAsync();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
var primaryEntry = CreateOplogEntry("Users", "p1", "node-a", 100, 0, "");
|
||||||
|
var logsEntry = CreateOplogEntry("Users", "l1", "node-a", 100, 1, "");
|
||||||
|
|
||||||
|
await harness.AppendOplogEntryAsync(primaryEntry, DatasetId.Primary);
|
||||||
|
await harness.AppendOplogEntryAsync(logsEntry, DatasetId.Logs);
|
||||||
|
|
||||||
|
var primary = (await harness.ExportAsync(DatasetId.Primary)).ToList();
|
||||||
|
var logs = (await harness.ExportAsync(DatasetId.Logs)).ToList();
|
||||||
|
|
||||||
|
primary.Count.ShouldBe(1);
|
||||||
|
primary[0].DatasetId.ShouldBe(DatasetId.Primary);
|
||||||
|
|
||||||
|
logs.Count.ShouldBe(1);
|
||||||
|
logs[0].DatasetId.ShouldBe(DatasetId.Logs);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task OplogStore_GetChainRangeAsync_ReturnsOrderedLinkedRange()
|
||||||
|
{
|
||||||
|
await using var harness = await CreateHarnessAsync();
|
||||||
|
IOplogStore store = harness.Store;
|
||||||
|
|
||||||
|
var entry1 = CreateOplogEntry("Users", "k1", "node1", 1000, 0, "");
|
||||||
|
var entry2 = CreateOplogEntry("Users", "k2", "node1", 2000, 0, entry1.Hash);
|
||||||
|
var entry3 = CreateOplogEntry("Users", "k3", "node1", 3000, 0, entry2.Hash);
|
||||||
|
|
||||||
|
await store.AppendOplogEntryAsync(entry1);
|
||||||
|
await store.AppendOplogEntryAsync(entry2);
|
||||||
|
await store.AppendOplogEntryAsync(entry3);
|
||||||
|
|
||||||
|
var range = (await store.GetChainRangeAsync(entry1.Hash, entry3.Hash)).ToList();
|
||||||
|
|
||||||
|
range.Count.ShouldBe(2);
|
||||||
|
range[0].Hash.ShouldBe(entry2.Hash);
|
||||||
|
range[1].Hash.ShouldBe(entry3.Hash);
|
||||||
|
}
|
||||||
|
|
||||||
|
protected abstract Task<IOplogStoreContractHarness> CreateHarnessAsync();
|
||||||
|
|
||||||
|
protected static OplogEntry CreateOplogEntry(
|
||||||
|
string collection,
|
||||||
|
string key,
|
||||||
|
string nodeId,
|
||||||
|
long wall,
|
||||||
|
int logic,
|
||||||
|
string previousHash)
|
||||||
|
{
|
||||||
|
return new OplogEntry(
|
||||||
|
collection,
|
||||||
|
key,
|
||||||
|
OperationType.Put,
|
||||||
|
JsonSerializer.SerializeToElement(new { key }),
|
||||||
|
new HlcTimestamp(wall, logic, nodeId),
|
||||||
|
previousHash);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public interface IOplogStoreContractHarness : IAsyncDisposable
|
||||||
|
{
|
||||||
|
IOplogStore Store { get; }
|
||||||
|
|
||||||
|
IOplogStore ReopenStore();
|
||||||
|
|
||||||
|
Task AppendOplogEntryAsync(OplogEntry entry, string datasetId, CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
Task<IEnumerable<OplogEntry>> ExportAsync(string datasetId, CancellationToken cancellationToken = default);
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user