Replace BLite with Surreal embedded persistence
All checks were successful
NuGet Package Publish / nuget (push) Successful in 1m21s
All checks were successful
NuGet Package Publish / nuget (push) Successful in 1m21s
This commit is contained in:
@@ -9,6 +9,7 @@ CBDDC supports multiple persistence backends to suit different deployment scenar
|
||||
| **SQLite (Direct)** | Embedded apps, single-node | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ✅ Yes |
|
||||
| **EF Core (Generic)** | Multi-DB support, migrations | ⭐⭐⭐ | ⭐⭐⭐ | ✅ Yes |
|
||||
| **PostgreSQL** | Production, high load, JSON queries | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ✅ Yes |
|
||||
| **Surreal Embedded (RocksDB)** | Embedded multi-peer sync with local CDC | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ✅ Yes |
|
||||
|
||||
## SQLite (Direct)
|
||||
|
||||
@@ -179,19 +180,68 @@ Host=prod-db.example.com;Database=CBDDC;Username=admin;Password=secret;SSL Mode=
|
||||
Host=localhost;Database=CBDDC;Username=admin;Password=secret;Pooling=true;Minimum Pool Size=5;Maximum Pool Size=100
|
||||
```
|
||||
|
||||
## Surreal Embedded (RocksDB)
|
||||
|
||||
**Package:** `ZB.MOM.WW.CBDDC.Persistence`
|
||||
|
||||
### Characteristics
|
||||
|
||||
- ✅ **Embedded + durable**: Uses local RocksDB storage via Surreal embedded endpoint
|
||||
- ✅ **CDC-native workflow**: Collection watches emit oplog entries and metadata updates
|
||||
- ✅ **Durable checkpointing**: CDC cursor state is persisted per consumer id
|
||||
- ✅ **Restart recovery**: Oplog + checkpoint data survive process restart and resume catch-up
|
||||
- ✅ **Loopback suppression**: Remote apply path suppresses local CDC re-emission
|
||||
- ✅ **Idempotent merge window**: Duplicate remote entries are merged by deterministic hash
|
||||
|
||||
### When to Use
|
||||
|
||||
- Embedded deployments that still need multi-peer replication
|
||||
- Edge nodes where local durability is required without an external DB server
|
||||
- CDC-heavy sync topologies that need restart-safe cursor tracking
|
||||
- Environments that benefit from document-style storage and local operation logs
|
||||
|
||||
### Configuration
|
||||
|
||||
```csharp
|
||||
services.AddCBDDCCore()
|
||||
.AddCBDDCSurrealEmbedded<SampleDocumentStore>(_ => new CBDDCSurrealEmbeddedOptions
|
||||
{
|
||||
Endpoint = "rocksdb://local",
|
||||
DatabasePath = "/var/lib/cbddc/node-a.rocksdb",
|
||||
Namespace = "cbddc",
|
||||
Database = "node_a",
|
||||
Cdc = new CBDDCSurrealCdcOptions
|
||||
{
|
||||
Enabled = true,
|
||||
ConsumerId = "sync-main",
|
||||
CheckpointTable = "cbddc_cdc_checkpoint",
|
||||
EnableLiveSelectAccelerator = true,
|
||||
LiveSelectReconnectDelay = TimeSpan.FromSeconds(2)
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### CDC Durability Notes
|
||||
|
||||
1. **Checkpoint semantics**: each consumer id has an independent durable cursor (`timestamp + hash`).
|
||||
2. **Catch-up on restart**: read checkpoint, then request oplog entries strictly after the stored timestamp.
|
||||
3. **Duplicate-window safety**: replayed windows are deduplicated by oplog hash merge semantics.
|
||||
4. **Delete durability**: deletes persist as oplog delete operations plus tombstone metadata.
|
||||
5. **Remote apply behavior**: remote sync applies documents without generating local loopback CDC entries.
|
||||
|
||||
## Feature Comparison
|
||||
|
||||
| Feature | SQLite (Direct) | EF Core | PostgreSQL |
|
||||
|---------|----------------|---------|------------|
|
||||
| **Storage Format** | File-based | Varies | Server-based |
|
||||
| **JSON Storage** | TEXT | NVARCHAR/TEXT | JSONB |
|
||||
| **JSON Indexing** | Standard | Standard | GIN/GIST |
|
||||
| **JSON Queries** | `json_extract()` | In-Memory | Native (future) |
|
||||
| **Concurrent Writes** | Good (WAL) | Varies | Excellent |
|
||||
| **Horizontal Scaling** | No | Limited | Yes (replication) |
|
||||
| **Migrations** | Manual SQL | EF Migrations | EF Migrations |
|
||||
| **Connection Pooling** | N/A | Built-in | Built-in |
|
||||
| **Cloud Support** | N/A | Varies | Excellent |
|
||||
| Feature | SQLite (Direct) | EF Core | PostgreSQL | Surreal Embedded |
|
||||
|---------|----------------|---------|------------|------------------|
|
||||
| **Storage Format** | File-based | Varies | Server-based | File-based (RocksDB) |
|
||||
| **JSON Storage** | TEXT | NVARCHAR/TEXT | JSONB | Native document records |
|
||||
| **JSON Indexing** | Standard | Standard | GIN/GIST | Table/index schema controls |
|
||||
| **JSON Queries** | `json_extract()` | In-Memory | Native (future) | Native document querying |
|
||||
| **Concurrent Writes** | Good (WAL) | Varies | Excellent | Good (embedded engine limits apply) |
|
||||
| **Horizontal Scaling** | No | Limited | Yes (replication) | Peer replication via CBDDC sync |
|
||||
| **Migrations** | Manual SQL | EF Migrations | EF Migrations | Schema initializer + scripts |
|
||||
| **Connection Pooling** | N/A | Built-in | Built-in | N/A (embedded) |
|
||||
| **Cloud Support** | N/A | Varies | Excellent | Excellent for edge/embedded nodes |
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
@@ -251,6 +301,10 @@ _*Benchmarks vary based on hardware, network, and configuration_
|
||||
- **Use**: PostgreSQL
|
||||
- **Why**: Best performance, scalability, reliability
|
||||
|
||||
### Production (Edge / Embedded Mesh)
|
||||
- **Use**: Surreal Embedded (RocksDB)
|
||||
- **Why**: Durable local CDC, restart-safe checkpoint resume, no external DB dependency
|
||||
|
||||
### Enterprise
|
||||
- **Use**: EF Core with SQL Server or PostgreSQL
|
||||
- **Why**: Enterprise support, compliance, familiarity
|
||||
@@ -272,6 +326,15 @@ _*Benchmarks vary based on hardware, network, and configuration_
|
||||
- Check for connection leaks
|
||||
- Consider connection pooler (PgBouncer)
|
||||
|
||||
### Surreal Embedded: "CDC replay after restart"
|
||||
- Ensure `Cdc.Enabled=true` and a stable `Cdc.ConsumerId` is configured
|
||||
- Verify checkpoint table contains cursor state for the consumer
|
||||
- Resume from checkpoint timestamp before requesting new oplog window
|
||||
|
||||
### Surreal Embedded: "Unexpected loopback oplog on remote sync"
|
||||
- Apply remote entries through CBDDC sync/orchestrator paths (not local collection writes)
|
||||
- Keep remote sync guards enabled in document store implementations
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **JSONB Query Translation**: Native PostgreSQL JSON queries from QueryNode
|
||||
|
||||
Reference in New Issue
Block a user