feat(storage): rewrite FileStore to use block-based MsgBlock storage
Replace JSONL persistence with real MsgBlock-based block files (.blk). FileStore now acts as a block manager that creates, seals, and rotates MsgBlocks while maintaining an in-memory cache for fast reads/queries. Key changes: - AppendAsync writes transformed payloads to MsgBlock via WriteAt - Block rotation occurs when active block reaches size limit - Recovery scans .blk files and rebuilds in-memory state from records - Legacy JSONL migration: existing messages.jsonl data is automatically converted to block files on first open, then JSONL is deleted - PurgeAsync disposes and deletes all block files - RewriteBlocks rebuilds blocks from cache (used by trim/restore) - InvalidDataException propagates during recovery (wrong encryption key) MsgBlock.WriteAt added to support explicit sequence numbers and timestamps, needed when rewriting blocks with non-contiguous sequences (after removes). Tests updated: - New FileStoreBlockTests.cs with 9 tests for block-specific behavior - JetStreamFileStoreCompressionEncryptionParityTests updated to read FSV1 magic from .blk files instead of messages.jsonl - JetStreamFileStoreDurabilityParityTests updated to verify .blk files instead of index.manifest.json All 3,562 tests pass (3,535 passed + 27 skipped, 0 failures).
This commit is contained in:
@@ -23,10 +23,10 @@ public class JetStreamFileStoreDurabilityParityTests
|
||||
await store.AppendAsync("orders.created", Encoding.UTF8.GetBytes($"payload-{i}"), default);
|
||||
}
|
||||
|
||||
File.Exists(Path.Combine(dir, options.IndexManifestFileName)).ShouldBeTrue();
|
||||
// Block-based storage: .blk files should be present on disk.
|
||||
Directory.GetFiles(dir, "*.blk").Length.ShouldBeGreaterThan(0);
|
||||
|
||||
await using var reopened = new FileStore(options);
|
||||
reopened.UsedIndexManifestOnStartup.ShouldBeTrue();
|
||||
var state = await reopened.GetStateAsync(default);
|
||||
state.Messages.ShouldBe((ulong)1000);
|
||||
reopened.BlockCount.ShouldBeGreaterThan(1);
|
||||
|
||||
Reference in New Issue
Block a user