Commit Graph

13 Commits

Author SHA1 Message Date
Joseph Doherty
d0068b121f feat(storage): add write cache and TTL scheduling (Go parity)
Add MsgBlock write cache (mirrors Go's msgBlock.cache) to serve reads
for recently-written records without disk I/O; cleared on block seal via
RotateBlock. Add HashWheel-based TTL expiry in FileStore (ExpireFromWheel /
RegisterTtl), replacing the O(n) linear scan on every append with an
O(expired) wheel scan. Implement StoreMsg sync method with per-message TTL
override support. Add 10 tests covering cache hits/eviction, wheel expiry,
retention, StoreMsg seq/ts, per-msg TTL, and recovery re-registration.
2026-02-24 13:54:37 -05:00
Joseph Doherty
b0b64292b3 feat(storage): add tombstone tracking and purge operations (Go parity)
Implement PurgeEx, Compact, Truncate, FilteredState, SubjectsState,
SubjectsTotals, State, FastState, GetSeqFromTime on FileStore. Add
MsgBlock.IsDeleted, DeletedSequences, EnumerateNonDeleted. Includes
wildcard subject support via SubjectMatch for all filtered operations.
2026-02-24 13:45:17 -05:00
Joseph Doherty
2816e8f048 feat(storage): rewrite FileStore to use block-based MsgBlock storage
Replace JSONL persistence with real MsgBlock-based block files (.blk).
FileStore now acts as a block manager that creates, seals, and rotates
MsgBlocks while maintaining an in-memory cache for fast reads/queries.

Key changes:
- AppendAsync writes transformed payloads to MsgBlock via WriteAt
- Block rotation occurs when active block reaches size limit
- Recovery scans .blk files and rebuilds in-memory state from records
- Legacy JSONL migration: existing messages.jsonl data is automatically
  converted to block files on first open, then JSONL is deleted
- PurgeAsync disposes and deletes all block files
- RewriteBlocks rebuilds blocks from cache (used by trim/restore)
- InvalidDataException propagates during recovery (wrong encryption key)

MsgBlock.WriteAt added to support explicit sequence numbers and timestamps,
needed when rewriting blocks with non-contiguous sequences (after removes).

Tests updated:
- New FileStoreBlockTests.cs with 9 tests for block-specific behavior
- JetStreamFileStoreCompressionEncryptionParityTests updated to read
  FSV1 magic from .blk files instead of messages.jsonl
- JetStreamFileStoreDurabilityParityTests updated to verify .blk files
  instead of index.manifest.json

All 3,562 tests pass (3,535 passed + 27 skipped, 0 failures).
2026-02-24 12:39:32 -05:00
Joseph Doherty
2c9683e7aa feat: upgrade JetStreamService to lifecycle orchestrator
Implements enableJetStream() semantics from golang/nats-server/server/jetstream.go:414-523.

- JetStreamService.StartAsync(): validates config, creates store directory
  (including nested paths via Directory.CreateDirectory), registers all
  $JS.API.> subjects, logs startup stats; idempotent on double-start
- JetStreamService.DisposeAsync(): clears registered subjects, marks not running
- New properties: RegisteredApiSubjects, MaxStreams, MaxConsumers, MaxMemory, MaxStore
- JetStreamOptions: adds MaxStreams and MaxConsumers limits (0 = unlimited)
- FileStoreConfig: removes duplicate StoreCipher/StoreCompression enum declarations
  now that AeadEncryptor.cs owns them; updates defaults to NoCipher/NoCompression
- FileStoreOptions/FileStore: align enum member names with AeadEncryptor.cs
  (NoCipher, NoCompression, S2Compression) to fix cross-task naming conflict
- 13 new tests in JetStreamServiceOrchestrationTests covering all lifecycle paths
2026-02-24 06:03:46 -05:00
Joseph Doherty
921554f410 feat: define StreamStore/ConsumerStore interfaces from Go store.go
Port IStreamStore, IConsumerStore, StoreMsg, StreamState, SimpleState,
ConsumerState, FileStoreConfig, StoreCipher, StoreCompression types.
Rename Models.StreamState → ApiStreamState to avoid namespace conflict.
2026-02-23 21:06:16 -05:00
Joseph Doherty
56177a7099 feat: enforce filestore durability and recovery invariants 2026-02-23 14:51:30 -05:00
Joseph Doherty
377ad4a299 feat: complete jetstream deep operational parity closure 2026-02-23 13:43:14 -05:00
Joseph Doherty
2b64d762f6 feat: execute full-repo remaining parity closure plan 2026-02-23 13:08:52 -05:00
Joseph Doherty
b41e6ff320 feat: execute post-baseline jetstream parity plan 2026-02-23 12:11:19 -05:00
Joseph Doherty
8bce096f55 feat: complete final jetstream parity transport and runtime baselines 2026-02-23 11:04:43 -05:00
Joseph Doherty
f46b331921 feat: complete remaining jetstream parity implementation plan 2026-02-23 10:16:16 -05:00
Joseph Doherty
d73e7e2f88 feat: enforce jetstream retention and limits 2026-02-23 06:04:23 -05:00
Joseph Doherty
788f4254b0 feat: implement jetstream filestore recovery baseline 2026-02-23 06:00:42 -05:00