Commit Graph

6 Commits

Author SHA1 Message Date
Joseph Doherty
09252b8c79 feat(storage): add MsgBlock block-based message storage unit
MsgBlock is the unit of storage in the file store — a single append-only
block file containing sequentially written binary message records. Blocks
are sealed (read-only) when they reach a configurable byte-size limit.

Key features:
- Write: appends MessageRecord-encoded messages with auto-incrementing
  sequence numbers and configurable first sequence offset
- Read: positional I/O via RandomAccess.Read for concurrent reader safety
- Delete: soft-delete with on-disk persistence (re-encodes flags byte +
  checksum so deletions survive recovery)
- Recovery: rebuilds in-memory index by scanning block file using
  MessageRecord.MeasureRecord for record boundary detection
- Thread safety: ReaderWriterLockSlim allows concurrent reads during writes

Also adds MessageRecord.MeasureRecord() — computes a record's byte length
by parsing varint field headers without full decode, needed for sequential
record scanning during block recovery.

Reference: golang/nats-server/server/filestore.go:217-267 (msgBlock struct)

12 tests covering write, read, delete, seal, recovery, concurrency,
and custom sequence offsets.
2026-02-24 12:21:33 -05:00
Joseph Doherty
17731e2af5 feat(storage): add binary message record encoding (Go parity)
Add MessageRecord class with Encode/Decode for the binary wire format
used in JetStream file store blocks. Uses varint-encoded lengths,
XxHash64 checksums, and a flags byte for deletion markers.

Go reference: filestore.go:6720-6724, 8180-8250, 8770-8777

13 tests covering round-trip, headers, checksum validation, corruption
detection, varint encoding, deleted flag, empty/large payloads.
2026-02-24 12:12:15 -05:00
Joseph Doherty
3e972f217e feat: add FileStore 6-way permutation tests (Go testFileStoreAllPermutations parity)
Port 20 tests across the {NoCipher, ChaCha, Aes} x {NoCompression, S2Compression}
matrix from Go's testFileStoreAllPermutations (filestore_test.go:55), yielding 120
total Theory executions in FileStorePermutationTests.cs.

Also fix PushFrame.Subject property missing compilation error that was blocking
the test build (JetStreamConsumerDeliveryEdgeTests.cs:119).

Tests covered (each runs 6x):
  Store_and_load_basic                       TestFileStoreBasics:86
  Store_multiple_messages_load_by_sequence   TestFileStoreBasics:86
  LoadLastBySubject_returns_most_recent_for_subject
  Remove_single_message_updates_state        TestFileStoreBasics:129
  Purge_clears_all_messages                  TestFileStorePurge:710
  TrimToMaxMessages_enforces_limit           TestFileStoreMsgLimitBug:518
  Block_rotation_when_exceeding_block_size   TestFileStoreAndRetrieveMultiBlock:1527
  GetState_returns_correct_counts            TestFileStoreBasics:104
  Snapshot_and_restore_round_trip            TestFileStoreSnapshot:1799
  ListAsync_returns_ordered_messages         TestFileStoreTimeStamps:683
  MaxAge_prunes_expired_messages             TestFileStoreAgeLimit:616
  Recovery_after_reopen_preserves_messages   TestFileStoreBasicWriteMsgsAndRestore:181
  Large_payload_store_and_load               64 KiB random payload variant
  Multiple_subjects_filter_by_subject        TestFileStoreBasics multi-subject
  Sequential_writes_maintain_ordering        TestFileStoreSelectNextFirst:304
  Store_creates_files_on_disk                disk-presence variant
  Write_and_read_same_block                  TestFileStoreWriteAndReadSameBlock:1510
  Stored_messages_have_non_decreasing_timestamps TestFileStoreTimeStamps:683
  Remove_out_of_order_collapses_first_seq    TestFileStoreCollapseDmap:1561
  Snapshot_after_removes_preserves_remaining TestFileStoreSnapshot:1904
2026-02-24 06:43:48 -05:00
Joseph Doherty
9cc9888bb4 feat: add S2 compression and AEAD encryption for FileStore (Go parity)
Replace Deflate+XOR with IronSnappy S2 block compression and ChaCha20-Poly1305 / AES-256-GCM
AEAD encryption, matching golang/nats-server/server/filestore.go. Introduces FSV2 envelope
format alongside existing FSV1 for backward compatibility. Adds 55 new tests across
S2CodecTests, AeadEncryptorTests, and FileStoreV2Tests covering all 6 cipher×compression
permutations, tamper detection, and legacy format round-trips.
2026-02-24 06:29:34 -05:00
Joseph Doherty
3ff801865a feat: Waves 3-5 — FileStore, RAFT, JetStream clustering, and concurrency tests
Add comprehensive Go-parity test coverage across 3 subsystems:
- FileStore: basic CRUD, limits, purge, recovery, subjects, encryption,
  compression, MemStore (161 tests, 24 skipped for not-yet-implemented)
- RAFT: core types, wire format, election, log replication, snapshots
  (95 tests)
- JetStream Clustering: meta controller, stream/consumer replica groups,
  concurrency stress tests (90 tests)

Total: ~346 new test annotations across 17 files (+7,557 lines)
Full suite: 2,606 passing, 0 failures, 27 skipped
2026-02-23 22:55:41 -05:00
Joseph Doherty
28d379e6b7 feat: phase B distributed substrate test parity — 39 new tests across 5 subsystems
FileStore basics (4), MemStore/retention (10), RAFT election/append (16),
config reload parity (3), monitoring endpoints varz/connz/healthz (6).
972 total tests passing, 0 failures.
2026-02-23 19:41:30 -05:00