Now that MemStore uses Unix epoch timestamps (13a3f81), restore the
original Go MaxAge values that were previously omitted as workarounds:
- JsCluster2: MaxAgeMs=500 (Go: 500ms)
- JsCluster34: MaxAgeMs=5000 (Go: 5s)
DateTime.UtcNow.Ticks * 100L overflows long (Ticks ~6.38e17, x100 =
6.38e19 > long.MaxValue 9.22e18), producing corrupted timestamps that
cause PruneExpiredMessages to immediately expire all messages when
MaxAgeMs is set. Use Unix epoch offset to match Go's time.Now().UnixNano().
Port 36 Go-parity tests covering redelivery, NAK, push consumer
heartbeats, interest retention with wildcards, ack-all with large
first seq, deliver last-per-subject, work-in-progress acks, flow
control stall, KV operations, multi-account isolation, and subject
transforms.
Go refs: TestJetStreamRedeliverAndLateAck, TestJetStreamInterestRetention,
TestJetStreamKVDelete, TestJetStreamMultipleAccountsBasics, and 23+ more.
Port Go consumer_test.go and jetstream_test.go consumer lifecycle tests.
Cover pause/resume state machine, replay rate config, priority pull
requests, max delivery, ephemeral recovery, durable reconnect, and
consumer file store state persistence.
49 new tests ported from consumer_test.go and jetstream_test.go.
Port Go jetstream_test.go storage/recovery/encryption tests. Add stream
recovery stubs, encryption key management, direct-get with time queries,
and subject delete marker handling.
23 new tests ported from jetstream_test.go.
Port Go store_test.go contract tests for IStreamStore interface using
MemStore. Fix CompactInternal to correctly walk forward to find new
FirstSeq before backward cleanup (matching Go behavior).
21 new tests ported from store_test.go.
Port Go filestore tombstone/deletion tests, consumer state encode/decode,
consumer file store persistence, and message TTL enforcement. Adds
ConsumerStateCodec and ConsumerFileStore implementations.
17 new tests ported from filestore_test.go.
Ports 34 Go FileStore tests from filestore_test.go to
FileStoreRecovery2Tests.cs (31 pass, 4 skipped). Tests cover block
recovery, compaction, PSIM indexing, skip-msg handling, TTL expiry,
corrupt index/state detection, and read-only permission checks.
Updates docs/test_parity.db with mapped/skipped status for all 34 tests.
Extended the BecomeLeader() fix to JetStreamClusterFixture,
ClusterFailoverFixture, and LeaderFailoverParityFixture. All three
fixtures now auto-simulate leader election after stepdown, matching
the MetaControllerFixture fix from the previous commit.
Re-enabled the leader check in JetStreamApiRouter.Route() that was
commented out during B8 agent work. Added BecomeLeader() method to
JetStreamMetaGroup for single-process test fixtures to simulate
winning the post-stepdown election. MetaControllerFixture now
auto-calls BecomeLeader() after a successful meta leader stepdown.
- RaftGroup, StreamAssignment, ConsumerAssignment types matching Go structs
(jetstream_cluster.go:154-266)
- PlacementEngine.SelectPeerGroup: topology-aware peer selection with cluster
affinity, tag filtering, exclude tags, and storage-weighted sorting
(Go ref: selectPeerGroup at line 7212)
- JetStreamMetaGroup: backward-compatible rewrite with full assignment tracking,
consumer proposal workflow, and delete operations
- 41 new tests in ClusterAssignmentAndPlacementTests
Add MsgBlock write cache (mirrors Go's msgBlock.cache) to serve reads
for recently-written records without disk I/O; cleared on block seal via
RotateBlock. Add HashWheel-based TTL expiry in FileStore (ExpireFromWheel /
RegisterTtl), replacing the O(n) linear scan on every append with an
O(expired) wheel scan. Implement StoreMsg sync method with per-message TTL
override support. Add 10 tests covering cache hits/eviction, wheel expiry,
retention, StoreMsg seq/ts, per-msg TTL, and recovery re-registration.
Implement PurgeEx, Compact, Truncate, FilteredState, SubjectsState,
SubjectsTotals, State, FastState, GetSeqFromTime on FileStore. Add
MsgBlock.IsDeleted, DeletedSequences, EnumerateNonDeleted. Includes
wildcard subject support via SubjectMatch for all filtered operations.
Replace JSONL persistence with real MsgBlock-based block files (.blk).
FileStore now acts as a block manager that creates, seals, and rotates
MsgBlocks while maintaining an in-memory cache for fast reads/queries.
Key changes:
- AppendAsync writes transformed payloads to MsgBlock via WriteAt
- Block rotation occurs when active block reaches size limit
- Recovery scans .blk files and rebuilds in-memory state from records
- Legacy JSONL migration: existing messages.jsonl data is automatically
converted to block files on first open, then JSONL is deleted
- PurgeAsync disposes and deletes all block files
- RewriteBlocks rebuilds blocks from cache (used by trim/restore)
- InvalidDataException propagates during recovery (wrong encryption key)
MsgBlock.WriteAt added to support explicit sequence numbers and timestamps,
needed when rewriting blocks with non-contiguous sequences (after removes).
Tests updated:
- New FileStoreBlockTests.cs with 9 tests for block-specific behavior
- JetStreamFileStoreCompressionEncryptionParityTests updated to read
FSV1 magic from .blk files instead of messages.jsonl
- JetStreamFileStoreDurabilityParityTests updated to verify .blk files
instead of index.manifest.json
All 3,562 tests pass (3,535 passed + 27 skipped, 0 failures).
MsgBlock is the unit of storage in the file store — a single append-only
block file containing sequentially written binary message records. Blocks
are sealed (read-only) when they reach a configurable byte-size limit.
Key features:
- Write: appends MessageRecord-encoded messages with auto-incrementing
sequence numbers and configurable first sequence offset
- Read: positional I/O via RandomAccess.Read for concurrent reader safety
- Delete: soft-delete with on-disk persistence (re-encodes flags byte +
checksum so deletions survive recovery)
- Recovery: rebuilds in-memory index by scanning block file using
MessageRecord.MeasureRecord for record boundary detection
- Thread safety: ReaderWriterLockSlim allows concurrent reads during writes
Also adds MessageRecord.MeasureRecord() — computes a record's byte length
by parsing varint field headers without full decode, needed for sequential
record scanning during block recovery.
Reference: golang/nats-server/server/filestore.go:217-267 (msgBlock struct)
12 tests covering write, read, delete, seal, recovery, concurrency,
and custom sequence offsets.
Add MessageRecord class with Encode/Decode for the binary wire format
used in JetStream file store blocks. Uses varint-encoded lengths,
XxHash64 checksums, and a flags byte for deletion markers.
Go reference: filestore.go:6720-6724, 8180-8250, 8770-8777
13 tests covering round-trip, headers, checksum validation, corruption
detection, varint encoding, deleted flag, empty/large payloads.
Add 60 tests in JsClusterConsumerReplicationTests covering consumer
creation, fetch/delivery, ack tracking, leader failover, state
consistency, and edge cases. Ported from Go jetstream_cluster_2_test.go.
Adds two new test files covering Task 10 of the full Go parity plan:
JsClusterAdvancedTests.cs (27 tests):
- Large 7-node cluster with R5 stream
- Stream with 20+ subjects and wildcard '>' subject
- 1000-message publish to R3 and R1 streams
- Stream state accuracy after 1000 messages
- 10 streams with mixed replica counts
- Create/publish/delete/recreate cycle (3x)
- Consumer on 1000-message stream with batch fetch
- AckAll for all 1000 messages
- Stream info consistency after 50 interleaved ops
- Meta state after creating and deleting 10 streams
- 5 independent consumers with correct pending counts
- Consumer with wildcard filter subject
- Stream update adding subjects after publishes
- Stream purge then republish
- Fetch empty after purge
- Stream delete cascades consumer removal
- Node removal preserves data reads
- Node restart lifecycle markers
- Leader stepdown with monotonic sequence verification
- Stream info after stepdown with 1000 messages
JsClusterLongRunningTests.cs (15 tests, [Trait("Category", "LongRunning")]):
- 5000 messages in R3 stream maintain consistency
- 100 sequential fetches of 50 messages each
- 50 consumers on same stream all see all messages
- 20 streams in 5-node cluster all independent
- Publish-ack-fetch cycle 100 times
- 10 stepdowns during continuous publishing
- Alternating publish and stepdown (20 iterations)
- Create-publish-delete 20 streams sequentially
- Consumer ack tracking after 10 leader failovers
- Fetch with batch=1 iterated 500 times
- Mixed operations across 5 streams
- Rapid meta stepdowns (20) with version verification
- 10000 small messages in R1 stream
- Stream with max_messages=100 enforces limit after 1000 publishes
- Consumer on max-messages stream tracks correct pending
All 42 tests pass (27 advanced + 15 long-running).
Adds 97 tests across two new files covering stream replication semantics
(R1/R3 creation, replica group size, publish preservation, state accuracy,
purge, update, delete, max limits, subjects, wildcards, storage type) and
placement semantics (replica caps at cluster size, various cluster sizes,
concurrent creation, stepdown resilience, long names, re-create after delete).
Adds a unified JetStreamClusterFixture consolidating the capabilities of all 7
per-suite fixtures (ClusterFormationFixture, ClusterStreamFixture, LeaderFailoverFixture, etc.)
into a single reusable helper for Tasks 6-10. Includes new Go-parity helpers
(WaitOnStreamLeaderAsync, WaitOnConsumerLeaderAsync, GetConsumerLeaderId,
StepDownMetaLeader, SimulateNodeRestart, RemoveNode) matching jetstream_helpers_test.go.
27 smoke tests verify all capabilities pass.
Replace Deflate+XOR with IronSnappy S2 block compression and ChaCha20-Poly1305 / AES-256-GCM
AEAD encryption, matching golang/nats-server/server/filestore.go. Introduces FSV2 envelope
format alongside existing FSV1 for backward compatibility. Adds 55 new tests across
S2CodecTests, AeadEncryptorTests, and FileStoreV2Tests covering all 6 cipher×compression
permutations, tamper detection, and legacy format round-trips.
Stream lifecycle, publish/ack, consumer delivery, retention policy,
API endpoints, cluster formation, and leader failover tests ported
from Go nats-server reference. 1006 total tests passing.
