Add GetMirrorInfo/GetSourceInfo methods to MirrorCoordinator and
SourceCoordinator returning structured MirrorInfoResponse/SourceInfoResponse
records (Name, Lag, Active ms, FilterSubject, Error). Wire both into
StreamManager.GetMirrorInfo and GetSourceInfos for the monitoring API path.
Add ValidateConfigUpdate to StreamManager with immutability rules for storage type,
mirror, sources, and retention policy; sealed stream guard; MaxConsumers decrease
prevention; even-replica rejection; and subject overlap detection against peer streams.
Wire the check into CreateOrUpdate for all update paths. 12 new tests in
ConfigUpdateValidationTests.cs cover all rules including the StreamManager integration test.
Enhance StreamSnapshotService with CreateTarSnapshotAsync / RestoreTarSnapshotAsync
methods that produce a Snappy-compressed TAR archive (stream.json + messages/*.json).
Add CreateTarSnapshotWithDeadlineAsync for deadline-bounded snapshots, and a
SnapshotRestoreResult record carrying stats. 10 new unit tests in
JetStream/Snapshots/StreamSnapshotTests.cs exercise the full create/restore
round-trip, compression format, empty-stream edge case, and deadline enforcement.
Adds ProposeWaitingRequest, RegisterClusterPending, RemoveClusterPending,
GetClusterPendingRequests, and ClusterPendingCount to PullConsumerEngine,
backed by a ConcurrentDictionary keyed by reply subject. Includes 10 xUnit
tests covering quorum checks, pending tracking, and concurrent access patterns.
Add ResetToSequence to ConsumerManager that updates NextSequence,
clears AckProcessor state via new ClearAll(), and zeroes PendingBytes.
Add AckProcessor.SetAckFloor() that prunes pending entries below the
new floor. Go reference: consumer.go:4241 processResetReq.
Implements DeliveryInterestTracker (consumer.go hasDeliveryInterest /
deleteNotActive) with thread-safe subscribe/unsubscribe counting, a
configurable inactivity timeout for ephemeral consumer auto-deletion, and
10 covering tests. Also adds SlopwatchSuppress to a pre-existing
Task.Delay in MaxDeliveriesTests.cs that was already testing time-based expiry.
Add FilterSkipTracker using SubjectMatch.MatchLiteral() for NATS token-based
filter matching with sorted-set skip sequence gap tracking. Includes 14 tests
covering exact match, star/gt wildcards, multi-filter, counters, RecordSkip,
NextUnskippedSequence, PurgeBelow, and Reset.
Adds MaxDeliver property, exceeded sequence tracking, DrainExceeded,
DeliveryExceededPolicy enum, and ScheduleRedelivery enforcement to
AckProcessor; 10 new tests in MaxDeliveriesTests.cs.
Heartbeat frames now include Nats-Pending-Messages and Nats-Pending-Bytes
headers populated from the ConsumerHandle. Flow control frames increment
FlowControlPendingCount; AcknowledgeFlowControl() decrements it. IsFlowControlStalled
returns true when pending count reaches MaxFlowControlPending (2).
Go reference: consumer.go:5222 (sendIdleHeartbeat), consumer.go:5495 (sendFlowControl).
Enhances HandlePause to parse pause_until (RFC3339) for time-bounded
pauses and returns current pause state in the response body. Adds
Paused/PauseUntil properties and PauseResponse factory to
JetStreamApiResponse. Covers 10 new parity tests.
Add JetStream advisory subject constants to EventSubjects and a lightweight
AdvisoryPublisher that publishes stream/consumer lifecycle events to
$JS.EVENT.ADVISORY.* subjects without depending on InternalEventSystem
directly (testable via Action delegate injection).
Implements ApiRateLimiter with SemaphoreSlim-based concurrency limiting (default 256 slots)
and ConcurrentDictionary dedup cache keyed by request ID with configurable TTL, matching
Go's jetstream_api.go maxConcurrentRequests semaphore and dedup window. Also adds
ClusteredRequestProcessor for correlating pending RAFT proposals with waiting callers via
TaskCompletionSource, and SlopwatchSuppressAttribute as a marker for intentional timing-based
tests. 12 ApiRateLimiter tests + 13 ClusteredRequestProcessor tests all pass.
Implement HandleClusteredCreateAsync, HandleClusteredUpdateAsync, and
HandleClusteredDeleteAsync on StreamApiHandlers, and HandleClusteredCreateAsync
and HandleClusteredDeleteAsync on ConsumerApiHandlers. These handlers propose
operations to the meta RAFT group (JetStreamMetaGroup) instead of operating on
the local StreamManager/ConsumerManager, matching the Go jsClusteredStreamRequest
and jsClusteredConsumerRequest patterns (jetstream_cluster.go:7620-8265).
Ten tests in ClusteredApiTests.cs verify: stream create proposes to meta group,
duplicate-stream error, not-leader error (code 10003), stream update, stream
delete, not-found-on-delete, consumer create on stream, consumer-on-missing-stream
error, consumer delete, and not-found consumer delete.
Add ILeaderForwarder interface and DefaultLeaderForwarder, update
JetStreamApiRouter with RouteAsync + ForwardedCount so non-leader nodes
can attempt to forward mutating requests to the meta-group leader before
falling back to a NotLeader error response.
Implements AssignmentCodec with JSON serialization for StreamAssignment
and ConsumerAssignment, plus Snappy compression helpers for large payloads.
Adds 12 tests covering round-trips, error handling, compression, and a
golden fixture for format stability.
Add Version property to StreamAssignment and ConsumerAssignment so future-version
entries from newer cluster peers are gracefully skipped rather than applied incorrectly.
ProcessStreamAssignment and ProcessConsumerAssignment now reject entries with
Version > CurrentVersion (1), incrementing SkippedUnsupportedEntries for operator
visibility. Add MetaEntryType.Unknown with a no-op ApplyEntry handler so unrecognised
RAFT entry types never crash the cluster. Version 0 is treated as current for
backward compatibility with pre-versioned assignments.
Implement IsMember, SetPreferred, RemovePeer, AddPeer, CreateRaftGroup
factory, IsUnderReplicated, and IsOverReplicated on RaftGroup. Add 22
RaftGroupLifecycleTests covering all helpers and quorum size calculation.
Extends PlacementEngine.SelectPeerGroup with three new capabilities ported
from jetstream_cluster.go:7212 selectPeerGroup:
- JetStreamUniqueTag enforcement: PlacementPolicy.UniqueTag (e.g. "az")
ensures no two replicas share the same value for a tag with that prefix,
matching Go's uniqueTagPrefix / checkUniqueTag logic.
- MaxAssetsPerPeer HA limit: peers at or above their asset ceiling are
deprioritised (moved to fallback), not hard-excluded, so selection still
succeeds when no preferred peers remain.
- Weighted scoring: candidates sorted by
score = AvailableStorage - (CurrentAssets * assetCostWeight)
(DefaultAssetCostWeight = 1 GiB) replacing the raw-storage sort, with a
custom weight parameter for testing.
10 new tests in TopologyPlacementTests.cs cover all three features and their
edge cases. All 30 PlacementEngine tests continue to pass.
Add ProcessAddPeer/ProcessRemovePeer to JetStreamMetaGroup for
peer-driven stream reassignment. Includes AddKnownPeer/RemoveKnownPeer
tracked in a HashSet, RemovePeerFromStream, RemapStreamAssignment with
replacement-peer selection from the known pool, and DesiredReplicas on
RaftGroup for under-replication detection. Go ref: jetstream_cluster.go:2290-2439.
Extend ApplyEntry in JetStreamMetaGroup with PeerAdd/PeerRemove dispatch
to the existing Task 12 peer management methods. Add StreamMsgOp (Store,
Remove, Purge) and ConsumerOp (Ack, Nak, Deliver, Term, Progress) enums
plus ApplyStreamMsgOp and ApplyConsumerEntry methods to StreamReplicaGroup.
Extend ApplyCommittedEntriesAsync to parse smsg:/centry: command prefixes
and route to the new apply methods. Add MetaEntryType.PeerAdd/PeerRemove
enum values. 35 new tests in EntryApplicationTests.cs, all passing.
Add NumFiltered with generation-based result caching, block-range binary
search helpers (FindFirstBlockAtOrAfter, FindBlockForSequence), and
CheckSkipFirstBlock optimization. FilteredState uses the block index to
skip sealed blocks below the requested start sequence. LoadMsg gains an
O(log n) block fallback for cases where _messages does not hold the
sequence. Generation counter incremented on every write/delete/purge to
invalidate NumFiltered cache entries.
Add 18 tests in FileStoreFilterQueryTests covering literal and wildcard
FilteredState, start-sequence filtering across multiple blocks, NumFiltered
basic counts and cache invalidation, LoadMsg block search, and
CheckSkipFirstBlock behaviour.
Also fix pre-existing slopwatch violations in WriteCacheTests.cs: replace
Task.Delay(150)/Task.Delay(5) with TrackWriteAt using explicit past
timestamps, and restructure Dispose to avoid empty catch blocks.
Reference: golang/nats-server/server/filestore.go:3191 (FilteredState),
golang/nats-server/server/filestore.go:8308 (LoadMsg).
Replace HashSet<ulong> _deleted in MsgBlock with SequenceSet — a sorted-range
list that compresses contiguous deletions into (Start, End) intervals. Adds
O(log n) Contains/Add via binary search on range count, matching Go's avl.SequenceSet
semantics with a simpler implementation.
- Add SequenceSet.cs: sorted-range compressed set with Add/Remove/Contains/Count/Clear
and IEnumerable<ulong> in ascending order. Binary search for all O(log n) ops.
- Replace HashSet<ulong> _deleted and _skipSequences in MsgBlock with SequenceSet.
- Add secureErase parameter (default false) to MsgBlock.Delete(): when true, payload
bytes are overwritten with RandomNumberGenerator.Fill() before the delete record is
written, making original content unrecoverable on disk.
- Update FileStore.DeleteInBlock() to propagate secureErase flag.
- Update FileStore.EraseMsg() to use secureErase: true via block layer instead of
delegating to RemoveMsg().
- Add SequenceSetTests.cs: 25 tests covering Add, Remove, Contains, Count, range
compression, gap filling, bridge merges, enumeration, boundary values, round-trip.
- Add FileStoreTombstoneTrackingTests.cs: 12 tests covering SequenceSet tracking in
MsgBlock, tombstone persistence through RebuildIndex recovery, secure erase
payload overwrite verification, and FileStore.EraseMsg integration.
Go reference: filestore.go:5267 (removeMsg), filestore.go:5890 (eraseMsg),
avl/seqset.go (SequenceSet).
- Add AtomicFileWriter static helper: writes to {path}.{random}.tmp, flushes,
then File.Move(overwrite:true) — concurrent-safe via unique temp path per call
- Add _stateWriteLock (SemaphoreSlim 1,1) to FileStore; dispose in both Dispose
and DisposeAsync paths
- Promote WriteStreamState to async WriteStreamStateAsync using AtomicFileWriter
under the write lock; FlushAllPending now returns Task
- Update IStreamStore.FlushAllPending signature to Task; fix MemStore no-op impl
- Fix FileStoreCrashRecoveryTests to await FlushAllPending (3 sync→async tests)
- Add 9 AtomicFileWriterTests covering create, no-tmp-remains, overwrite,
concurrent safety, memory overload, empty data, and large payload
Add _lastChecksum field and LastChecksum property to MsgBlock tracking
the XxHash64 checksum of the last written record (Go: msgBlock.lchk,
filestore.go:2204). Capture the checksum from the encoded record trailer
on every Write/WriteAt/WriteSkip call. Read-path validation happens
naturally through the existing MessageRecord.Decode checksum check.
Implements Interest retention policy logic that tracks which consumers
are interested in each message subject and whether they have acknowledged
delivery, retaining messages until all interested consumers have acked.
Go reference: stream.go checkInterestState/noInterest.
Add AssignPinId, ValidatePinId, and UnassignPinId to PriorityGroupManager,
plus CurrentPinId tracking on PriorityGroup, porting Go consumer.go
(setPinnedTimer, assignNewPinId) pin ID semantics. Covered by 7 new tests.
Adds PauseUntilUtc to ConsumerHandle, a new Pause(DateTime) overload,
Resume, IsPaused, and GetPauseUntil to ConsumerManager. A System.Threading.Timer
fires when the deadline passes and calls AutoResume, raising OnAutoResumed so
tests can synchronise via SemaphoreSlim instead of Task.Delay. ConsumerManager
now implements IDisposable to clean up outstanding timers. Timer is also
cancelled on explicit Resume and Delete.
Go reference: consumer.go (pauseConsumer / resumeConsumer / isPaused).
Implements a FIFO pull-request queue (WaitingRequestQueue) with per-request
mutable batch countdown and byte budget tracking, plus RemoveExpired cleanup.
Complements the existing priority-based PullRequestWaitQueue for pull consumer delivery.
Go reference: consumer.go waitQueue / processNextMsgRequest (lines 4276-4450).
Add RedeliveryTracker constructor overload, Register(ulong, string) for
tracker-based ack-wait, ProcessAck(ulong) payload-free overload, GetDeadline,
CanRegister for maxAckPending enforcement, ParseAckType static parser, and
AckType enum. All existing API signatures are preserved; 9 new tests added in
AckProcessorEnhancedTests.cs with no regressions to existing 10 tests.
Add new constructor, Schedule(DateTimeOffset), GetDue(DateTimeOffset),
IncrementDeliveryCount, IsMaxDeliveries(), and GetBackoffDelay() to
RedeliveryTracker without breaking existing API. Uses PriorityQueue<ulong,
DateTimeOffset> for deadline-ordered dispatch mirroring Go consumer.go rdq.
Add ProcessLeaderChange(bool) method and OnLeaderChange event to JetStreamMetaGroup.
Refactor StepDown() to delegate inflight clearing through ProcessLeaderChange,
enabling subscribers to react to leadership transitions.
Go reference: jetstream_cluster.go:7001-7074 processLeaderChange.
Replace the simple string-keyed inflight dictionaries with account-scoped
ConcurrentDictionary<string, Dictionary<string, InflightInfo>> structures.
Adds InflightInfo record with OpsCount for duplicate proposal tracking,
TrackInflight/RemoveInflight/IsInflight methods for streams and consumers,
and ClearAllInflight(). Updates existing Propose* methods to use $G account.
Go reference: jetstream_cluster.go:1193-1278.
Add 5 validated Process* methods to JetStreamMetaGroup for stream and
consumer assignment processing: ProcessStreamAssignment, ProcessUpdateStreamAssignment,
ProcessStreamRemoval, ProcessConsumerAssignment, and ProcessConsumerRemoval.
Each returns a bool indicating success, with validation guards matching
Go reference jetstream_cluster.go:4541-5925. Includes 12 new unit tests.
Implements the background loop (Go: monitorCluster) that reads RaftLogEntry
items from a channel and dispatches assignStream, removeStream, assignConsumer,
removeConsumer, and snapshot operations to JetStreamMetaGroup.
- Add five public mutation helpers to JetStreamMetaGroup (AddStreamAssignment,
RemoveStreamAssignment, AddConsumerAssignment, RemoveConsumerAssignment,
ReplaceAllAssignments) used by the monitor path
- JetStreamClusterMonitor: async loop over ChannelReader<RaftLogEntry>, JSON
dispatch, ILogger injection with NullLogger default, malformed entries logged
and skipped rather than aborting the loop
- WaitForProcessedAsync uses Monitor.PulseAll for race-free test synchronisation
with no Task.Delay — satisfies slopwatch SW003/SW004 rules
- 10 new targeted tests all pass; 1101 cluster regression tests unchanged
Implements binary codec for meta-group snapshots: 2-byte little-endian version
header followed by S2-compressed JSON of the stream assignment map. Adds
[JsonObjectCreationHandling(Populate)] to StreamAssignment.Consumers so the
getter-only dictionary is populated in-place during deserialization. 8 tests
covering round-trip, compression ratio, field fidelity, multi-consumer restore,
version rejection, and truncation guard.
Go reference: jetstream_cluster.go:2075-2145 (encodeMetaSnapshot/decodeMetaSnapshot)
Complete IStreamStore Batch 1 — all core operations now have FileStore
implementations instead of throwing NotSupportedException:
- StoreRawMsg: caller-specified seq/ts for replication/mirroring
- LoadPrevMsg: backward scan for message before given sequence
- Type: returns StorageType.File
- Stop: flush + dispose blocks, reject further writes
14 new tests in FileStoreStreamStoreTests.
Add FlushAllPending() to FileStore, fulfilling the IStreamStore interface
contract. The method flushes the active MsgBlock to disk and atomically
writes a stream.state checkpoint using write-to-temp + rename, matching
Go's flushPendingWritesUnlocked / writeFullState pattern.
Add FileStoreCrashRecoveryTests with 5 tests covering:
- FlushAllPending flushes block data to .blk file
- FlushAllPending writes a valid atomic stream.state JSON checkpoint
- FlushAllPending is idempotent (second call overwrites with latest state)
- Recovery prunes messages backdated past the MaxAgeMs cutoff
- Recovery handles a tail-truncated block without throwing
Reference: golang/nats-server/server/filestore.go:5783-5842
StreamManager.Capture now accounts for full message size (subject +
payload + 16-byte overhead) when checking MaxBytes, matching Go's
memStoreMsgSize. PullConsumerEngine uses stream FirstSeq instead of
hardcoded 1 for DeliverAll after purge. Fix 6 tests with Go parity
assertions and updated MaxBytes values.
Now that MemStore uses Unix epoch timestamps (13a3f81), restore the
original Go MaxAge values that were previously omitted as workarounds:
- JsCluster2: MaxAgeMs=500 (Go: 500ms)
- JsCluster34: MaxAgeMs=5000 (Go: 5s)
DateTime.UtcNow.Ticks * 100L overflows long (Ticks ~6.38e17, x100 =
6.38e19 > long.MaxValue 9.22e18), producing corrupted timestamps that
cause PruneExpiredMessages to immediately expire all messages when
MaxAgeMs is set. Use Unix epoch offset to match Go's time.Now().UnixNano().
Port 36 Go-parity tests covering redelivery, NAK, push consumer
heartbeats, interest retention with wildcards, ack-all with large
first seq, deliver last-per-subject, work-in-progress acks, flow
control stall, KV operations, multi-account isolation, and subject
transforms.
Go refs: TestJetStreamRedeliverAndLateAck, TestJetStreamInterestRetention,
TestJetStreamKVDelete, TestJetStreamMultipleAccountsBasics, and 23+ more.
