Commit Graph

27 Commits

Author SHA1 Message Date
Joseph Doherty b9f6a8cc0b feat: add cluster-aware pending request tracking for pull consumers (Gap 3.14)
Adds ProposeWaitingRequest, RegisterClusterPending, RemoveClusterPending,
GetClusterPendingRequests, and ClusterPendingCount to PullConsumerEngine,
backed by a ConcurrentDictionary keyed by reply subject. Includes 10 xUnit
tests covering quorum checks, pending tracking, and concurrent access patterns.
2026-02-25 11:21:21 -05:00
Joseph Doherty aad9cf17e4 feat: add token bucket rate limiter for consumers (Gap 3.13)
Implements TokenBucketRateLimiter with refill-over-time semantics,
TryConsume/EstimateWait/WaitForTokensAsync API, and dynamic rate updates.
12 tests covering all behaviors including SW004-suppressed refill timing test.
2026-02-25 11:15:58 -05:00
Joseph Doherty 778687cf6f feat: add consumer reset to specific sequence (Gap 3.12)
Add ResetToSequence to ConsumerManager that updates NextSequence,
clears AckProcessor state via new ClearAll(), and zeroes PendingBytes.
Add AckProcessor.SetAckFloor() that prunes pending entries below the
new floor. Go reference: consumer.go:4241 processResetReq.
2026-02-25 11:15:33 -05:00
Joseph Doherty b9aa62ae99 feat: add sample/observe mode with latency measurement (Gap 3.11)
Implements SampleTracker with stochastic delivery sampling (ParseSampleFrequency,
ShouldSample, RecordLatency) and LatencySample for consumer observability advisories.
Ports consumer.go sampleFrequency / shouldSample / parseSampleFrequency logic.
15 new tests covering parsing, rate extremes, deterministic seeded sampling, and field capture.
2026-02-25 11:14:58 -05:00
Joseph Doherty 8b4b236968 feat: add delivery interest tracking with auto-cleanup (Gap 3.8)
Implements DeliveryInterestTracker (consumer.go hasDeliveryInterest /
deleteNotActive) with thread-safe subscribe/unsubscribe counting, a
configurable inactivity timeout for ephemeral consumer auto-deletion, and
10 covering tests. Also adds SlopwatchSuppress to a pre-existing
Task.Delay in MaxDeliveriesTests.cs that was already testing time-based expiry.
2026-02-25 11:13:06 -05:00
Joseph Doherty 3d721c6ff1 feat: add filter skip tracking using SubjectMatch (Gap 3.10)
Add FilterSkipTracker using SubjectMatch.MatchLiteral() for NATS token-based
filter matching with sorted-set skip sequence gap tracking. Includes 14 tests
covering exact match, star/gt wildcards, multi-filter, counters, RecordSkip,
NextUnskippedSequence, PurgeBelow, and Reset.
2026-02-25 11:12:57 -05:00
Joseph Doherty 5b0283adf4 feat: add max delivery enforcement with advisory generation (Gap 3.9)
Adds MaxDeliver property, exceeded sequence tracking, DrainExceeded,
DeliveryExceededPolicy enum, and ScheduleRedelivery enforcement to
AckProcessor; 10 new tests in MaxDeliveriesTests.cs.
2026-02-25 11:12:49 -05:00
Joseph Doherty 7611bcc464 feat: add idle heartbeat with pending count headers and flow control stall detection (Gap 3.5)
Heartbeat frames now include Nats-Pending-Messages and Nats-Pending-Bytes
headers populated from the ConsumerHandle. Flow control frames increment
FlowControlPendingCount; AcknowledgeFlowControl() decrements it. IsFlowControlStalled
returns true when pending count reaches MaxFlowControlPending (2).

Go reference: consumer.go:5222 (sendIdleHeartbeat), consumer.go:5495 (sendFlowControl).
2026-02-25 11:05:31 -05:00
Joseph Doherty 2eaa736b21 feat(consumer): add priority group pin ID management
Add AssignPinId, ValidatePinId, and UnassignPinId to PriorityGroupManager,
plus CurrentPinId tracking on PriorityGroup, porting Go consumer.go
(setPinnedTimer, assignNewPinId) pin ID semantics. Covered by 7 new tests.
2026-02-25 02:23:02 -05:00
Joseph Doherty 8fb80acafe feat(consumer): add WaitingRequestQueue with expiry and batch/maxBytes tracking
Implements a FIFO pull-request queue (WaitingRequestQueue) with per-request
mutable batch countdown and byte budget tracking, plus RemoveExpired cleanup.
Complements the existing priority-based PullRequestWaitQueue for pull consumer delivery.
Go reference: consumer.go waitQueue / processNextMsgRequest (lines 4276-4450).
2026-02-25 02:18:01 -05:00
Joseph Doherty 3183fd2dc7 feat(consumer): enhance AckProcessor with NAK delay, TERM, WPI, and maxAckPending
Add RedeliveryTracker constructor overload, Register(ulong, string) for
tracker-based ack-wait, ProcessAck(ulong) payload-free overload, GetDeadline,
CanRegister for maxAckPending enforcement, ParseAckType static parser, and
AckType enum. All existing API signatures are preserved; 9 new tests added in
AckProcessorEnhancedTests.cs with no regressions to existing 10 tests.
2026-02-25 02:15:46 -05:00
Joseph Doherty 55de052009 feat(consumer): add PriorityQueue-based RedeliveryTracker overloads
Add new constructor, Schedule(DateTimeOffset), GetDue(DateTimeOffset),
IncrementDeliveryCount, IsMaxDeliveries(), and GetBackoffDelay() to
RedeliveryTracker without breaking existing API. Uses PriorityQueue<ulong,
DateTimeOffset> for deadline-ordered dispatch mirroring Go consumer.go rdq.
2026-02-25 02:12:37 -05:00
Joseph Doherty 51ebded300 fix: correct MaxBytes enforcement and consumer start sequence after purge
StreamManager.Capture now accounts for full message size (subject +
payload + 16-byte overhead) when checking MaxBytes, matching Go's
memStoreMsgSize. PullConsumerEngine uses stream FirstSeq instead of
hardcoded 1 for DeliverAll after purge. Fix 6 tests with Go parity
assertions and updated MaxBytes values.
2026-02-24 23:59:37 -05:00
Joseph Doherty a9967d3077 test(parity): port consumer pull queue & filter tests (T14) + DB update
Port 48 Go parity tests from TestJetStreamConsumerPull* and related
functions in jetstream_consumer_test.go to unit-level .NET tests.

Enhancements to PullConsumerEngine:
- MaxBytes enforcement in FetchAsync loop (stops delivery when budget exceeded)
- PullRequestWaitQueue with priority-ordered stable enqueue and popAndRequeue
  round-robin semantics within same-priority groups
- PullWaitingRequest record with Priority, RemainingBatch, Reply fields

Enhancements to ConsumerConfig:
- MaxWaiting, MaxRequestBatch, MaxRequestMaxBytes, MaxRequestExpiresMs

New tests (51 total in ConsumerPullQueueTests.cs):
- Pull MaxAckPending enforcement, FIFO delivery, one-shot semantics
- Pull timeout (ExpiresMs), NoWait behavior, MaxBytes byte budget
- Three-filter and multi-filter subject filtering, filter update
- WaitQueue priority ordering and popAndRequeue round-robin
- Pending count tracking, ack floor advancement, redelivery
- DeliverPolicy Last/LastPerSubject/ByStartTime with filters
- ConsumerIsFiltered detection, overlapping subject filters

DB: 37 Go tests mapped → ConsumerPullQueueTests.cs (1,386 total mapped)
2026-02-24 19:53:25 -05:00
Joseph Doherty 662b2e0d87 feat(consumers): add PriorityGroupManager and PullConsumer timeout/compiled filters (C5+C6) 2026-02-24 15:11:30 -05:00
Joseph Doherty 612b15c781 feat: add PushConsumer delivery loop and RedeliveryTracker (C3+C4)
C3 – PushConsumerEngine delivery dispatch:
- Add DeliverSubject property (mirrors consumer.go:1131 dsubj field)
- Add StartDeliveryLoop / StopDeliveryLoop: background Task that drains
  ConsumerHandle.PushFrames and calls a sendMessage delegate per frame
- Delivery loop honours AvailableAtUtc for rate-limiting (consumer.go:5120)
- Data frames: HMSG headers Nats-Sequence, Nats-Time-Stamp, Nats-Subject
  (stream.go:586 JSSequence / JSTimeStamp / JSSubject constants)
- Flow-control frames: "NATS/1.0 100 FlowControl Request" (consumer.go:5501)
- Heartbeat frames: "NATS/1.0 100 Idle Heartbeat" (consumer.go:5222)
- Add DeliverSubject field to ConsumerConfig (consumer.go:115)

C4 – RedeliveryTracker with backoff schedules:
- Schedule(seq, deliveryCount, ackWaitMs): computes deadline using backoff
  array indexed by (deliveryCount-1), clamped at last entry (consumer.go:5540)
- GetDue(): returns sequences whose deadline has passed
- Acknowledge(seq): removes sequence from tracking
- IsMaxDeliveries(seq, maxDeliver): checks threshold for drop decision
- Empty backoff array falls back to ackWaitMs

Tests: 7 PushConsumerDelivery tests + 10 RedeliveryTracker tests (17 total)
2026-02-24 15:01:15 -05:00
Joseph Doherty a7f1243d4f feat(consumers): add NAK/TERM/PROGRESS ack types with backoff (Go parity) 2026-02-24 14:49:20 -05:00
Joseph Doherty 3e972f217e feat: add FileStore 6-way permutation tests (Go testFileStoreAllPermutations parity)
Port 20 tests across the {NoCipher, ChaCha, Aes} x {NoCompression, S2Compression}
matrix from Go's testFileStoreAllPermutations (filestore_test.go:55), yielding 120
total Theory executions in FileStorePermutationTests.cs.

Also fix PushFrame.Subject property missing compilation error that was blocking
the test build (JetStreamConsumerDeliveryEdgeTests.cs:119).

Tests covered (each runs 6x):
  Store_and_load_basic                       TestFileStoreBasics:86
  Store_multiple_messages_load_by_sequence   TestFileStoreBasics:86
  LoadLastBySubject_returns_most_recent_for_subject
  Remove_single_message_updates_state        TestFileStoreBasics:129
  Purge_clears_all_messages                  TestFileStorePurge:710
  TrimToMaxMessages_enforces_limit           TestFileStoreMsgLimitBug:518
  Block_rotation_when_exceeding_block_size   TestFileStoreAndRetrieveMultiBlock:1527
  GetState_returns_correct_counts            TestFileStoreBasics:104
  Snapshot_and_restore_round_trip            TestFileStoreSnapshot:1799
  ListAsync_returns_ordered_messages         TestFileStoreTimeStamps:683
  MaxAge_prunes_expired_messages             TestFileStoreAgeLimit:616
  Recovery_after_reopen_preserves_messages   TestFileStoreBasicWriteMsgsAndRestore:181
  Large_payload_store_and_load               64 KiB random payload variant
  Multiple_subjects_filter_by_subject        TestFileStoreBasics multi-subject
  Sequential_writes_maintain_ordering        TestFileStoreSelectNextFirst:304
  Store_creates_files_on_disk                disk-presence variant
  Write_and_read_same_block                  TestFileStoreWriteAndReadSameBlock:1510
  Stored_messages_have_non_decreasing_timestamps TestFileStoreTimeStamps:683
  Remove_out_of_order_collapses_first_seq    TestFileStoreCollapseDmap:1561
  Snapshot_after_removes_preserves_remaining TestFileStoreSnapshot:1904
2026-02-24 06:43:48 -05:00
Joseph Doherty 081ff1b0db docs: synchronize strict full go parity evidence and status 2026-02-23 15:03:35 -05:00
Joseph Doherty 7bea35aaa8 feat: harden jetstream consumer state machine parity 2026-02-23 14:48:47 -05:00
Joseph Doherty 377ad4a299 feat: complete jetstream deep operational parity closure 2026-02-23 13:43:14 -05:00
Joseph Doherty b41e6ff320 feat: execute post-baseline jetstream parity plan 2026-02-23 12:11:19 -05:00
Joseph Doherty 8bce096f55 feat: complete final jetstream parity transport and runtime baselines 2026-02-23 11:04:43 -05:00
Joseph Doherty f46b331921 feat: complete remaining jetstream parity implementation plan 2026-02-23 10:16:16 -05:00
Joseph Doherty d3aad48096 feat: enforce jetstream ack and redelivery semantics 2026-02-23 06:09:26 -05:00
Joseph Doherty fecb51095f feat: implement jetstream push delivery and heartbeat 2026-02-23 06:08:14 -05:00
Joseph Doherty 9a0de19c2d feat: implement jetstream pull consumer fetch 2026-02-23 06:07:02 -05:00