natsnet/docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md

Batch 39 Consumer Dispatch Design

Date: 2026-02-27
Batch: 39 (Consumer Dispatch)
Scope: 93 features + 53 unit tests
Dependencies: Batch 38
Go source: golang/nats-server/server/consumer.go (dispatch-heavy region around lines 3925-6706)

Problem

Batch 39 contains the JetStream consumer dispatch/data-plane logic: pull request queueing, next-message request parsing, pending accounting, heartbeats/flow-control, delivery tracking, redelivery queue semantics, ack reply parsing, stream purge/stop paths, and monitor/signal flow.

If this batch is advanced with placeholders, the runtime behavior for pull consumers, redelivery correctness, and no-interest cleanup will look green in PortTracker while remaining behaviorally incomplete.

Context Findings

Required command outputs

• /usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 39 --db porting.db
  • Status: pending
  • Features: 93 (all currently deferred)
  • Tests: 53 (all currently deferred)
  • Depends on: 38
  • Go file: server/consumer.go
• /usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db
  • Confirms dependency chain 38 -> 39
• /usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db
  • Overall progress snapshot: 1924/6942 (27.7%)

Environment note: dotnet is not on PATH in this shell; use /usr/local/share/dotnet/dotnet.

Batch ownership split

Features by class:

• NatsConsumer: 91
• NextMsgReq: 1
• NatsStream: 1

Tests by class:

• JetStreamEngineTests: 34
• NatsConsumerTests: 14
• ConcurrencyTests1: 2
• JetStreamClusterTests1: 1
• JwtProcessorTests: 1
• RouteHandlerTests: 1

Current .NET baseline findings

• dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs is still a compact lifecycle shell and does not expose most Batch 39 mapped methods.
• WaitQueue and WaitingRequest exist in StreamTypes.cs, but only part of the Go dispatch behavior is represented.
• ImplBacklog files for Batch 39-related tests are mostly template placeholders and need full behavioral ports.

Approach Options

Approach A: Keep all dispatch work in NatsConsumer.cs

• Pros: fewer files and simpler discovery.
• Cons: 90+ methods in one file, poor reviewability, high merge risk, and weaker checkpoint isolation.

Approach B (Recommended): Dispatch-focused partial decomposition for NatsConsumer

• Pros: aligns with Go dispatch domains, improves review boundaries, supports group-by-group verification loops.
• Cons: more files and up-front organization overhead.

Approach C: Port all 53 tests first, then implement features

• Pros: strict red/green pressure.
• Cons: too many missing APIs initially, broad low-signal red phase, high churn in test scaffolding.

Decision: Approach B.

Proposed Design

1. Code Organization

Keep NatsConsumer as the central type but split dispatch concerns into partial files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs (core state, shared lock helpers, partial type anchor)
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Waiting.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Acks.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Delivery.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Redelivery.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.ReplyParsing.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Shutdown.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs (DeleteConsumer mapping)
• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs (WaitQueue, WaitingRequest, PriorityGroup helpers)

2. Functional Decomposition

• Waiting queue and request ingress: pending request map, pin assignment/ttl, request parsing (ProcessNextMsgReq, ProcessResetReq, ProcessNextMsgRequest).
• Dispatch loop and pending accounting: GetNextMsg, ProcessWaiting, CheckAckFloor, LoopAndGatherMsgs, pending counters.
• Delivery and flow control: heartbeat emission, flow-control reply generation, message delivery envelope conversion.
• Redelivery and ack reply parsing: pending tracking, redelivery queue operations, ack subject parsing (ReplyInfo, AckReplyInfo, seq extraction).
• Lifecycle and stream-interest cleanup: durable/push/pull helpers, purge/stop/cleanup, stream signals, monitor guard methods.

3. Test Design

Replace template tests with behavioral tests tied to mapped IDs:

• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs (14)
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs (34)
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs (2)
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs (1)
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs (1)
• Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs (1)

Also add/expand focused non-backlog unit tests where internal behaviors can be tested deterministically:

• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerDispatchTests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs

4. Verification Model (Design-level)

Batch 39 execution must be evidence-driven:

• Per-feature implementation loop with targeted tests.
• Per-test single-test and class-level verification.
• Stub scan before any status promotion.
• Build and test gates between groups.
• Status updates in max 15-ID chunks.
• Deferred-with-reason instead of stubbing when blocked.

Feature Groups (max ~20 each)

• Group A (18): waiting queue + pull request ingress
  696,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715
• Group B (18): dispatch selection + ack floor + inbound loops
  716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733
• Group C (19): delivery payload + flow control + redelivery foundation
  734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752
• Group D (19): reply parsing + sequence selection + consumer identity helpers
  753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,772
• Group E (19): stop/purge/no-interest/monitor signaling path
  774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792

Test Waves

• Wave T1 (14): 1230,1232,1251,1261,1265,1267,1273,1277,1283,1284,1285,1286,1339,1370
• Wave T2 (17): 1469,1484,1485,1486,1487,1488,1489,1490,1492,1493,1495,1496,1498,1499,1500,1501,1502
• Wave T3 (17): 1508,1514,1515,1516,1517,1518,1519,1520,1521,1522,1526,1530,1531,1545,1547,1567,1665
• Wave T4 (5): 814,1840,2389,2407,2858

Risks and Mitigations

• Risk: dispatch loops can mask bugs via timing-sensitive behavior.
  Mitigation: isolate pure logic from timers/network side-effects; unit-test selectors and parsers first.
• Risk: placeholder tests could slip through due green-but-empty assertions.
  Mitigation: mandatory stub scan and assertion-quality checks before status updates.
• Risk: over-bulk status updates can misclassify unverified IDs.
  Mitigation: enforce max 15 IDs per update and per-task checkpoints.

Non-Goals

• Executing Batch 39 implementation in this session.
• Re-scoping IDs outside Batch 39.
• Marking any feature/test as verified during planning.