dohertj2/natsnet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ee1ab96d7c729c6ee8bc5709541537044a2c19b9
natsnet/docs/plans/2026-02-27-batch-37-stream-messages-design.md
Joseph Doherty 8a126c4932 Add batch plans for batches 37-41 (rounds 19-21) 
Generated design docs and implementation plans via Codex for:
- Batch 37: Stream Messages
- Batch 38: Consumer Lifecycle
- Batch 39: Consumer Dispatch
- Batch 40: MQTT Server/JSA
- Batch 41: MQTT Client/IO

All plans include mandatory verification protocol and anti-stub guardrails.
Updated batches.md with file paths and planned status.
All 42 batches (0-41) now have design docs and implementation plans.
2026-02-27 17:27:51 -05:00

158 lines
8.0 KiB
Markdown
Raw Blame History

# Batch 37 Stream Messages Design
**Date:** 2026-02-27
**Batch:** 37 (`Stream Messages`)
**Scope:** 86 features + 13 unit tests
**Dependencies:** Batch `36`
**Go source:** `golang/nats-server/server/stream.go` (focus from ~line 4616 onward)
## Problem
Batch 37 is the stream message data plane: dedupe/message-id handling, direct-get APIs, inbound publish pipeline, consumer signaling, interest/pre-ack tracking, snapshot/restore, and monitor/replication accounting. It is a high-risk batch because most mapped methods are currently absent in .NET, while many mapped tests are still template placeholders.
If this batch is implemented with stubs, Batch 38 (`Consumer Lifecycle`) and later stream/consumer correctness work will be blocked by false greens.
## Context Findings
### Required command outputs
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 37 --db porting.db`
- Status: `pending`
- Features: `86` (IDs `3294-3387` with mapped gaps)
- Tests: `13`
- Depends on: `36`
- Go file: `server/stream.go`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db`
- Confirms dependency chain includes `36 -> 37`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db`
- Overall progress: `1924/6942 (27.7%)`
Environment note: `dotnet` is not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`.
### Feature ownership split (from `porting.db`)
- `NatsStream`: 78
- `JsOutQ`: 3
- `JsPubMsg`: 2
- `Account`: 1
- `CMsg`: 1
- `InMsg`: 1
### Test ownership split (from `porting.db`)
- `RaftNodeTests`: 3
- `JetStreamClusterTests2`: 2
- `JetStreamEngineTests`: 2
- `ConcurrencyTests1`: 1
- `GatewayHandlerTests`: 1
- `JetStreamClusterTests1`: 1
- `JetStreamFileStoreTests`: 1
- `JwtProcessorTests`: 1
- `LeafNodeHandlerTests`: 1
Additional findings:
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` is currently minimal (357 lines) and does not contain mapped Batch 37 methods.
- `JsOutQ` does not currently exist in source and must be introduced.
- `JetStreamClusterTests1.Impltests.cs` and `RaftNodeTests.Impltests.cs` do not currently exist and must be created.
- Many `ImplBacklog` classes still use placeholder patterns (`var goFile = ...`, string-literal assertions), which must be replaced with behavioral tests.
## Approaches
### Approach A: Monolithic `NatsStream.cs` implementation
Port all 86 features directly in `NatsStream.cs` and append helper logic in existing type files.
- Pros: fewer files.
- Cons: poor reviewability, hard to isolate regressions, high merge conflict risk.
### Approach B (Recommended): Message-domain partial decomposition with strict verification gates
Convert `NatsStream` to partial and split message functionality by concern (headers/dedupe, direct-get, inbound pipeline, consumers/interest, snapshot/monitor), plus targeted type helpers (`InMsg`, `CMsg`, `JsPubMsg`, `JsOutQ`) and account restore path.
- Pros: bounded review units, clearer ownership, easier per-group verification, aligns with anti-stub enforcement.
- Cons: requires class splitting and extra file setup.
### Approach C: Test-wave-first before feature groups
Port all 13 mapped tests first and drive production code entirely from failing tests.
- Pros: stronger behavior pressure.
- Cons: high churn because many mapped APIs/types do not yet exist (`JsOutQ`, many stream methods), so red state will be noisy.
**Decision:** Approach B.
## Proposed Design
### 1. Code organization strategy
Keep mapped class ownership intact while splitting by concern:
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` (convert to `partial`, keep core state/lifecycle)
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` (convert mapped message carrier types to partial if needed)
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs` (`InMsg.ReturnToPool`, `CMsg.ReturnToPool`, `JsPubMsg` helpers, `JsOutQ`)
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs` (`Account.RestoreStream`)
Design intent: avoid duplicating existing `JetStreamHeaderHelpers` behavior; mapped `NatsStream` header methods should delegate where appropriate and preserve mapped method surface.
### 2. Functional decomposition
- **Headers + dedupe + scheduling metadata:** `unsubscribe`, `setupStore`, dedupe tables, msg-id checks, expected headers, TTL/schedule/batch header parsing, clustered checks.
- **Direct get + ingress path:** `queueInbound`, direct-get handlers, inbound publish entry points, `processJetStreamMsg`, `processJetStreamBatchMsg`.
- **Message carrier/outbound queue primitives:** `newCMsg`, pooled return methods, js pub message sizing/pool access, `JsOutQ.Send*`, unregister.
- **Consumers + interest + pre-ack:** signaling loop, consumer registry/listing, filtered-interest checks, pre-ack register/clear/ack.
- **Snapshot + restore + monitor traffic:** `snapshot`, `Account.RestoreStream`, orphan/replication checks, monitor running flags, replication traffic accounting.
### 3. Test strategy
Port mapped tests in three waves, replacing placeholders with behavior assertions:
- **Wave T1 (5):** direct-get + rollup/mirror dedupe (`828,954,987,1642,1643`)
- **Wave T2 (4):** snapshot/restore/raft catchup (`383,2617,2672,2695`)
- **Wave T3 (4):** gateway/jwt/leaf/perf interaction (`635,1892,1961,2426`)
Test files:
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`
### 4. Verification architecture
- Mandatory per-feature and per-test loops with evidence before status promotion.
- Mandatory stub scan, build gate, and targeted/full test gates.
- Status changes chunked to max 15 IDs per `batch-update`.
- Mandatory checkpoints between every task.
- If blocked, keep item `deferred` with specific reason; never create fake-pass stubs.
## Feature Grouping (for implementation plan)
- Group A (18): `3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3310,3311,3312`
- Group B (16): `3314,3315,3316,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330`
- Group C (16): `3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3345,3346,3347`
- Group D (17): `3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3364,3366,3367,3368`
- Group E (19): `3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387`
## Constraints
- Planning only in this session; no implementation execution.
- Must follow `docs/standards/dotnet-standards.md` (`xUnit 3`, `Shouldly`, `NSubstitute`, nullable, naming conventions).
- Batch 37 work must not start unless Batch 36 is complete/ready.
## Non-Goals
- Executing Batch 37 code changes in this session.
- Expanding beyond mapped Batch 37 IDs.
- Introducing large integration harnesses unrelated to mapped tests.
Reference in New Issue View Git Blame Copy Permalink