dohertj2/natsnet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add batch plans for batches 37-41 (rounds 19-21)

Browse Source 
Generated design docs and implementation plans via Codex for:
- Batch 37: Stream Messages
- Batch 38: Consumer Lifecycle
- Batch 39: Consumer Dispatch
- Batch 40: MQTT Server/JSA
- Batch 41: MQTT Client/IO

All plans include mandatory verification protocol and anti-stub guardrails.
Updated batches.md with file paths and planned status.
All 42 batches (0-41) now have design docs and implementation plans.
This commit is contained in:
Joseph Doherty
2026-02-27 17:27:51 -05:00
parent f8dce79ac0
commit 8a126c4932
13 changed files with 2963 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/batches.md
View File

@@ -51,8 +51,8 @@
| 34 | JS Cluster Consumers | [design](plans/2026-02-27-batch-34-js-cluster-consumers-design.md) | [plan](plans/2026-02-27-batch-34-js-cluster-consumers-implementation-plan.md) | planned |
| 35 | JS Cluster Remaining | [design](plans/2026-02-27-batch-35-js-cluster-remaining-design.md) | [plan](plans/2026-02-27-batch-35-js-cluster-remaining-implementation-plan.md) | planned |
| 36 | Stream Lifecycle | [design](plans/2026-02-27-batch-36-stream-lifecycle-design.md) | [plan](plans/2026-02-27-batch-36-stream-lifecycle-implementation-plan.md) | planned |
| 37 | Stream Messages | | | not_planned |
| 38 | Consumer Lifecycle | | | not_planned |
| 39 | Consumer Dispatch | | | not_planned |
| 40 | MQTT Server/JSA | | | not_planned |
| 41 | MQTT Client/IO | | | not_planned |
| 37 | Stream Messages | [design](plans/2026-02-27-batch-37-stream-messages-design.md) | [plan](plans/2026-02-27-batch-37-stream-messages-implementation-plan.md) | planned |
| 38 | Consumer Lifecycle | [design](plans/2026-02-27-batch-38-consumer-lifecycle-design.md) | [plan](plans/2026-02-27-batch-38-consumer-lifecycle-implementation-plan.md) | planned |
| 39 | Consumer Dispatch | [design](plans/2026-02-27-batch-39-consumer-dispatch-design.md) | [plan](plans/2026-02-27-batch-39-consumer-dispatch-implementation-plan.md) | planned |
| 40 | MQTT Server/JSA | [design](plans/2026-02-27-batch-40-mqtt-server-jsa-design.md) | [plan](plans/2026-02-27-batch-40-mqtt-server-jsa-implementation-plan.md) | planned |
| 41 | MQTT Client/IO | [design](plans/2026-02-27-batch-41-mqtt-client-io-design.md) | [plan](plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md) | planned |

157
docs/plans/2026-02-27-batch-37-stream-messages-design.md Normal file
View File

@@ -0,0 +1,157 @@
# Batch 37 Stream Messages Design
**Date:** 2026-02-27
**Batch:** 37 (`Stream Messages`)
**Scope:** 86 features + 13 unit tests
**Dependencies:** Batch `36`
**Go source:** `golang/nats-server/server/stream.go` (focus from ~line 4616 onward)
## Problem
Batch 37 is the stream message data plane: dedupe/message-id handling, direct-get APIs, inbound publish pipeline, consumer signaling, interest/pre-ack tracking, snapshot/restore, and monitor/replication accounting. It is a high-risk batch because most mapped methods are currently absent in .NET, while many mapped tests are still template placeholders.
If this batch is implemented with stubs, Batch 38 (`Consumer Lifecycle`) and later stream/consumer correctness work will be blocked by false greens.
## Context Findings
### Required command outputs
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 37 --db porting.db`
- Status: `pending`
- Features: `86` (IDs `3294-3387` with mapped gaps)
- Tests: `13`
- Depends on: `36`
- Go file: `server/stream.go`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db`
- Confirms dependency chain includes `36 -> 37`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db`
- Overall progress: `1924/6942 (27.7%)`
Environment note: `dotnet` is not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`.
### Feature ownership split (from `porting.db`)
- `NatsStream`: 78
- `JsOutQ`: 3
- `JsPubMsg`: 2
- `Account`: 1
- `CMsg`: 1
- `InMsg`: 1
### Test ownership split (from `porting.db`)
- `RaftNodeTests`: 3
- `JetStreamClusterTests2`: 2
- `JetStreamEngineTests`: 2
- `ConcurrencyTests1`: 1
- `GatewayHandlerTests`: 1
- `JetStreamClusterTests1`: 1
- `JetStreamFileStoreTests`: 1
- `JwtProcessorTests`: 1
- `LeafNodeHandlerTests`: 1
Additional findings:
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` is currently minimal (357 lines) and does not contain mapped Batch 37 methods.
- `JsOutQ` does not currently exist in source and must be introduced.
- `JetStreamClusterTests1.Impltests.cs` and `RaftNodeTests.Impltests.cs` do not currently exist and must be created.
- Many `ImplBacklog` classes still use placeholder patterns (`var goFile = ...`, string-literal assertions), which must be replaced with behavioral tests.
## Approaches
### Approach A: Monolithic `NatsStream.cs` implementation
Port all 86 features directly in `NatsStream.cs` and append helper logic in existing type files.
- Pros: fewer files.
- Cons: poor reviewability, hard to isolate regressions, high merge conflict risk.
### Approach B (Recommended): Message-domain partial decomposition with strict verification gates
Convert `NatsStream` to partial and split message functionality by concern (headers/dedupe, direct-get, inbound pipeline, consumers/interest, snapshot/monitor), plus targeted type helpers (`InMsg`, `CMsg`, `JsPubMsg`, `JsOutQ`) and account restore path.
- Pros: bounded review units, clearer ownership, easier per-group verification, aligns with anti-stub enforcement.
- Cons: requires class splitting and extra file setup.
### Approach C: Test-wave-first before feature groups
Port all 13 mapped tests first and drive production code entirely from failing tests.
- Pros: stronger behavior pressure.
- Cons: high churn because many mapped APIs/types do not yet exist (`JsOutQ`, many stream methods), so red state will be noisy.
**Decision:** Approach B.
## Proposed Design
### 1. Code organization strategy
Keep mapped class ownership intact while splitting by concern:
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` (convert to `partial`, keep core state/lifecycle)
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` (convert mapped message carrier types to partial if needed)
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs` (`InMsg.ReturnToPool`, `CMsg.ReturnToPool`, `JsPubMsg` helpers, `JsOutQ`)
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs` (`Account.RestoreStream`)
Design intent: avoid duplicating existing `JetStreamHeaderHelpers` behavior; mapped `NatsStream` header methods should delegate where appropriate and preserve mapped method surface.
### 2. Functional decomposition
- **Headers + dedupe + scheduling metadata:** `unsubscribe`, `setupStore`, dedupe tables, msg-id checks, expected headers, TTL/schedule/batch header parsing, clustered checks.
- **Direct get + ingress path:** `queueInbound`, direct-get handlers, inbound publish entry points, `processJetStreamMsg`, `processJetStreamBatchMsg`.
- **Message carrier/outbound queue primitives:** `newCMsg`, pooled return methods, js pub message sizing/pool access, `JsOutQ.Send*`, unregister.
- **Consumers + interest + pre-ack:** signaling loop, consumer registry/listing, filtered-interest checks, pre-ack register/clear/ack.
- **Snapshot + restore + monitor traffic:** `snapshot`, `Account.RestoreStream`, orphan/replication checks, monitor running flags, replication traffic accounting.
### 3. Test strategy
Port mapped tests in three waves, replacing placeholders with behavior assertions:
- **Wave T1 (5):** direct-get + rollup/mirror dedupe (`828,954,987,1642,1643`)
- **Wave T2 (4):** snapshot/restore/raft catchup (`383,2617,2672,2695`)
- **Wave T3 (4):** gateway/jwt/leaf/perf interaction (`635,1892,1961,2426`)
Test files:
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`
### 4. Verification architecture
- Mandatory per-feature and per-test loops with evidence before status promotion.
- Mandatory stub scan, build gate, and targeted/full test gates.
- Status changes chunked to max 15 IDs per `batch-update`.
- Mandatory checkpoints between every task.
- If blocked, keep item `deferred` with specific reason; never create fake-pass stubs.
## Feature Grouping (for implementation plan)
- Group A (18): `3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3310,3311,3312`
- Group B (16): `3314,3315,3316,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330`
- Group C (16): `3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3345,3346,3347`
- Group D (17): `3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3364,3366,3367,3368`
- Group E (19): `3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387`
## Constraints
- Planning only in this session; no implementation execution.
- Must follow `docs/standards/dotnet-standards.md` (`xUnit 3`, `Shouldly`, `NSubstitute`, nullable, naming conventions).
- Batch 37 work must not start unless Batch 36 is complete/ready.
## Non-Goals
- Executing Batch 37 code changes in this session.
- Expanding beyond mapped Batch 37 IDs.
- Introducing large integration harnesses unrelated to mapped tests.

457
docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md Normal file
View File

@@ -0,0 +1,457 @@
# Batch 37 Stream Messages Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Port and verify Batch 37 (`Stream Messages`) from `server/stream.go` so all 86 mapped features and 13 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blockers.
**Architecture:** Implement Batch 37 in five feature groups (`18/16/16/17/19`) using `NatsStream` partials plus message carrier helpers (`InMsg`, `CMsg`, `JsPubMsg`, `JsOutQ`) and account restore support. Execute three test waves with strict per-feature/per-test evidence loops and mandatory anti-stub gates before any status promotion.
**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)
**Design doc:** `docs/plans/2026-02-27-batch-37-stream-messages-design.md`
---
## Batch 37 Scope
- Batch ID: `37`
- Name: `Stream Messages`
- Dependency: `36`
- Go source: `golang/nats-server/server/stream.go`
- Features: `86` (`3294-3387` mapped IDs)
- Tests: `13`
If `dotnet` is not on `PATH`, use:
```bash
DOTNET=/usr/local/share/dotnet/dotnet
```
Expected production files:
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs`
Expected test files:
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`
---
## MANDATORY VERIFICATION PROTOCOL
> **NON-NEGOTIABLE:** Every Batch 37 feature and test must pass this loop. No shortcut status updates.
### Dependency Preflight (before any status change)
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 37 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```
Start only when Batch 37 is ready:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch start 37 --db porting.db
```
Capture baseline:
```bash
$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### What Counts as a Real Port
A **feature** is real only if all are true:
1. Mapped method/type exists with non-placeholder behavior.
2. Logic uses actual state/inputs (not hardcoded defaults).
3. At least one behavioral test exercises that code path.
4. No forbidden stub patterns are present.
A **test** is real only if all are true:
1. Arrange/Act/Assert is present.
2. It calls production code in `ZB.MOM.NatsNet.Server.*`.
3. Assertions validate behavior from Act output/effects.
4. It is not a template/literal stub.
### Per-Feature Verification Loop (REQUIRED per feature ID)
1. Inspect mapping and Go context:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
```
2. Claim work:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
```
3. Add or update focused behavioral test(s) for that feature.
4. Implement minimal real logic for the mapped method/type.
5. Run **Stub Detection Check** on changed files.
6. Run **Build Gate**.
7. Run targeted **Test Gate** (specific class/filter for touched area).
8. Promote only proven feature IDs to `complete` (or `verified` only after checkpoint evidence).
### Per-Test Verification Loop (REQUIRED per test ID)
1. Inspect mapping and Go test location:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
```
2. Claim work:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
```
3. Port full behavior (no template assertions).
4. Run single test method:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
```
5. Confirm `Passed: 1, Failed: 0` (never `Passed: 0`).
6. Run touched class filter.
7. Run **Stub Detection Check** + **Build Gate**.
8. Promote only proven test IDs to `complete` (or `verified` only after checkpoint evidence).
### Stub Detection Check (REQUIRED after every feature/test loop)
```bash
changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$changed_files" ]; then
echo "$changed_files" | xargs rg -n "(NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|// TODO|// PLACEHOLDER|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary<string, string>\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$)"
fi
```
Any match blocks status promotion until fixed or deferred with reason.
### Build Gate (REQUIRED)
```bash
$DOTNET build dotnet/
```
Run this:
- after each feature loop
- after each test loop
- before every `batch-update`
- at every task checkpoint
### Test Gate (REQUIRED)
Minimum targeted gates by touched domain:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamEngineTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamClusterTests1|FullyQualifiedName~JetStreamClusterTests2"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamFileStoreTests|FullyQualifiedName~RaftNodeTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~JwtProcessorTests|FullyQualifiedName~LeafNodeHandlerTests|FullyQualifiedName~ConcurrencyTests1"
```
Checkpoint/full gate:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update)
- Never pass more than `15` IDs per `feature batch-update` or `test batch-update`.
- Never update IDs outside current task scope.
- Never promote to `verified` without checkpoint evidence.
- For blocked items, keep `deferred` with explicit reason.
Use:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
$DOTNET run --project tools/NatsNet.PortTracker -- \
test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```
### Checkpoint Protocol Between Tasks (REQUIRED)
Before starting the next task:
1. Run **Stub Detection Check** on current diff.
2. Run **Build Gate**.
3. Run targeted class test filters for touched classes.
4. Run full unit test gate.
5. Apply status updates in `<=15` ID chunks only.
6. Commit checkpoint before next task.
---
## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)
### Forbidden Patterns
Forbidden in mapped Batch 37 feature/test work:
- `throw new NotImplementedException()`
- Empty mapped method bodies or no-op method shims
- Placeholder comments (`TODO`, `PLACEHOLDER`, `later`)
- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`)
- Template assertions that do not validate behavior:
- `var goFile = "server/..."`
- `"<MethodName>".ShouldContain("Should")`
- `GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty)` as core assertion
- Constant-return shortcuts in mapped methods without behavioral justification:
- `=> default;`
- `return default;`
- `return null;`
- Catch-and-ignore patterns that hide failures
### Hard Limits
- Max feature group size: `~20` IDs.
- Max IDs per `feature batch-update`: `15`.
- Max IDs per `test batch-update`: `15`.
- One active feature loop at a time.
- One active test loop at a time.
- Mandatory checkpoint between every task.
- No `verified` promotion without full checkpoint evidence.
### If You Get Stuck (REQUIRED)
1. Stop on the current ID immediately.
2. Do not write placeholder logic or fake assertions.
3. Mark item `deferred` with concrete blocker.
4. Continue with next unblocked item.
Feature deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
Test deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
`deferred` with explicit reason is correct. Stubbed progress is not.
---
## Feature Groups (max ~20 each)
### Group A (18): Dedupe and header extraction foundation
IDs:
`3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3310,3311,3312`
### Group B (16): Scheduling/batch headers and direct-get ingress path
IDs:
`3314,3315,3316,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330`
### Group C (16): Message carriers, publish out queue, and internal loop hooks
IDs:
`3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3345,3346,3347`
### Group D (17): Consumer registry and interest-state operations
IDs:
`3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3364,3366,3367,3368`
### Group E (19): Pre-ack, snapshot/restore, monitor/replication checks
IDs:
`3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387`
## Test Waves
### Wave T1 (5): direct-get and rollup/mirror semantics
IDs:
`828,954,987,1642,1643`
### Wave T2 (4): snapshot/restore and raft recovery paths
IDs:
`383,2617,2672,2695`
### Wave T3 (4): gateway/jwt/leaf/perf regression coverage
IDs:
`635,1892,1961,2426`
---
### Task 1: Preflight, Baseline, and Batch Start
**Files:**
- Read: `docs/plans/2026-02-27-batch-37-stream-messages-design.md`
- Read: `golang/nats-server/server/stream.go`
**Steps:**
1. Run dependency preflight and verify Batch 36 completion.
2. Start Batch 37.
3. Capture baseline build/test evidence.
4. Do not change any status without baseline logs.
### Task 2: Implement Feature Group A (18 IDs)
**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs`
**Steps:**
1. Move Group A IDs to `stub` in `15 + 3` chunks.
2. Run Per-Feature Verification Loop for each ID.
3. Run required stub/build/test gates per ID.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.
### Task 3: Implement Feature Group B (16 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs`
**Steps:**
1. Move Group B IDs to `stub` in `15 + 1` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.
### Task 4: Implement Feature Group C (16 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs`
**Steps:**
1. Move Group C IDs to `stub` in `15 + 1` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Ensure `JsOutQ` exists with mapped methods.
4. Run mandatory gates.
5. Promote proven IDs in `<=15` chunks.
6. Run checkpoint protocol.
### Task 5: Implement Feature Group D (17 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
**Steps:**
1. Move Group D IDs to `stub` in `15 + 2` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.
### Task 6: Implement Feature Group E (19 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs`
**Steps:**
1. Move Group E IDs to `stub` in `15 + 4` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.
### Task 7: Port Test Wave T1 (5 IDs)
**Files:**
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
**Steps:**
1. Move Wave T1 IDs to `stub`.
2. Execute Per-Test Verification Loop per ID.
3. Run mandatory stub/build/class/full test gates.
4. Promote proven IDs in one `<=15` batch.
5. Run checkpoint protocol.
### Task 8: Port Test Wave T2 (4 IDs)
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`
**Steps:**
1. Move Wave T2 IDs to `stub`.
2. Execute Per-Test Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in one `<=15` batch.
5. Run checkpoint protocol.
### Task 9: Port Test Wave T3 (4 IDs)
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
**Steps:**
1. Move Wave T3 IDs to `stub`.
2. Execute Per-Test Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in one `<=15` batch.
5. Run checkpoint protocol.
### Task 10: Batch Closeout
**Files:**
- Modify: `porting.db`
- Modify: `reports/current.md` (via report script output)
**Steps:**
1. Run final stub detection on current diff.
2. Run final build gate and full test gate.
3. Run dry-run audits for features and tests:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db
```
4. Complete batch:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 37 --db porting.db
```
5. Regenerate summary report:
```bash
./reports/generate-report.sh
```
---
Plan complete and saved to `docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md`. Two execution options:
1. Subagent-Driven (this session) - I dispatch fresh subagent per task, review between tasks, fast iteration.
2. Parallel Session (separate) - Open new session with `executeplan`, batch execution with checkpoints.
Which approach?

165
docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md Normal file
View File

@@ -0,0 +1,165 @@
# Batch 38 Consumer Lifecycle Design
**Date:** 2026-02-27
**Batch:** 38 (`Consumer Lifecycle`)
**Scope:** 96 features + 71 unit tests
**Dependencies:** Batches `34`, `36`
**Go source:** `golang/nats-server/server/consumer.go` (primary focus lines ~176-3761 for mapped features)
## Problem
Batch 38 carries the JetStream consumer control plane and lifecycle core: enum/config normalization, consumer creation/update paths, advisories, delivery-interest checks, ack processing, pending request tracking, store-state persistence, info snapshots, sampling, filtering, and wait-queue mechanics.
The current .NET surface is far smaller than the mapped scope (`NatsConsumer.cs` is 246 lines vs. `consumer.go` at 6715 lines). If this batch is advanced with placeholders, Batch 39 (`Consumer Dispatch`) will inherit false-green behavior and brittle semantics.
## Context Findings
### Required command outputs
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db`
- Status: `pending`
- Features: `96` (IDs `584-684` with mapped gaps)
- Tests: `71`
- Depends on: `34,36`
- Go file: `server/consumer.go`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db`
- Confirms dependency chain includes `34/36 -> 38 -> 39`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db`
- Overall progress: `1924/6942 (27.7%)`
Environment note: `dotnet` is not on shell `PATH`; use `/usr/local/share/dotnet/dotnet`.
### Dependency readiness
- `batch ready` currently does **not** include Batch 38.
- Dependencies (`34`, `36`) are still `pending` and must be complete before Batch 38 starts.
### Feature ownership split (from `porting.db`)
- `NatsConsumer`: 77
- `ConsumerAction`: 3
- `PriorityPolicy`: 3
- `NatsStream`: 3
- `WaitingRequest`: 3
- `DeliverPolicy`: 1
- `AckPolicy`: 1
- `ReplayPolicy`: 1
- `SubjectTokens`: 1
- `NatsServer`: 1
- `Account`: 1
- `JsPubMsg`: 1
### Test ownership split (from `porting.db`)
- `NatsConsumerTests`: 38
- `JetStreamEngineTests`: 20
- `JetStreamClusterTests1`: 3
- `JetStreamClusterTests3`: 3
- `JetStreamClusterTests4`: 2
- `ConcurrencyTests1`: 2
- `AccountTests`: 1
- `JetStreamBenchmarks`: 1
- `MqttHandlerTests`: 1
Additional findings:
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` currently exposes only a small subset of mapped methods.
- `NatsStream` does not currently expose mapped `AddConsumer*` methods.
- `Account.CheckNewConsumerConfig` and `NatsServer.HasGatewayInterest` are not present.
- `JetStreamClusterTests1/3/4.Impltests.cs` and `JetStreamBenchmarks.Impltests.cs` do not currently exist.
- Existing `ImplBacklog` files for this scope heavily use placeholder patterns (`var goFile = ...`, literal-string assertions), requiring replacement with behavioral tests.
## Approaches
### Approach A: Expand a single `NatsConsumer.cs` monolith
- Pros: fewer files.
- Cons: weak review boundaries, high merge conflict risk, difficult to validate incrementally across 96 features.
### Approach B (Recommended): Partial-class decomposition by lifecycle domain + strict verification gates
- Pros: bounded feature groups, cleaner method ownership, easier per-group verification and anti-stub enforcement.
- Cons: requires file reorganization and additional partial files.
### Approach C: Port all mapped tests first, then backfill production code
- Pros: forces behavior-driven progress.
- Cons: too noisy early because many mapped APIs/types do not yet exist; red phase would be broad and low-signal.
**Decision:** Approach B.
## Proposed Design
### 1. Code organization strategy
Use partial decomposition for consumer-heavy logic while keeping mapped ownership:
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` (core state, constructor, shared locking)
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Config.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Lifecycle.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Advisories.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.WaitQueue.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` (`AddConsumer*`)
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.ConsumerPolicies.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.ConsumerConfig.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.ConsumerInterest.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Internal/SubjectTokens.ConsumerFilters.cs`
Design intent: keep method surfaces aligned to mapped `dotnet_class` + `dotnet_method` while avoiding one large file.
### 2. Functional decomposition
- **Enums and config validation:** consumer actions/policies string+JSON mapping, defaults, config checks.
- **Creation and lifecycle wiring:** `AddConsumer*`, assignment, monitor channels, leadership checks.
- **Advisories and delivery-interest:** internal subscribe/unsubscribe, advisory publication, gateway interest integration.
- **Ack and pending-request flow:** ack reply message parsing/building, `processAck/Nak/Term`, redelivery and pending-request transitions.
- **Store state and info path:** read/apply/write state, info snapshots, sampling, filter checks, pull request parsing, wait-queue recycling.
### 3. Test strategy
Port mapped tests by class waves and remove template assertions:
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` (38)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` (20)
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` (3)
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` (3)
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` (2)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` (2)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` (1)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` (1)
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBenchmarks.Impltests.cs` (1; may remain deferred if benchmark-only)
### 4. Verification architecture
Adopt Batch-0-style hard verification controls, adapted for feature + test work:
- Mandatory per-feature loop (evidence required before status promotion).
- Mandatory per-test loop (single-test and class-level pass evidence).
- Stub detection required after each loop.
- Build + targeted test gates required before each status update.
- Status updates chunked to max 15 IDs per `batch-update`.
- Mandatory checkpoint between every task.
- Explicit deferred handling with reason when blocked; no placeholder stubs.
## Feature Grouping (for implementation plan)
- Group A (19): `584,585,586,587,588,589,590,591,592,594,595,596,597,598,599,600,601,602,603`
- Group B (19): `604,605,606,607,608,610,612,613,614,615,616,617,618,619,620,621,622,623,624`
- Group C (19): `625,626,627,629,630,631,632,633,635,636,637,638,639,640,641,642,643,644,645`
- Group D (19): `646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664`
- Group E (20): `665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684`
## Constraints
- Planning only in this session; no implementation execution.
- Must follow `docs/standards/dotnet-standards.md` (xUnit 3 + Shouldly + NSubstitute, nullable, naming conventions).
- Batch 38 implementation must start only after Batches 34 and 36 are complete/ready.
## Non-Goals
- Executing Batch 38 code changes in this session.
- Re-scoping features/tests outside mapped Batch 38 IDs.
- Fabricating benchmark/integration parity via fake-pass unit-test templates.

481
docs/plans/2026-02-27-batch-38-consumer-lifecycle-implementation-plan.md Normal file
View File

@@ -0,0 +1,481 @@
# Batch 38 Consumer Lifecycle Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Port and verify Batch 38 (`Consumer Lifecycle`) from `server/consumer.go` so all 96 mapped features and 71 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blocker notes.
**Architecture:** Implement Batch 38 in five feature groups (`19/19/19/19/20`) using `NatsConsumer` partial decomposition plus supporting updates in `NatsStream`, `StoreTypes`, `StreamTypes`, `Account`, `NatsServer`, and `SubjectTokens`. Execute five test waves with strict per-feature/per-test evidence loops, mandatory stub detection, and checkpoint gates before status promotion.
**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)
**Design doc:** `docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md`
---
## Batch 38 Scope
- Batch ID: `38`
- Name: `Consumer Lifecycle`
- Dependencies: `34,36`
- Go source: `golang/nats-server/server/consumer.go`
- Features: `96` (`584-684` mapped IDs)
- Tests: `71`
If `dotnet` is not on `PATH`, use:
```bash
DOTNET=/usr/local/share/dotnet/dotnet
```
Primary production files (expected):
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Config.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Lifecycle.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Advisories.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.WaitQueue.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.ConsumerPolicies.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.ConsumerConfig.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.ConsumerInterest.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Internal/SubjectTokens.ConsumerFilters.cs`
Primary test files (expected):
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerPoliciesTests.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerStateTests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBenchmarks.Impltests.cs`
---
## MANDATORY VERIFICATION PROTOCOL
> **NON-NEGOTIABLE:** Every Batch 38 feature and test must pass this loop. No shortcut updates.
### Dependency Preflight (before any status change)
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 34 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```
Start only when Batch 38 is ready:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch start 38 --db porting.db
```
Capture baseline:
```bash
$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### What Counts as a Real Port
A **feature** is real only if all are true:
1. Mapped method/type exists with behavioral logic (no placeholder body).
2. Logic consumes real inputs/state (not constants that bypass behavior).
3. At least one behavioral test exercises the path.
4. Stub scan and gates are clean.
A **test** is real only if all are true:
1. Has Arrange/Act/Assert.
2. Calls production code in `ZB.MOM.NatsNet.Server.*`.
3. Assertions validate behavior derived from Act output/effects.
4. Contains no template-stub patterns.
### Per-Feature Verification Loop (REQUIRED per feature ID)
1. Inspect mapping and Go context:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
```
2. Claim feature:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
```
3. Add/adjust focused behavioral test(s) for that feature.
4. Implement minimal real logic for the mapped method/type.
5. Run **Stub Detection Check**.
6. Run **Build Gate**.
7. Run targeted **Test Gate** for touched classes.
8. Promote proven feature to `complete` (or `verified` only after checkpoint evidence).
### Per-Test Verification Loop (REQUIRED per test ID)
1. Inspect mapping and Go test source:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
```
2. Claim test:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
```
3. Port full behavior (no placeholders).
4. Run single test:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
```
5. Confirm `Passed: 1, Failed: 0` (never `Passed: 0`).
6. Run class-level filter for touched class.
7. Run **Stub Detection Check** + **Build Gate**.
8. Promote proven test to `complete` (or `verified` only after checkpoint evidence).
### Stub Detection Check (REQUIRED after every feature/test loop)
```bash
changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$changed_files" ]; then
echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary<string, string>\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$|throw new Exception\\(\"TODO\")"
fi
```
Any match blocks status promotion until fixed or explicitly deferred.
### Build Gate (REQUIRED)
```bash
$DOTNET build dotnet/
```
Run after each feature loop, each test loop, before each `batch-update`, and at each checkpoint.
### Test Gate (REQUIRED)
Minimum targeted gates per touched domain:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream.NatsConsumerTests|FullyQualifiedName~JetStream.WaitQueueTests|FullyQualifiedName~JetStream.ConsumerPoliciesTests|FullyQualifiedName~JetStream.ConsumerStateTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.JetStreamEngineTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamClusterTests1|FullyQualifiedName~ImplBacklog.JetStreamClusterTests3|FullyQualifiedName~ImplBacklog.JetStreamClusterTests4"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.AccountTests|FullyQualifiedName~ImplBacklog.MqttHandlerTests|FullyQualifiedName~ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ImplBacklog.JetStreamBenchmarks"
```
Checkpoint/full gate:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update)
- Never send more than `15` IDs per `feature batch-update` or `test batch-update`.
- Never promote `verified` without checkpoint evidence.
- Never update IDs outside active task scope.
- Blocked items remain `deferred` with concrete reason.
Use:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
$DOTNET run --project tools/NatsNet.PortTracker -- \
test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```
### Checkpoint Protocol Between Tasks (REQUIRED)
Before starting the next task:
1. Run **Stub Detection Check** on current diff.
2. Run **Build Gate**.
3. Run targeted **Test Gate** for touched classes.
4. Run full unit-test gate.
5. Apply status updates in `<=15` ID chunks only.
6. Commit checkpoint before moving on.
---
## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)
### Forbidden Patterns
Forbidden in mapped Batch 38 feature/test work:
- `throw new NotImplementedException()`
- Empty/no-op mapped method bodies
- Placeholder comments (`TODO`, `PLACEHOLDER`, `later`)
- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`)
- Template assertions disconnected from behavior:
- `var goFile = "server/..."`
- `"<MethodName>".ShouldContain("Should")`
- `GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty)` as a core assertion
- Constant-return shortcuts for mapped behavior without justification and coverage:
- `=> default;`
- `return default;`
- `return null;`
- Catch-and-ignore blocks used to force green tests
### Hard Limits
- Max feature group size: `~20`.
- Max IDs per `feature batch-update`: `15`.
- Max IDs per `test batch-update`: `15`.
- One active feature loop at a time.
- One active test loop at a time.
- Mandatory checkpoint between all tasks.
- No `verified` promotion without checkpoint evidence.
### If You Get Stuck (REQUIRED)
1. Stop on the current feature/test ID immediately.
2. Do **not** write placeholder logic or fake assertions.
3. Mark the item `deferred` with a specific blocker reason.
4. Continue with the next unblocked ID.
Feature deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
Test deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
`deferred` with a concrete reason is correct. Stubbing is not.
---
## Feature Groups (max ~20 each)
### Group A (19): enums, subject filters, config defaults/validation, add-consumer and assignment entry
IDs:
`584,585,586,587,588,589,590,591,592,594,595,596,597,598,599,600,601,602,603`
### Group B (19): monitor channels, leadership/advisory path, delivery-interest bridge to server
IDs:
`604,605,606,607,608,610,612,613,614,615,616,617,618,619,620,621,622,623,624`
### Group C (19): delivery-subject/rate-limit/ack ingress and early delivery state transitions
IDs:
`625,626,627,629,630,631,632,633,635,636,637,638,639,640,641,642,643,644,645`
### Group D (19): proposals, ack floor updates, pending-request cluster flow, nak/term/ackWait foundation
IDs:
`646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664`
### Group E (20): store-state persistence, info snapshots, sampling/filtering, next-request parsing, wait-queue recycle
IDs:
`665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684`
## Test Waves
### Wave T1 (19): consumer filters/actions/pinned behavior
IDs:
`1220,1221,1222,1223,1224,1225,1226,1228,1239,1240,1241,1242,1243,1244,1248,1250,1259,1260,1263`
### Wave T2 (19): consumer max-deliver/ack/rate/replay/multi-subject behavior
IDs:
`1266,1268,1274,1278,1279,1280,1281,1282,1289,1311,1312,1313,1316,1318,1319,1320,1321,1325,1344`
### Wave T3 (8): cluster consumer lifecycle consistency
IDs:
`812,870,892,1123,1124,1126,1172,1173`
### Wave T4 (12): core engine consumer lifecycle integration
IDs:
`96,1472,1473,1491,1494,1497,1505,1513,1523,1525,1527,1546`
### Wave T5 (13): perf/race/cross-protocol and benchmark-adjacent coverage
IDs:
`749,1557,1558,1559,1561,1624,1647,1670,1671,1764,2212,2385,2388`
---
### Task 1: Preflight, Baseline, and Batch Start
**Files:**
- Read: `docs/plans/2026-02-27-batch-38-consumer-lifecycle-design.md`
- Read: `golang/nats-server/server/consumer.go`
**Steps:**
1. Run dependency preflight for batches `34`, `36`, `38`.
2. Start Batch 38 only if ready.
3. Capture baseline build + full unit test.
4. Record the baseline counts before edits.
### Task 2: Implement Feature Group A (19 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.ConsumerPolicies.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Internal/SubjectTokens.ConsumerFilters.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Config.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Modify/Create tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerPoliciesTests.cs`
**Steps:**
1. Execute per-feature loop for each Group A ID.
2. Keep each status update batch at max 15 IDs.
3. Run checkpoint protocol.
### Task 3: Implement Feature Group B (19 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Lifecycle.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Advisories.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.ConsumerInterest.cs`
**Steps:**
1. Execute per-feature loop for each Group B ID.
2. Validate advisory and delivery-interest paths with focused tests.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 4: Implement Feature Group C (19 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.ConsumerConfig.cs`
- Modify tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs`
**Steps:**
1. Execute per-feature loop for each Group C ID.
2. Cover ack message creation/parsing and deliver-subject update behavior.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 5: Implement Feature Group D (19 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Acks.cs`
- Modify/Create tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/ConsumerStateTests.cs`
**Steps:**
1. Execute per-feature loop for each Group D ID.
2. Verify proposal forwarding, pending-request transitions, and `processNak/processTerm`.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 6: Implement Feature Group E (20 IDs)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.State.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.WaitQueue.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.ConsumerLifecycle.cs`
- Modify tests: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/WaitQueueTests.cs`
**Steps:**
1. Execute per-feature loop for each Group E ID.
2. Verify state read/write/info sampling/filter parsing/wait-queue recycle behavior.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 7: Port Test Wave T1 (19 IDs)
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
**Steps:**
1. Execute per-test loop for each T1 ID.
2. Replace template assertions with behavioral checks.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 8: Port Test Wave T2 (19 IDs)
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
**Steps:**
1. Execute per-test loop for each T2 ID.
2. Validate ack/rate/replay/multi-subject semantics using produced feature behavior.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 9: Port Test Wave T3 (8 IDs)
**Files:**
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs`
**Steps:**
1. Execute per-test loop for each T3 ID.
2. Keep cluster-only blockers as deferred with explicit reason if environment-dependent.
3. Run checkpoint protocol.
### Task 10: Port Test Wave T4 (12 IDs)
**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
**Steps:**
1. Execute per-test loop for each T4 ID.
2. Verify integration behavior for `nextReqFromMsg`, filtering, ack reply and lifecycle interactions.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 11: Port Test Wave T5 (13 IDs)
**Files:**
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBenchmarks.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
**Steps:**
1. Execute per-test loop for each T5 ID.
2. For benchmark-only cases not meaningful as unit tests, defer with explicit benchmark blocker reason.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol.
### Task 12: Final Verification, Status Closure, and Batch Completion
**Files:**
- Modify: `porting.db`
- Generate: `reports/current.md`
**Steps:**
1. Run full stub detection across changed source and backlog tests.
2. Run full build + full unit-test suite.
3. Verify all Batch 38 IDs are `complete/verified/n_a/deferred(with reason)` with no silent stubs.
4. Run:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db
./reports/generate-report.sh
```
5. Complete batch:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 38 --db porting.db
```

159
docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md Normal file
View File

@@ -0,0 +1,159 @@
# Batch 39 Consumer Dispatch Design
**Date:** 2026-02-27
**Batch:** 39 (`Consumer Dispatch`)
**Scope:** 93 features + 53 unit tests
**Dependencies:** Batch `38`
**Go source:** `golang/nats-server/server/consumer.go` (dispatch-heavy region around lines 3925-6706)
## Problem
Batch 39 contains the JetStream consumer dispatch/data-plane logic: pull request queueing, next-message request parsing, pending accounting, heartbeats/flow-control, delivery tracking, redelivery queue semantics, ack reply parsing, stream purge/stop paths, and monitor/signal flow.
If this batch is advanced with placeholders, the runtime behavior for pull consumers, redelivery correctness, and no-interest cleanup will look green in PortTracker while remaining behaviorally incomplete.
## Context Findings
### Required command outputs
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 39 --db porting.db`
- Status: `pending`
- Features: `93` (all currently `deferred`)
- Tests: `53` (all currently `deferred`)
- Depends on: `38`
- Go file: `server/consumer.go`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db`
- Confirms dependency chain `38 -> 39`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db`
- Overall progress snapshot: `1924/6942 (27.7%)`
Environment note: `dotnet` is not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`.
### Batch ownership split
Features by class:
- `NatsConsumer`: 91
- `NextMsgReq`: 1
- `NatsStream`: 1
Tests by class:
- `JetStreamEngineTests`: 34
- `NatsConsumerTests`: 14
- `ConcurrencyTests1`: 2
- `JetStreamClusterTests1`: 1
- `JwtProcessorTests`: 1
- `RouteHandlerTests`: 1
### Current .NET baseline findings
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` is still a compact lifecycle shell and does not expose most Batch 39 mapped methods.
- `WaitQueue` and `WaitingRequest` exist in `StreamTypes.cs`, but only part of the Go dispatch behavior is represented.
- `ImplBacklog` files for Batch 39-related tests are mostly template placeholders and need full behavioral ports.
## Approach Options
### Approach A: Keep all dispatch work in `NatsConsumer.cs`
- Pros: fewer files and simpler discovery.
- Cons: 90+ methods in one file, poor reviewability, high merge risk, and weaker checkpoint isolation.
### Approach B (Recommended): Dispatch-focused partial decomposition for `NatsConsumer`
- Pros: aligns with Go dispatch domains, improves review boundaries, supports group-by-group verification loops.
- Cons: more files and up-front organization overhead.
### Approach C: Port all 53 tests first, then implement features
- Pros: strict red/green pressure.
- Cons: too many missing APIs initially, broad low-signal red phase, high churn in test scaffolding.
**Decision:** Approach B.
## Proposed Design
### 1. Code Organization
Keep `NatsConsumer` as the central type but split dispatch concerns into partial files:
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` (core state, shared lock helpers, partial type anchor)
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Waiting.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Acks.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Delivery.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Redelivery.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.ReplyParsing.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Shutdown.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs` (`DeleteConsumer` mapping)
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` (`WaitQueue`, `WaitingRequest`, `PriorityGroup` helpers)
### 2. Functional Decomposition
- **Waiting queue and request ingress:** pending request map, pin assignment/ttl, request parsing (`ProcessNextMsgReq`, `ProcessResetReq`, `ProcessNextMsgRequest`).
- **Dispatch loop and pending accounting:** `GetNextMsg`, `ProcessWaiting`, `CheckAckFloor`, `LoopAndGatherMsgs`, pending counters.
- **Delivery and flow control:** heartbeat emission, flow-control reply generation, message delivery envelope conversion.
- **Redelivery and ack reply parsing:** pending tracking, redelivery queue operations, ack subject parsing (`ReplyInfo`, `AckReplyInfo`, seq extraction).
- **Lifecycle and stream-interest cleanup:** durable/push/pull helpers, purge/stop/cleanup, stream signals, monitor guard methods.
### 3. Test Design
Replace template tests with behavioral tests tied to mapped IDs:
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` (14)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` (34)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` (2)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` (1)
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` (1)
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` (1)
Also add/expand focused non-backlog unit tests where internal behaviors can be tested deterministically:
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerDispatchTests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs`
### 4. Verification Model (Design-level)
Batch 39 execution must be evidence-driven:
- Per-feature implementation loop with targeted tests.
- Per-test single-test and class-level verification.
- Stub scan before any status promotion.
- Build and test gates between groups.
- Status updates in max 15-ID chunks.
- Deferred-with-reason instead of stubbing when blocked.
## Feature Groups (max ~20 each)
- **Group A (18): waiting queue + pull request ingress**
`696,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715`
- **Group B (18): dispatch selection + ack floor + inbound loops**
`716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733`
- **Group C (19): delivery payload + flow control + redelivery foundation**
`734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752`
- **Group D (19): reply parsing + sequence selection + consumer identity helpers**
`753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,772`
- **Group E (19): stop/purge/no-interest/monitor signaling path**
`774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792`
## Test Waves
- **Wave T1 (14):** `1230,1232,1251,1261,1265,1267,1273,1277,1283,1284,1285,1286,1339,1370`
- **Wave T2 (17):** `1469,1484,1485,1486,1487,1488,1489,1490,1492,1493,1495,1496,1498,1499,1500,1501,1502`
- **Wave T3 (17):** `1508,1514,1515,1516,1517,1518,1519,1520,1521,1522,1526,1530,1531,1545,1547,1567,1665`
- **Wave T4 (5):** `814,1840,2389,2407,2858`
## Risks and Mitigations
- **Risk:** dispatch loops can mask bugs via timing-sensitive behavior.
**Mitigation:** isolate pure logic from timers/network side-effects; unit-test selectors and parsers first.
- **Risk:** placeholder tests could slip through due green-but-empty assertions.
**Mitigation:** mandatory stub scan and assertion-quality checks before status updates.
- **Risk:** over-bulk status updates can misclassify unverified IDs.
**Mitigation:** enforce max 15 IDs per update and per-task checkpoints.
## Non-Goals
- Executing Batch 39 implementation in this session.
- Re-scoping IDs outside Batch 39.
- Marking any feature/test as verified during planning.

418
docs/plans/2026-02-27-batch-39-consumer-dispatch-implementation-plan.md Normal file
View File

@@ -0,0 +1,418 @@
# Batch 39 Consumer Dispatch Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Port and verify Batch 39 (`Consumer Dispatch`) from `server/consumer.go` so all 93 mapped features and 53 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blocker reasons.
**Architecture:** Implement Batch 39 in five feature groups (`18/18/19/19/19`) with dispatch-focused `NatsConsumer` partial decomposition. After each feature group, execute mapped test waves and enforce strict evidence gates (stub scan, build gate, targeted tests, checkpoint) before any status promotion.
**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)
**Design doc:** `docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md`
---
## Batch 39 Scope
- Batch ID: `39`
- Name: `Consumer Dispatch`
- Dependency: `38`
- Go source: `golang/nats-server/server/consumer.go`
- Features: `93`
- Tests: `53`
If `dotnet` is not on `PATH`, use:
```bash
DOTNET=/usr/local/share/dotnet/dotnet
```
Primary production files (expected):
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Waiting.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Acks.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Delivery.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Redelivery.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.ReplyParsing.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Shutdown.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs`
Primary test files (expected):
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerDispatchTests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
---
## MANDATORY VERIFICATION PROTOCOL
> **NON-NEGOTIABLE:** Every Batch 39 feature and test must pass this protocol. No shortcuts.
### 1. Dependency Preflight and Baseline
Run before touching statuses:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 38 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 39 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```
Start Batch 39 only when ready:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch start 39 --db porting.db
```
Capture baseline:
```bash
$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### 2. What Counts as a Real Port
A **feature** is real only if all are true:
1. Mapped method/type exists and contains behavioral logic (not placeholder control flow).
2. Code path is exercised by at least one behavioral test.
3. Stub scan, build gate, and targeted tests are green.
4. Status promotion has traceable evidence.
A **test** is real only if all are true:
1. Has Arrange/Act/Assert.
2. Calls production code from `ZB.MOM.NatsNet.Server.*`.
3. Uses meaningful Shouldly assertions on Act results/effects.
4. Contains no stub/template patterns.
### 3. Per-Feature Verification Loop (REQUIRED per feature ID)
1. Inspect mapped feature and Go source intent:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
```
2. Claim the feature:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
```
3. Add/adjust at least one behavioral test for the feature.
4. Implement minimal correct logic.
5. Run **Stub Detection Check**.
6. Run **Build Gate**.
7. Run relevant **Test Gate** filters.
8. Promote proven feature to `complete` (or `verified` only after checkpoint evidence).
### 4. Per-Test Verification Loop (REQUIRED per test ID)
1. Inspect mapped test and Go source:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
```
2. Claim the test:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
```
3. Port complete behavior.
4. Run single test and verify discovery:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
```
5. Confirm `Passed: 1, Failed: 0` (never `Passed: 0`).
6. Run class-level filter.
7. Run **Stub Detection Check** + **Build Gate**.
8. Promote proven test to `complete` (or `verified` only after checkpoint evidence).
### 5. Stub Detection Check (REQUIRED after each feature/test loop)
```bash
changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$changed_files" ]; then
echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary<string, string>\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$|throw new Exception\\(\"TODO\"\\))"
fi
```
Any match blocks status promotion until fixed or explicitly deferred.
### 6. Build Gate (REQUIRED)
```bash
$DOTNET build dotnet/
```
Run after every feature loop, every test loop, before any `batch-update`, and at each task checkpoint.
### 7. Test Gate (REQUIRED)
Run the smallest relevant set after each change; run all of the following at checkpoints:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream.NatsConsumerTests|FullyQualifiedName~JetStream.NatsConsumerDispatchTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.JetStreamEngineTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog.JetStreamClusterTests1|FullyQualifiedName~ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ImplBacklog.JwtProcessorTests|FullyQualifiedName~ImplBacklog.RouteHandlerTests"
```
Checkpoint/full gate:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### 8. Status Update Protocol (HARD LIMIT: 15 IDs per batch-update)
- Never send more than `15` IDs per `feature batch-update` or `test batch-update` call.
- Never mark `verified` without checkpoint evidence.
- Never update IDs outside the active task scope.
- Keep blocked items `deferred` with specific reason text.
Use:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
$DOTNET run --project tools/NatsNet.PortTracker -- \
test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```
### 9. Checkpoint Protocol Between Tasks (REQUIRED)
Before moving to the next task:
1. Run **Stub Detection Check**.
2. Run **Build Gate**.
3. Run **Test Gate** (targeted filters + full unit test suite).
4. Apply status updates in max-15-ID chunks only.
5. Commit the task checkpoint before continuing.
---
## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)
### Forbidden Patterns
The following are forbidden in Batch 39 feature/test work:
- `throw new NotImplementedException()`
- Empty mapped method bodies / no-op ports
- `TODO` / `PLACEHOLDER` comments as unresolved implementation markers
- `Assert.True(true)` / `Assert.Pass()`
- Template assertions disconnected from behavior:
- `var goFile = "server/..."`
- `"<name>".ShouldContain("Should")`
- `GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty)` as primary assertion
- Placeholder return shortcuts in mapped code:
- `=> default;`
- `return default;`
- `return null;`
- Catch-and-ignore blocks used only to force green tests
### Hard Limits
- Max feature group size: `~20`.
- Max IDs per `feature batch-update`: `15`.
- Max IDs per `test batch-update`: `15`.
- One active feature loop at a time.
- One active test loop at a time.
- Mandatory checkpoint between tasks.
- No `verified` promotion without checkpoint evidence.
### If You Get Stuck (REQUIRED)
1. Stop on the current feature/test immediately.
2. Do **not** stub or fake-pass.
3. Mark item `deferred` with a specific blocker reason.
4. Continue with the next unblocked item.
Feature deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
Test deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
`deferred` with concrete reason is the correct fallback; stubs are not.
---
## Feature Groups (max ~20 each)
### Group A (18): waiting queue and pull ingress
IDs:
`696,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715`
### Group B (18): dispatch core, ack floor, inbound loops
IDs:
`716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733`
### Group C (19): delivery, flow-control, redelivery basics
IDs:
`734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752`
### Group D (19): reply parsing, seq selection, identity helpers
IDs:
`753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,772`
### Group E (19): purge/stop/no-interest/monitor signaling
IDs:
`774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792`
## Test Waves
### Wave T1 (14): `NatsConsumerTests`
IDs:
`1230,1232,1251,1261,1265,1267,1273,1277,1283,1284,1285,1286,1339,1370`
### Wave T2 (17): `JetStreamEngineTests` (pull/ack-first half)
IDs:
`1469,1484,1485,1486,1487,1488,1489,1490,1492,1493,1495,1496,1498,1499,1500,1501,1502`
### Wave T3 (17): `JetStreamEngineTests` (pending/purge-second half)
IDs:
`1508,1514,1515,1516,1517,1518,1519,1520,1521,1522,1526,1530,1531,1545,1547,1567,1665`
### Wave T4 (5): cross-suite integration/backlog edges
IDs:
`814,1840,2389,2407,2858`
---
### Task 1: Preflight and Baseline
**Files:**
- Read: `docs/plans/2026-02-27-batch-39-consumer-dispatch-design.md`
- Read: `golang/nats-server/server/consumer.go`
**Steps:**
1. Run dependency preflight and confirm Batch 39 readiness.
2. Start Batch 39.
3. Capture baseline build + full unit test results.
4. Record baseline pass/fail/skip counts.
### Task 2: Implement Feature Group A + Test Wave T1
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Waiting.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerTests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
**Steps:**
1. Process each feature ID in Group A through the per-feature loop.
2. Port all tests in Wave T1 with per-test loop.
3. Run checkpoint protocol.
4. Update Group A/T1 statuses in max-15-ID chunks.
5. Commit.
### Task 3: Implement Feature Group B
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Pull.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Acks.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/NatsConsumerDispatchTests.cs`
**Steps:**
1. Process Group B feature IDs through per-feature loop.
2. Add deterministic unit tests for dispatch loop and ack-floor transitions.
3. Run checkpoint protocol.
4. Update Group B feature statuses in max-15-ID chunks.
5. Commit.
### Task 4: Implement Feature Group C + Test Wave T2
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Delivery.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Redelivery.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
**Steps:**
1. Process Group C feature IDs through per-feature loop.
2. Port Wave T2 tests through per-test loop.
3. Run checkpoint protocol.
4. Update Group C/T2 statuses in max-15-ID chunks.
5. Commit.
### Task 5: Implement Feature Group D + Test Wave T3
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.ReplyParsing.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
**Steps:**
1. Process Group D feature IDs through per-feature loop.
2. Port Wave T3 tests through per-test loop.
3. Run checkpoint protocol.
4. Update Group D/T3 statuses in max-15-ID chunks.
5. Commit.
### Task 6: Implement Feature Group E + Test Wave T4
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Dispatch.Shutdown.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs`
**Steps:**
1. Process Group E feature IDs through per-feature loop.
2. Port Wave T4 tests through per-test loop.
3. Run checkpoint protocol.
4. Update Group E/T4 statuses in max-15-ID chunks.
5. Commit.
### Task 7: Final Batch 39 Verification and Closure
**Files:**
- Modify: `porting.db`
- Generate: `reports/current.md`
**Steps:**
1. Run full stub scan over changed source + test files.
2. Run full build gate.
3. Run full unit test suite; require `Failed: 0`.
4. Run:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 39 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db
```
5. If all IDs are complete/verified/deferred-with-reason, execute batch completion:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 39 --db porting.db
```
6. Generate updated report and commit:
```bash
./reports/generate-report.sh
git add dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests porting.db reports/
git commit -m "feat(batch39): implement consumer dispatch features and tests"
```

136
docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md Normal file
View File

@@ -0,0 +1,136 @@
# Batch 40 MQTT Server/JSA Design
**Date:** 2026-02-27
**Batch:** 40 (`MQTT Server/JSA`)
**Scope:** 78 features + 323 unit tests
**Dependencies:** Batches `19` (Accounts Core), `27` (JetStream Core)
**Go source:** `golang/nats-server/server/mqtt.go` (Batch scope through line ~3500)
## Problem
Batch 40 is the core MQTT server-side control plane and JetStream adapter layer:
- MQTT listener/bootstrap and option validation
- MQTT parser/trace/header helpers
- Per-account session manager and retained-message handling
- MQTT JetStream request/reply adapter methods
- MQTT session persistence and packet identifier tracking
If this batch is advanced with placeholder implementations or placeholder tests, PortTracker can look healthy while MQTT behavior remains non-functional.
## Context Findings
### Required command outputs
Executed with explicit dotnet path:
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db`
- Status: `pending`
- Features: `78` (all `deferred`)
- Tests: `323` (all `deferred`)
- Depends on: `19,27`
- Go file: `server/mqtt.go`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db`
- Confirms dependency chain includes `19 -> 40` and `27 -> 40`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db`
- Overall snapshot: `1924/6942 (27.7%)`
Environment note: `dotnet` is not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`.
### Current .NET baseline
- MQTT types/constants exist in:
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs`
- Core behavior is mostly stubs:
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- Batch 40 mapped features target classes not fully represented yet (notably `MqttJetStreamAdapter`).
- Batch 40 mapped tests are mostly placeholder-template tests in `ImplBacklog` files with non-behavioral assertions.
## Approach Options
### Approach A: Keep all MQTT behavior in one file (`MqttHandler.cs`)
- Pros: minimal file churn.
- Cons: difficult review boundaries, high merge risk, and weak checkpoint isolation for 78 features.
### Approach B: Port tests first, then features
- Pros: strict red/green pressure.
- Cons: too many missing APIs and placeholder methods produce noisy red phase with low signal.
### Approach C (Recommended): Domain-sliced feature groups + staged test waves
- Pros: aligns with Go method clusters, enables per-group verification and status updates, and supports strict anti-stub gates.
- Cons: requires planned decomposition across several files and explicit mapping alignment.
**Decision:** Approach C.
## Proposed Design
### 1. Code organization and ownership
Implement Batch 40 in MQTT-focused partial/domain files so each group has clear ownership:
- `NatsServer` MQTT entrypoints (start/config/session manager bridge)
- `ClientConnection` MQTT parser/client-id/trace helpers
- `MqttJetStreamAdapter` request/reply and stream/consumer/message operations
- `MqttAccountSessionManager` retained/session orchestration
- `MqttSession` lifecycle/persistence/pending publish tracking
- Stateless helpers for retained-message encoding/decoding and header parsing
This keeps behavioral slices reviewable while preserving .NET naming and namespace rules.
### 2. Feature grouping strategy (<= 20 per group)
- Group A (20): MQTT bootstrap/parser/auth and request-builder foundations
- Group B (20): JSA operations + JS API reply processing bridge
- Group C (20): session-manager retained/subscription core
- Group D (18): retained encode/decode + session lifecycle tracking
Each group is independently verifiable and status-updated in controlled ID chunks.
### 3. Test strategy for 323 mapped tests
Use wave-based execution by test class risk profile:
- MQTT-focused and deterministic first (`MqttHandlerTests`, helper-centric classes)
- then consumer/engine deterministic slices
- cluster/supercluster/benchmark/norace-heavy tests last, with explicit deferral when runtime infra is required
Rule: deferred with concrete blocker reason is valid; fake pass/stub is not.
### 4. Mapping alignment
Because mapped class names include `MqttJetStreamAdapter` and method names using Go-style labels, include an early mapping-alignment checkpoint:
- either implement mapped classes/methods directly,
- or add well-documented wrappers/adapters that preserve mapped method discoverability.
No status promotion happens before this alignment compiles and has test coverage.
### 5. Verification model (design-level)
Batch 40 execution must be evidence-driven and include:
- per-feature verification loop
- per-test verification loop
- stub detection scan before any promotion
- build gate and targeted/full test gates
- checkpoint protocol between tasks
- status updates with hard limit of 15 IDs per command
## Risks and Mitigations
- Risk: large count of integration/cluster tests may encourage placeholder conversions.
Mitigation: mandatory anti-stub patterns and forced deferred-with-reason fallback.
- Risk: mapping/class drift between tracker metadata and code structure.
Mitigation: explicit mapping-alignment task before major implementation.
- Risk: broad behavioral surface across parser/session/JSA code paths.
Mitigation: 4 feature groups with mandatory checkpoint gates after each task.
## Non-Goals
- Executing Batch 40 in this planning session.
- Re-scoping IDs outside Batch 40.
- Mass-promoting statuses without per-item evidence.

427
docs/plans/2026-02-27-batch-40-mqtt-server-jsa-implementation-plan.md Normal file
View File

@@ -0,0 +1,427 @@
# Batch 40 MQTT Server/JSA Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Port and verify Batch 40 (`MQTT Server/JSA`) from `server/mqtt.go` so all 78 mapped features and 323 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blocker reasons.
**Architecture:** Implement Batch 40 in four feature groups (`20/20/20/18`) aligned to MQTT bootstrap/parser, JSA adapter, account session manager, and session persistence/retained-message codec behavior. After each feature group, execute mapped test waves with strict evidence gates before any status promotion.
**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)
**Design doc:** `docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md`
---
## Batch 40 Scope
- Batch ID: `40`
- Name: `MQTT Server/JSA`
- Dependencies: `19`, `27`
- Go source: `golang/nats-server/server/mqtt.go`
- Features: `78`
- Tests: `323`
If `dotnet` is not on `PATH`, use:
```bash
DOTNET=/usr/local/share/dotnet/dotnet
```
Expected production files (create/modify as needed):
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (MQTT partials)
- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection.cs` (or MQTT partial)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttJetStreamAdapter*.cs` (new if absent)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttAccountSessionManager*.cs` (new/partials)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs` (new/partials)
Expected test files (create/modify as needed):
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBatchingTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MessageTracerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs`
- Create missing mapped backlog files as needed:
- `JetStreamClusterTests1.Impltests.cs`
- `JetStreamClusterTests3.Impltests.cs`
- `JetStreamClusterTests4.Impltests.cs`
- `JetStreamSuperClusterTests.Impltests.cs`
- `JetStreamClusterLongTests.Impltests.cs`
- `JetStreamSourcingScalingTests.Impltests.cs`
- `JetStreamBenchmarks.Impltests.cs`
---
## MANDATORY VERIFICATION PROTOCOL
> **NON-NEGOTIABLE:** Every Batch 40 feature and test must pass this protocol. No shortcuts.
### 1. Dependency Preflight and Baseline
Run before any status changes:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 19 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```
Start batch only when ready:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch start 40 --db porting.db
```
Capture baseline:
```bash
$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### 2. Per-Feature Verification Loop (REQUIRED for each feature ID)
1. Inspect mapping and Go source intent:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
```
2. Claim the feature:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
```
3. Add/adjust at least one behavioral test path for this feature.
4. Implement minimal correct logic (no placeholders).
5. Run **Stub Detection Check**.
6. Run **Build Gate**.
7. Run relevant **Test Gate** filters.
8. Promote feature to `complete` only after evidence is captured.
9. Promote to `verified` only during task checkpoint after full gates pass.
### 3. Per-Test Verification Loop (REQUIRED for each test ID)
1. Inspect mapped test:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
```
2. Claim the test:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
```
3. Port full Arrange/Act/Assert behavior.
4. Run single-test filter and verify discovery:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
```
5. Confirm `Passed: 1, Failed: 0` (never accept `Passed: 0`).
6. Run class-level filter.
7. Run **Stub Detection Check** and **Build Gate**.
8. Promote test to `complete`; defer `verified` to checkpoint.
### 4. Stub Detection Check (REQUIRED)
Run after each feature loop, each test loop, and before any status promotion:
```bash
changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\.cs$" || true)
if [ -n "$changed_files" ]; then
echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass\(\)|var goFile = \"server/|\.ShouldContain\(\"Should\"\)|GetRequiredApiLevel\(new Dictionary<string, string>\)\.ShouldBe\(string\.Empty\)|=>\s*default;|return\s+default;|return\s+null;\s*$)"
fi
```
Any match blocks status promotion until fixed or explicitly deferred.
### 5. Build Gate (REQUIRED)
```bash
$DOTNET build dotnet/
```
Run after each feature loop, each test loop, before any batch-update, and at every task checkpoint.
### 6. Test Gate (REQUIRED)
Run smallest relevant filters continuously; run all sections at checkpoints.
MQTT-first filter:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ImplBacklog.MqttHandlerTests|FullyQualifiedName~ImplBacklog.MessageTracerTests|FullyQualifiedName~ImplBacklog.ConfigReloaderTests"
```
Deterministic JS/consumer filter:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.JetStreamBatchingTests|FullyQualifiedName~ImplBacklog.JetStreamEngineTests|FullyQualifiedName~ImplBacklog.JetStreamJwtTests|FullyQualifiedName~ImplBacklog.JetStreamFileStoreTests|FullyQualifiedName~ImplBacklog.JetStreamLeafNodeTests"
```
Cluster/concurrency filter:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ImplBacklog.JetStreamClusterTests1|FullyQualifiedName~ImplBacklog.JetStreamClusterTests2|FullyQualifiedName~ImplBacklog.JetStreamClusterTests3|FullyQualifiedName~ImplBacklog.JetStreamClusterTests4|FullyQualifiedName~ImplBacklog.JetStreamSuperClusterTests|FullyQualifiedName~ImplBacklog.JetStreamClusterLongTests|FullyQualifiedName~ImplBacklog.JetStreamSourcingScalingTests|FullyQualifiedName~ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ImplBacklog.ConcurrencyTests2|FullyQualifiedName~ImplBacklog.JetStreamBenchmarks"
```
Checkpoint/full gate:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### 7. Status Update Protocol (HARD LIMIT: 15 IDs per batch-update)
- Never send more than `15` IDs per `feature batch-update` command.
- Never send more than `15` IDs per `test batch-update` command.
- Never promote `verified` without checkpoint evidence.
- Never update IDs outside active task scope.
- Deferred items must include explicit blocker reason.
Use:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
$DOTNET run --project tools/NatsNet.PortTracker -- \
test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```
### 8. Checkpoint Protocol Between Tasks (REQUIRED)
Before moving to the next task:
1. Run **Stub Detection Check**.
2. Run **Build Gate**.
3. Run **Test Gate** (targeted filters + full unit suite).
4. Apply status updates in <=15-ID chunks only.
5. Run:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db
```
6. Commit checkpoint before continuing.
---
## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)
### Forbidden Patterns
The following patterns are forbidden for Batch 40 feature/test work:
- `throw new NotImplementedException(...)` in mapped feature methods
- Empty method bodies for mapped features
- `TODO` / `PLACEHOLDER` markers left in mapped logic
- `Assert.True(true)` / `Assert.Pass()`
- Placeholder template assertions disconnected from behavior:
- `var goFile = "server/..."`
- `"<name>".ShouldContain("Should")`
- `GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty)` as primary assertion
- Placeholder return shortcuts in mapped code:
- `=> default;`
- `return default;`
- `return null;`
### Hard Limits
- Max feature group size: `~20` features.
- Max IDs per `feature batch-update`: `15`.
- Max IDs per `test batch-update`: `15`.
- One active feature loop at a time.
- One active test loop at a time.
- Mandatory checkpoint between tasks.
- No `verified` promotion without checkpoint evidence.
### If You Get Stuck (REQUIRED)
1. Stop current item immediately.
2. Do not stub or fake-pass.
3. Mark item `deferred` with explicit blocker reason.
4. Continue with next unblocked feature/test.
Feature deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
Test deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
`deferred` with concrete reason is correct. Stubs are not.
---
## Feature Groups (max ~20 each)
### Group A (20): bootstrap/parser/auth/JSA request foundations
IDs:
`2252,2253,2254,2255,2257,2258,2259,2260,2261,2262,2263,2264,2265,2266,2267,2268,2269,2270,2271,2272`
### Group B (20): JSA operations + reply processing bridge
IDs:
`2273,2274,2275,2276,2277,2278,2279,2280,2281,2282,2283,2284,2285,2286,2287,2288,2289,2290,2291,2292`
### Group C (20): account session manager retained/subscription core
IDs:
`2293,2294,2295,2296,2297,2298,2299,2300,2301,2302,2303,2304,2305,2306,2307,2308,2309,2310,2311,2312`
### Group D (18): retained encode/decode + session lifecycle/publish tracking
IDs:
`2313,2314,2315,2316,2317,2318,2319,2320,2321,2322,2323,2324,2325,2326,2327,2328,2329,2330`
## Test Waves (323 total)
- Wave T1 (51): `MqttHandlerTests`, `MessageTracerTests`, `ConfigReloaderTests`
- Wave T2 (89): `NatsConsumerTests`, `JetStreamBatchingTests`, `JetStreamEngineTests`, `JetStreamJwtTests`, `JetStreamFileStoreTests`, `JetStreamLeafNodeTests`
- Wave T3 (70): `JetStreamClusterTests1`, `JetStreamClusterTests2`
- Wave T4 (77): `JetStreamClusterTests3`, `JetStreamClusterTests4`, `JetStreamSuperClusterTests`, `JetStreamClusterLongTests`, `JetStreamSourcingScalingTests`
- Wave T5 (36): `ConcurrencyTests1`, `ConcurrencyTests2`, `JetStreamBenchmarks`
ID extraction helper:
```bash
sqlite3 -header -column porting.db "
SELECT bt.test_id AS id, t.dotnet_class, t.name
FROM batch_tests bt
JOIN unit_tests t ON t.id=bt.test_id
WHERE bt.batch_id=40
AND t.dotnet_class IN (<wave classes here>)
ORDER BY bt.test_id;"
```
---
### Task 1: Preflight, Mapping Alignment, and Baseline
**Files:**
- Read: `docs/plans/2026-02-27-batch-40-mqtt-server-jsa-design.md`
- Read: `golang/nats-server/server/mqtt.go`
- Modify (if needed): `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/*` for class/method mapping alignment
**Steps:**
1. Run dependency preflight and start Batch 40.
2. Confirm mapped class/method targets compile (notably `MqttJetStreamAdapter`).
3. Capture baseline build + full unit test counts.
4. Run checkpoint protocol.
### Task 2: Implement Feature Group A + Test Wave T1
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (MQTT methods)
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs` (MQTT parse/trace helpers)
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/MessageTracerTests.Impltests.cs`, `ImplBacklog/ConfigReloaderTests.Impltests.cs`
**Steps:**
1. Process each Group A feature through per-feature loop.
2. Port/verify Wave T1 tests through per-test loop.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol and commit.
### Task 3: Implement Feature Group B + Test Wave T2 (Part 1)
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttJetStreamAdapter*.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttAccountSessionManager*.cs`
- Modify tests: `ImplBacklog/JetStreamBatchingTests.Impltests.cs`, `ImplBacklog/JetStreamEngineTests.Impltests.cs`, `ImplBacklog/NatsConsumerTests.Impltests.cs`, `ImplBacklog/JetStreamJwtTests.Impltests.cs`, `ImplBacklog/JetStreamFileStoreTests.Impltests.cs`, `ImplBacklog/JetStreamLeafNodeTests.Impltests.cs`
**Steps:**
1. Process each Group B feature through per-feature loop.
2. Port deterministic tests from Wave T2 first; defer runtime-blocked tests with reason.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol and commit.
### Task 4: Implement Feature Group C + Test Wave T3
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttAccountSessionManager*.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs`
- Modify/Create tests: `ImplBacklog/JetStreamClusterTests1.Impltests.cs`, `ImplBacklog/JetStreamClusterTests2.Impltests.cs`
**Steps:**
1. Process each Group C feature through per-feature loop.
2. Execute Wave T3 with strict infra eligibility checks.
3. Defer cluster-runtime blocked tests explicitly; do not stub.
4. Apply status updates in <=15-ID chunks.
5. Run checkpoint protocol and commit.
### Task 5: Implement Feature Group D + Test Waves T4/T5
**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs`
- Modify/Create tests:
- `ImplBacklog/JetStreamClusterTests3.Impltests.cs`
- `ImplBacklog/JetStreamClusterTests4.Impltests.cs`
- `ImplBacklog/JetStreamSuperClusterTests.Impltests.cs`
- `ImplBacklog/JetStreamClusterLongTests.Impltests.cs`
- `ImplBacklog/JetStreamSourcingScalingTests.Impltests.cs`
- `ImplBacklog/ConcurrencyTests1.Impltests.cs`
- `ImplBacklog/ConcurrencyTests2.Impltests.cs`
- `ImplBacklog/JetStreamBenchmarks.Impltests.cs`
**Steps:**
1. Process each Group D feature through per-feature loop.
2. Execute Waves T4/T5 with strict defer-with-reason handling.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol and commit.
### Task 6: Batch 40 Closure Verification
**Files:**
- Modify: `porting.db`
- Generate: `reports/current.md` (via report script)
**Steps:**
1. Run final stub detection across changed MQTT source/test files.
2. Run `build` and full unit test suite.
3. Verify Batch 40 state:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db
```
4. Generate updated report:
```bash
./reports/generate-report.sh
```
5. Final checkpoint commit.

132
docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md Normal file
View File

@@ -0,0 +1,132 @@
# Batch 41 MQTT Client/IO Design
**Date:** 2026-02-27
**Batch:** 41 (`MQTT Client/IO`)
**Scope:** 74 features + 28 unit tests
**Dependency:** Batch `40` (`MQTT Server/JSA`)
**Go source:** `golang/nats-server/server/mqtt.go` (line ~3503 through ~5882)
## Problem
Batch 41 is the MQTT client protocol and I/O execution surface:
- CONNECT/PUBLISH/SUBSCRIBE/UNSUBSCRIBE parse and response paths
- QoS1/QoS2 flow control and PUBREL lifecycle
- retain handling and retained-message permission checks
- topic/filter <-> subject conversion logic
- byte-level reader/writer utilities used across MQTT packet handling
This batch can easily appear "complete" via placeholders because many mapped tests are currently template-style and several mapped test IDs require non-MQTT infrastructure. The design must force evidence-based completion and explicit deferrals instead of stubs.
## Context Findings
### Required command outputs
Executed with explicit runtime path because `dotnet` is not on PATH in this shell:
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db`
- Status: `pending`
- Features: `74` (all currently `deferred`)
- Tests: `28` (all currently `deferred`)
- Depends on: `40`
- Go file: `server/mqtt.go`
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db`
- Confirms Batch 41 is the final batch and depends only on Batch 40.
- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db`
- Current snapshot: `1924/6942 (27.7%)`.
### Current .NET baseline
- MQTT types/constants exist in:
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs`
- Batch 41 mapped methods are not implemented yet and many target methods/classes are currently absent.
- `MqttHandler.cs` still contains broad NotImplemented stubs in server extension methods.
- `MqttHandlerTests.Impltests.cs` and related backlog files still include placeholder assertion patterns.
- Mapped test set includes cross-module classes (`AuthCalloutTests`, `NatsConsumerTests`, `LeafNodeHandlerTests`, `WebSocketHandlerTests`, `JetStreamClusterTests3`) in addition to MQTT-specific classes.
## Approach Options
### Approach A: Single-file incremental port in `MqttHandler.cs`
- Pros: minimal file churn.
- Cons: high merge conflict risk, poor reviewability, weak ownership boundaries for 74 methods.
### Approach B: Tests-first for all 28 IDs before feature work
- Pros: strict red/green discipline.
- Cons: too many missing method surfaces creates high-noise red phase; many test IDs depend on deferred non-B41 features.
### Approach C (Recommended): Feature-sliced implementation groups + test waves with explicit deferred protocol
- Pros: aligns with Go function clusters, keeps each group <=20 features, enables mandatory checkpoint evidence, and handles cross-module tests without fake pass pressure.
- Cons: requires initial mapping-alignment step and disciplined status batching.
**Decision:** Approach C.
## Proposed Design
### 1. File and component layout
Implement Batch 41 in focused MQTT slices (new files/partials as needed), not a single monolith:
- `ClientConnection` MQTT packet handlers/parsers
- `MqttParseConnect`, `MqttParsePub`, `MqttParseSubsOrUnsubs`, `MqttProcessPublishReceived`, enqueue acks
- `NatsServer` MQTT publish/session bridge handlers
- `MqttProcessConnect`, `MqttProcessPub`, `MqttProcessPubRel`, retained permissions audit
- `MqttSession` QoS2 consumer/pubrel helpers
- `TrackAsPubRel`, `UntrackPubRel`, `DeleteConsumer`, `ProcessJSConsumer`
- `MqttReader` / `MqttWriter` byte I/O utilities
- varint, length-prefixed strings/bytes, publish header encoding
- Stateless conversion/trace/helper methods
- topic/filter conversion, reserve-sub logic, trace formatters, Sparkplug-B helpers
### 2. Feature grouping model (max ~20 each)
- Group A (19): session + connect + initial publish path (`2331-2349`)
- Group B (19): retained/perms + QoS ack processing + subscribe callbacks (`2350-2368`)
- Group C (17): reserved-sub/sparkplug + subscribe processing + unsubscribe/ping (`2369-2385`)
- Group D (19): conversion + reader/writer I/O (`2386-2404`)
### 3. Test-wave model (28 tests)
- Wave T1 (10): deterministic MQTT parser/conversion tests (`2170,2171,2190,2191,2194,2195,2196,2199,2200,2229`)
- Wave T2 (11): publish/retain/session behavior tests (`2182,2204,2234,2235,2236,2237,2238,2246,2251,2253,2285`)
- Wave T3 (4): cross-module integration-touching tests (`115,1258,1924,3095`)
- Wave T4 (3): cluster-dependent tests (`1055,1056,1113`)
### 4. Deferred strategy by design
Because some mapped tests depend on deferred non-B41 features, completion criteria is:
- implement and verify what is truly executable in local unit-test context,
- mark blockers `deferred` with specific reason and evidence,
- never substitute with placeholders.
### 5. Verification architecture
The implementation plan will enforce:
- per-feature verification loop before promotion,
- per-test verification loop with discovered/pass evidence,
- stub detection checks on source and tests,
- build/test gates before every status update,
- status updates limited to <=15 IDs per batch-update,
- mandatory checkpoints between task groups.
## Risks and Mitigations
- Risk: class/method mapping drift (mapped methods not yet present).
- Mitigation: dedicated mapping-alignment preflight task before feature status changes.
- Risk: placeholder tests pass without exercising MQTT logic.
- Mitigation: anti-stub scans + assertion minimum checks + single-test evidence requirement.
- Risk: cluster/integration tests block throughput.
- Mitigation: explicit deferred-with-reason path and continue with unblocked items.
- Risk: large parser/I/O surface causes hidden regressions.
- Mitigation: incremental group checkpoints with full build + targeted/full test gates.
## Non-Goals
- Executing Batch 41 during this planning session.
- Marking statuses without verification evidence.
- Re-scoping features outside Batch 41 except documented test dependency blockers.

388
docs/plans/2026-02-27-batch-41-mqtt-client-io-implementation-plan.md Normal file
View File

@@ -0,0 +1,388 @@
# Batch 41 MQTT Client/IO Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Port and verify Batch 41 (`MQTT Client/IO`) so all 74 mapped features and 28 mapped tests are either truly implemented and verified or explicitly deferred with concrete blocker reasons.
**Architecture:** Implement in four feature groups (`19/19/17/19`) aligned to connect/publish/session handling, retained and QoS flow, subscription lifecycle, and reader/writer conversion I/O. Execute test waves after each group with strict evidence gates and explicit deferred handling for runtime-blocked cross-module tests.
**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)
**Design doc:** `docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md`
---
## Batch 41 Scope
- Batch ID: `41`
- Name: `MQTT Client/IO`
- Dependency: Batch `40`
- Go source: `golang/nats-server/server/mqtt.go`
- Feature IDs: `2331-2404` (74)
- Test IDs: `115,1055,1056,1113,1258,1924,2170,2171,2182,2190,2191,2194,2195,2196,2199,2200,2204,2229,2234,2235,2236,2237,2238,2246,2251,2253,2285,3095`
Use this runtime path if `dotnet` is unavailable on PATH:
```bash
DOTNET=/usr/local/share/dotnet/dotnet
```
Expected source files (create/modify as needed):
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttTypes.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttConstants.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs` (MQTT partials)
- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs` (MQTT partials)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttReader*.cs` (new/partial)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttWriter*.cs` (new/partial)
- `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttSession*.cs` (new/partial)
Expected test files (create/modify as needed):
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttExternalTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthCalloutTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` (create if missing)
---
## MANDATORY VERIFICATION PROTOCOL
> **NON-NEGOTIABLE:** Every Batch 41 feature/test item must pass this protocol before status promotion.
### 1. Dependency and Baseline Preflight (REQUIRED)
Before touching any status:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 40 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```
Start only when Batch 41 is ready:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch start 41 --db porting.db
```
Capture baseline:
```bash
$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### 2. Per-Feature Verification Loop (REQUIRED per feature ID)
For each feature ID in active group:
1. Inspect mapping and Go behavior:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
```
2. Claim the feature:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
```
3. Implement minimal correct logic in mapped .NET class/method.
4. Add or update at least one behavioral test path covering this feature.
5. Run stub detection checks (source + test).
6. Run build gate.
7. Run targeted test gate for touched classes/methods.
8. Promote to `complete` only when gates pass.
9. Promote to `verified` only at task checkpoint.
### 3. Per-Test Verification Loop (REQUIRED per test ID)
For each mapped test ID in active wave:
1. Inspect mapped test details:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
```
2. Claim the test:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
```
3. Port full Arrange/Act/Assert behavior (no placeholders).
4. Run single-test filter and verify discovery:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
```
5. Confirm output includes `Passed: 1, Failed: 0` (never accept `Passed: 0`).
6. Run class-level test filter.
7. Run stub detection + build gate.
8. Promote to `complete`; promote `verified` only at checkpoint.
### 4. Stub Detection Check (REQUIRED)
Run after each feature loop, each test loop, and before any status update.
Source stub scan:
```bash
src_changed=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server | rg "\\.cs$" || true)
if [ -n "$src_changed" ]; then
echo "$src_changed" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|=>\\s*default;|return\\s+default;|return\\s+null;|throw\\s+new\\s+NotImplementedException)"
fi
```
Test stub scan:
```bash
test_changed=$(git diff --name-only -- dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$test_changed" ]; then
echo "$test_changed" | xargs rg -n "(NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|// TODO|// PLACEHOLDER|goFile\\s*=\\s*\"server/|ShouldContain\\(\"Should\"\\)|ShouldBe\\(string\\.Empty\\))"
fi
```
Assertion depth check for changed test files:
```bash
# per file: Shouldly assertion count / test count should be >= 1.5
```
### 5. Build Gate (REQUIRED)
```bash
$DOTNET build dotnet/
```
Run after every feature/test loop, before any batch-update, and at every checkpoint.
### 6. Test Gate (REQUIRED)
Run minimal relevant filters continuously, and full suite at checkpoints.
MQTT core filters:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ImplBacklog.MqttHandlerTests|FullyQualifiedName~ImplBacklog.MqttExternalTests"
```
Cross-module mapped filters:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
--filter "FullyQualifiedName~ImplBacklog.AuthCalloutTests|FullyQualifiedName~ImplBacklog.NatsConsumerTests|FullyQualifiedName~ImplBacklog.LeafNodeHandlerTests|FullyQualifiedName~ImplBacklog.WebSocketHandlerTests|FullyQualifiedName~ImplBacklog.JetStreamClusterTests3"
```
Checkpoint full unit gate:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
### 7. Status Update Protocol (HARD LIMIT: 15 IDs)
- Maximum `15` IDs per `feature batch-update` call.
- Maximum `15` IDs per `test batch-update` call.
- Never promote `verified` without checkpoint evidence.
- Deferred items must include explicit blocker reason.
- Status updates are scoped to the active task only.
Commands:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
$DOTNET run --project tools/NatsNet.PortTracker -- \
test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```
### 8. Checkpoint Protocol Between Tasks (REQUIRED)
Before starting next task:
1. Run source + test stub detection scans.
2. Run build gate.
3. Run targeted test filters plus full unit test suite.
4. Apply status updates in <=15-ID chunks.
5. Run batch progress check:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db
```
6. Commit checkpoint before moving on.
---
## ANTI-STUB GUARDRAILS
> **NON-NEGOTIABLE:** Placeholder behavior is not valid progress.
### Forbidden Patterns
Any of these in mapped feature/test bodies blocks status promotion:
- `throw new NotImplementedException(...)`
- Empty mapped method body
- `// TODO` or `// PLACEHOLDER` left in mapped logic
- `=> default;`, `return default;`, `return null;` in mapped feature methods
- `Assert.True(true)` or `Assert.Pass()`
- Placeholder template assertions not tied to behavior:
- `var goFile = "server/..."` as test core
- `"...".ShouldContain("Should")`
- `GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty)` as primary assertion
- Test methods with only trivial constructor/null checks for complex protocol behavior
### Hard Limits
- Feature groups must stay <=20 IDs.
- Max 15 IDs per status batch-update command.
- One active feature loop at a time.
- One active test loop at a time.
- Mandatory checkpoint between tasks.
- No cross-task bulk status updates.
- No `verified` promotion without checkpoint evidence.
### If You Get Stuck (REQUIRED)
If an item needs infra or dependencies not available:
1. Stop work on that ID.
2. Do not add placeholders or force a pass.
3. Mark `deferred` with exact blocker reason.
4. Continue with the next unblocked ID.
Feature deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
Test deferral:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```
`deferred` with concrete reason is correct behavior. Stubs are not.
---
## Feature Groups (max ~20 each)
### Group A (19): session, CONNECT, publish ingress
IDs:
`2331,2332,2333,2334,2335,2336,2337,2338,2339,2340,2341,2342,2343,2344,2345,2346,2347,2348,2349`
### Group B (19): retained permissions, QoS ack flow, subscribe callbacks
IDs:
`2350,2351,2352,2353,2354,2355,2356,2357,2358,2359,2360,2361,2362,2363,2364,2365,2366,2367,2368`
### Group C (17): reserved-sub/sparkplug, subscribe/unsubscribe processing
IDs:
`2369,2370,2371,2372,2373,2374,2375,2376,2377,2378,2379,2380,2381,2382,2383,2384,2385`
### Group D (19): subject conversion and reader/writer I/O
IDs:
`2386,2387,2388,2389,2390,2391,2392,2393,2394,2395,2396,2397,2398,2399,2400,2401,2402,2403,2404`
## Test Waves (28 total)
- Wave T1 (10): `2170,2171,2190,2191,2194,2195,2196,2199,2200,2229`
- Wave T2 (11): `2182,2204,2234,2235,2236,2237,2238,2246,2251,2253,2285`
- Wave T3 (4): `115,1258,1924,3095`
- Wave T4 (3): `1055,1056,1113`
---
### Task 1: Preflight, Mapping Alignment, and Baseline
**Files:**
- Read: `docs/plans/2026-02-27-batch-41-mqtt-client-io-design.md`
- Read: `golang/nats-server/server/mqtt.go`
- Modify/Create: Batch 41 mapped class/method surfaces under `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/`, `dotnet/src/ZB.MOM.NatsNet.Server/ClientConnection*.cs`, and `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer*.cs`
**Steps:**
1. Verify Batch 40 completion/readiness and start Batch 41.
2. Ensure all mapped Batch 41 methods exist and compile.
3. Capture baseline build/test state.
4. Run checkpoint protocol.
### Task 2: Implement Group A + Wave T1 (Parser/Connect Core)
**Files:**
- Modify/Create: `ClientConnection` MQTT parse/connect files
- Modify/Create: `NatsServer` MQTT connect/publish ingress files
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Mqtt/MqttHandler.cs`
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/MqttExternalTests.Impltests.cs`
**Steps:**
1. Process all Group A feature IDs through per-feature loop.
2. Port/verify T1 parser/conversion tests through per-test loop.
3. Apply status updates in <=15-ID chunks.
4. Run checkpoint protocol and commit.
### Task 3: Implement Group B + Wave T2 (Retain/QoS Core)
**Files:**
- Modify/Create: MQTT retained/perms and QoS ack handling files
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`
**Steps:**
1. Process Group B features through per-feature loop.
2. Port/verify T2 tests tied to retain, QoS flags, pub ack flow.
3. Defer blocked IDs with explicit reasons (no stubs).
4. Apply <=15-ID status updates.
5. Run checkpoint protocol and commit.
### Task 4: Implement Group C + Remaining T2/T3 (Subscription Lifecycle)
**Files:**
- Modify/Create: MQTT subscription/publish delivery/session consumer files
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/AuthCalloutTests.Impltests.cs`, `ImplBacklog/NatsConsumerTests.Impltests.cs`, `ImplBacklog/LeafNodeHandlerTests.Impltests.cs`, `ImplBacklog/WebSocketHandlerTests.Impltests.cs`
**Steps:**
1. Process Group C features through per-feature loop.
2. Complete remaining T2 tests and attempt T3 cross-module tests.
3. For cross-module blockers, defer with specific dependency reason.
4. Apply <=15-ID status updates.
5. Run checkpoint protocol and commit.
### Task 5: Implement Group D + T4 Cluster-Mapped Tests (Reader/Writer + Conversion)
**Files:**
- Modify/Create: `MqttReader*`, `MqttWriter*`, conversion helper files
- Modify tests: `ImplBacklog/MqttHandlerTests.Impltests.cs`, `ImplBacklog/JetStreamClusterTests3.Impltests.cs` (create if absent)
**Steps:**
1. Process Group D features through per-feature loop.
2. Port/verify reader/writer conversion tests first.
3. Attempt T4 cluster tests only where dependencies allow; otherwise defer explicitly.
4. Apply <=15-ID status updates.
5. Run checkpoint protocol and commit.
### Task 6: Final Batch 41 Verification and Closure
**Files:**
- Modify: `porting.db`
- Generate: `reports/current.md` (via report script)
**Steps:**
1. Run final source/test stub scans on all changed files.
2. Run build and full unit test suite.
3. Verify Batch 41 status:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 41 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db
```
4. Generate report:
```bash
./reports/generate-report.sh
```
5. Commit final checkpoint.