natsnet/docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md

# Batch 37 Stream Messages Implementation Plan

> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.

**Goal:** Port and verify Batch 37 (`Stream Messages`) from `server/stream.go` so all 86 mapped features and 13 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blockers.

**Architecture:** Implement Batch 37 in five feature groups (`18/16/16/17/19`) using `NatsStream` partials plus message carrier helpers (`InMsg`, `CMsg`, `JsPubMsg`, `JsOutQ`) and account restore support. Execute three test waves with strict per-feature/per-test evidence loops and mandatory anti-stub gates before any status promotion.

**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)

**Design doc:** `docs/plans/2026-02-27-batch-37-stream-messages-design.md`

---

## Batch 37 Scope

- Batch ID: `37`
- Name: `Stream Messages`
- Dependency: `36`
- Go source: `golang/nats-server/server/stream.go`
- Features: `86` (`3294-3387` mapped IDs)
- Tests: `13`

If `dotnet` is not on `PATH`, use:

```bash
DOTNET=/usr/local/share/dotnet/dotnet
```

Expected production files:

- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs`

Expected test files:

- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`

---

## MANDATORY VERIFICATION PROTOCOL

> **NON-NEGOTIABLE:** Every Batch 37 feature and test must pass this loop. No shortcut status updates.

### Dependency Preflight (before any status change)

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 37 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```

Start only when Batch 37 is ready:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch start 37 --db porting.db
```

Capture baseline:

```bash
$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```

### What Counts as a Real Port

A **feature** is real only if all are true:

1. Mapped method/type exists with non-placeholder behavior.
2. Logic uses actual state/inputs (not hardcoded defaults).
3. At least one behavioral test exercises that code path.
4. No forbidden stub patterns are present.

A **test** is real only if all are true:

1. Arrange/Act/Assert is present.
2. It calls production code in `ZB.MOM.NatsNet.Server.*`.
3. Assertions validate behavior from Act output/effects.
4. It is not a template/literal stub.

### Per-Feature Verification Loop (REQUIRED per feature ID)

1. Inspect mapping and Go context:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
```
2. Claim work:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
```
3. Add or update focused behavioral test(s) for that feature.
4. Implement minimal real logic for the mapped method/type.
5. Run **Stub Detection Check** on changed files.
6. Run **Build Gate**.
7. Run targeted **Test Gate** (specific class/filter for touched area).
8. Promote only proven feature IDs to `complete` (or `verified` only after checkpoint evidence).

### Per-Test Verification Loop (REQUIRED per test ID)

1. Inspect mapping and Go test location:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
```
2. Claim work:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
```
3. Port full behavior (no template assertions).
4. Run single test method:
```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
```
5. Confirm `Passed: 1, Failed: 0` (never `Passed: 0`).
6. Run touched class filter.
7. Run **Stub Detection Check** + **Build Gate**.
8. Promote only proven test IDs to `complete` (or `verified` only after checkpoint evidence).

### Stub Detection Check (REQUIRED after every feature/test loop)

```bash
changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$changed_files" ]; then
  echo "$changed_files" | xargs rg -n "(NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|// TODO|// PLACEHOLDER|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary<string, string>\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$)"
fi
```

Any match blocks status promotion until fixed or deferred with reason.

### Build Gate (REQUIRED)

```bash
$DOTNET build dotnet/
```

Run this:

- after each feature loop
- after each test loop
- before every `batch-update`
- at every task checkpoint

### Test Gate (REQUIRED)

Minimum targeted gates by touched domain:

```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamEngineTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamClusterTests1|FullyQualifiedName~JetStreamClusterTests2"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamFileStoreTests|FullyQualifiedName~RaftNodeTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~JwtProcessorTests|FullyQualifiedName~LeafNodeHandlerTests|FullyQualifiedName~ConcurrencyTests1"
```

Checkpoint/full gate:

```bash
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```

### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update)

- Never pass more than `15` IDs per `feature batch-update` or `test batch-update`.
- Never update IDs outside current task scope.
- Never promote to `verified` without checkpoint evidence.
- For blocked items, keep `deferred` with explicit reason.

Use:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

$DOTNET run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```

### Checkpoint Protocol Between Tasks (REQUIRED)

Before starting the next task:

1. Run **Stub Detection Check** on current diff.
2. Run **Build Gate**.
3. Run targeted class test filters for touched classes.
4. Run full unit test gate.
5. Apply status updates in `<=15` ID chunks only.
6. Commit checkpoint before next task.

---

## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)

### Forbidden Patterns

Forbidden in mapped Batch 37 feature/test work:

- `throw new NotImplementedException()`
- Empty mapped method bodies or no-op method shims
- Placeholder comments (`TODO`, `PLACEHOLDER`, `later`)
- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`)
- Template assertions that do not validate behavior:
  - `var goFile = "server/..."`
  - `"<MethodName>".ShouldContain("Should")`
  - `GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty)` as core assertion
- Constant-return shortcuts in mapped methods without behavioral justification:
  - `=> default;`
  - `return default;`
  - `return null;`
- Catch-and-ignore patterns that hide failures

### Hard Limits

- Max feature group size: `~20` IDs.
- Max IDs per `feature batch-update`: `15`.
- Max IDs per `test batch-update`: `15`.
- One active feature loop at a time.
- One active test loop at a time.
- Mandatory checkpoint between every task.
- No `verified` promotion without full checkpoint evidence.

### If You Get Stuck (REQUIRED)

1. Stop on the current ID immediately.
2. Do not write placeholder logic or fake assertions.
3. Mark item `deferred` with concrete blocker.
4. Continue with next unblocked item.

Feature deferral:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
  feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```

Test deferral:

```bash
$DOTNET run --project tools/NatsNet.PortTracker -- \
  test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```

`deferred` with explicit reason is correct. Stubbed progress is not.

---

## Feature Groups (max ~20 each)

### Group A (18): Dedupe and header extraction foundation

IDs:  
`3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3310,3311,3312`

### Group B (16): Scheduling/batch headers and direct-get ingress path

IDs:  
`3314,3315,3316,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330`

### Group C (16): Message carriers, publish out queue, and internal loop hooks

IDs:  
`3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3345,3346,3347`

### Group D (17): Consumer registry and interest-state operations

IDs:  
`3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3364,3366,3367,3368`

### Group E (19): Pre-ack, snapshot/restore, monitor/replication checks

IDs:  
`3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387`

## Test Waves

### Wave T1 (5): direct-get and rollup/mirror semantics

IDs:  
`828,954,987,1642,1643`

### Wave T2 (4): snapshot/restore and raft recovery paths

IDs:  
`383,2617,2672,2695`

### Wave T3 (4): gateway/jwt/leaf/perf regression coverage

IDs:  
`635,1892,1961,2426`

---

### Task 1: Preflight, Baseline, and Batch Start

**Files:**
- Read: `docs/plans/2026-02-27-batch-37-stream-messages-design.md`
- Read: `golang/nats-server/server/stream.go`

**Steps:**
1. Run dependency preflight and verify Batch 36 completion.
2. Start Batch 37.
3. Capture baseline build/test evidence.
4. Do not change any status without baseline logs.

### Task 2: Implement Feature Group A (18 IDs)

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs`

**Steps:**
1. Move Group A IDs to `stub` in `15 + 3` chunks.
2. Run Per-Feature Verification Loop for each ID.
3. Run required stub/build/test gates per ID.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.

### Task 3: Implement Feature Group B (16 IDs)

**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs`

**Steps:**
1. Move Group B IDs to `stub` in `15 + 1` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.

### Task 4: Implement Feature Group C (16 IDs)

**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs`
- Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs`

**Steps:**
1. Move Group C IDs to `stub` in `15 + 1` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Ensure `JsOutQ` exists with mapped methods.
4. Run mandatory gates.
5. Promote proven IDs in `<=15` chunks.
6. Run checkpoint protocol.

### Task 5: Implement Feature Group D (17 IDs)

**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs`

**Steps:**
1. Move Group D IDs to `stub` in `15 + 2` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.

### Task 6: Implement Feature Group E (19 IDs)

**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs`

**Steps:**
1. Move Group E IDs to `stub` in `15 + 4` chunks.
2. Execute Per-Feature Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in `<=15` chunks.
5. Run checkpoint protocol.

### Task 7: Port Test Wave T1 (5 IDs)

**Files:**
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs`

**Steps:**
1. Move Wave T1 IDs to `stub`.
2. Execute Per-Test Verification Loop per ID.
3. Run mandatory stub/build/class/full test gates.
4. Promote proven IDs in one `<=15` batch.
5. Run checkpoint protocol.

### Task 8: Port Test Wave T2 (4 IDs)

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`

**Steps:**
1. Move Wave T2 IDs to `stub`.
2. Execute Per-Test Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in one `<=15` batch.
5. Run checkpoint protocol.

### Task 9: Port Test Wave T3 (4 IDs)

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`

**Steps:**
1. Move Wave T3 IDs to `stub`.
2. Execute Per-Test Verification Loop per ID.
3. Run mandatory gates.
4. Promote proven IDs in one `<=15` batch.
5. Run checkpoint protocol.

### Task 10: Batch Closeout

**Files:**
- Modify: `porting.db`
- Modify: `reports/current.md` (via report script output)

**Steps:**
1. Run final stub detection on current diff.
2. Run final build gate and full test gate.
3. Run dry-run audits for features and tests:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db
```
4. Complete batch:
```bash
$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 37 --db porting.db
```
5. Regenerate summary report:
```bash
./reports/generate-report.sh
```

---

Plan complete and saved to `docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md`. Two execution options:

1. Subagent-Driven (this session) - I dispatch fresh subagent per task, review between tasks, fast iteration.
2. Parallel Session (separate) - Open new session with `executeplan`, batch execution with checkpoints.

Which approach?