natsnet/docs/plans/2026-02-27-batch-33-js-cluster-streams-implementation-plan.md

# Batch 33 JS Cluster Streams Implementation Plan

> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.

**Goal:** Port and verify Batch 33 (`JS Cluster Streams`) from `server/jetstream_cluster.go`, with evidence-based status transitions for all 58 features and 22 tests.

**Architecture:** Implement stream/consumer cluster logic in three feature groups (`20/20/18`) across `JetStream`, `NatsStream`, `JetStreamCluster` data helpers, and `NatsServer` partials. Then port mapped tests in three waves (`5/9/8`) with method-level execution evidence. Every transition (`deferred/not_started -> stub -> complete -> verified`) is gated by stub scans, build/test gates, and checkpoint review.

**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)

**Design doc:** `docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md`

---

## Batch 33 Scope

- Batch ID: `33`
- Name: `JS Cluster Streams`
- Dependency: `32`
- Go source: `golang/nats-server/server/jetstream_cluster.go`
- Features: `58` (`1578-1635`)
- Tests: `22` (`1118,1214,1402,2144,2504,2616,2620,2622,2624,2627,2628,2630,2631,2634,2637,2638,2652,2657,2670,2671,2698,2699`)

If `dotnet` is unavailable on `PATH`, use `/usr/local/share/dotnet/dotnet` in all commands.

Primary production files (expected touch set):

- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs`

Primary test files (expected touch set):

- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`

---

## MANDATORY VERIFICATION PROTOCOL

> **NON-NEGOTIABLE:** Every Batch 33 feature/test ID must pass this protocol before promotion.

### Dependency Preflight Gate (before any status changes)

```bash
dotnet run --project tools/NatsNet.PortTracker -- batch show 32 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch show 33 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
```

Only if dependency is satisfied and Batch 33 is ready:

```bash
dotnet run --project tools/NatsNet.PortTracker -- batch start 33 --db porting.db
```

Capture baseline:

```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```

### Per-Feature Verification Loop (REQUIRED per feature ID)

1. Inspect mapping and Go span:
```bash
dotnet run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
```
2. Claim only that ID as in-progress marker:
```bash
dotnet run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
```
3. Add or update at least one focused test that fails for the intended behavior.
4. Run focused test and confirm failure is for the target behavior.
5. Implement minimal production logic for that feature ID.
6. Run `Stub Detection Check`.
7. Run `Build Gate`.
8. Run `Test Gate` (focused).
9. If green, mark that feature `complete` (promotion to `verified` only at checkpoint).

### Per-Test Verification Loop (REQUIRED per test ID)

1. Inspect mapping and Go test source:
```bash
dotnet run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
```
2. Claim only that test ID as `stub`.
3. Port behavior (Arrange/Act/Assert; no placeholder assertions).
4. Run the single test method:
```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
```
5. Confirm `Passed: 1, Failed: 0` (not `Passed: 0`).
6. Run class-level filter for the touched class.
7. Run `Stub Detection Check` + `Build Gate`.
8. If green, mark test `complete` (promotion to `verified` only at checkpoint).

### Stub Detection Check (REQUIRED after each feature/test loop)

Run only on changed C# files:

```bash
changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$changed_files" ]; then
  echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass|ShouldBe\\(true\\)|=>\\s*default;|=>\\s*null;|return\\s+null;\\s*$|return\\s+0;\\s*$|return\\s+false;\\s*$)"
fi
```

Any match in mapped methods/tests blocks promotion.

### Build Gate (REQUIRED)

Run `dotnet build dotnet/`:

- after each feature loop,
- after each test loop,
- before every `batch-update`,
- at every task checkpoint.

### Test Gate (REQUIRED)

Minimum gates:

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream"
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog"
```

Plus per-item focused gate:

- Per feature loop: run nearest impacted backlog or unit class filters.
- Per test loop: run single-method filter, then class filter.

### Status Update Protocol (HARD LIMIT: 15 IDs max per `batch-update`)

- Never pass more than `15` IDs in one `feature/test batch-update` command.
- Never mark `verified` without loop evidence + task checkpoint evidence.
- Never update IDs outside the active task.
- For blocked IDs, use deferred override with explicit reason.

Templates:

```bash
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute
```

### Checkpoint Protocol Between Tasks (REQUIRED)

At the end of each task before moving on:

1. Run `Stub Detection Check`.
2. Run `Build Gate`.
3. Run targeted test filters for touched classes.
4. Run full unit suite:
```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```
5. Promote only proven IDs (`complete` and/or `verified`) in `<=15` chunks.
6. Record evidence and commit checkpoint.

---

## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)

### Forbidden Patterns (Features + Tests)

- `throw new NotImplementedException()` in mapped Batch 33 methods.
- Empty method body for mapped features.
- Placeholder comments in mapped code/tests (`TODO`, `PLACEHOLDER`, `later`).
- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`, trivial self-string assertions).
- Constant-return shortcuts for mapped methods without input/state use (`return false;`, `return 0;`, `return null;`, `return string.Empty;`).
- Catch-and-ignore blocks that suppress failures to make tests pass.

### Hard Limits

- Feature group size max: `~20` IDs.
- `feature batch-update` max IDs: `15`.
- `test batch-update` max IDs: `15`.
- One active feature loop at a time.
- One active test loop at a time.
- No `verified` promotion without checkpoint evidence.
- Mandatory checkpoint between every task.

### If You Get Stuck (REQUIRED)

1. Stop the current ID immediately.
2. Do not keep/add stubbed placeholder behavior.
3. Mark blocked item `deferred` with explicit reason:

```bash
dotnet run --project tools/NatsNet.PortTracker -- \
  feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db

dotnet run --project tools/NatsNet.PortTracker -- \
  test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db
```

4. Continue with next unblocked ID.

Deferred with reason is acceptable. Stubbed progress is not.

---

## Feature Groups (max ~20 each)

### Group A (20): monitor/snapshot/recovery primitives

IDs:  
`1578,1579,1580,1581,1582,1583,1584,1585,1586,1587,1588,1589,1590,1591,1592,1593,1594,1595,1596,1597`

### Group B (20): meta entries + raft group + stream monitor/apply + leader change

IDs:  
`1598,1599,1600,1601,1602,1603,1604,1605,1606,1607,1608,1609,1610,1611,1612,1613,1614,1615,1616,1617`

### Group C (18): advisories + stream/consumer assignment lifecycle

IDs:  
`1618,1619,1620,1621,1622,1623,1624,1625,1626,1627,1628,1629,1630,1631,1632,1633,1634,1635`

## Test Waves

### Wave T1 (5): cluster/jwt/monitor/concurrency anchors

IDs:  
`1118,1214,1402,2144,2504`

### Wave T2 (9): raft election/term/recovery baseline

IDs:  
`2616,2620,2622,2624,2627,2628,2630,2631,2634`

### Wave T3 (8): raft replay/catchup/chain-of-blocks

IDs:  
`2637,2638,2652,2657,2670,2671,2698,2699`

---

### Task 1: Preflight and Baseline

**Files:**
- Read: `docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md`
- Read: `golang/nats-server/server/jetstream_cluster.go`

**Steps:**
1. Run dependency preflight gate and confirm Batch 33 can start.
2. Start Batch 33 only when ready.
3. Capture baseline build + unit test run.
4. Do not change any feature/test status before baseline evidence is captured.

### Task 2: Implement Feature Group A (20 IDs)

**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs`

**Steps:**
1. Set Group A to `stub` in two commands (`15 + 5` IDs).
2. Run Per-Feature Verification Loop for each ID in Group A.
3. Apply Stub Detection + Build/Test gates after each ID.
4. At checkpoint, promote proven IDs to `complete` then `verified` in `<=15` chunks.

### Task 3: Implement Feature Group B (20 IDs)

**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs`

**Steps:**
1. Set Group B to `stub` in two commands (`15 + 5` IDs).
2. Execute Per-Feature Verification Loop for each Group B ID.
3. Run all required gates.
4. Execute checkpoint and chunked promotions (`<=15`).

### Task 4: Implement Feature Group C (18 IDs)

**Files:**
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs`
- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs`

**Steps:**
1. Set Group C to `stub` in two commands (`15 + 3` IDs).
2. Execute Per-Feature Verification Loop for each Group C ID.
3. Run all required gates.
4. Execute checkpoint and chunked promotions (`<=15`).

### Task 5: Port Test Wave T1 (5 IDs)

**Files:**
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs`
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs`

**Steps:**
1. Set T1 IDs to `stub` (single update command).
2. Execute Per-Test Verification Loop for each ID.
3. Run class-level filters for each touched class.
4. Execute checkpoint and promote passing IDs in `<=15` chunks.

### Task 6: Port Test Wave T2 (9 IDs)

**Files:**
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`

**Steps:**
1. Set T2 IDs to `stub` (single update command).
2. Execute Per-Test Verification Loop for each raft ID.
3. Run `RaftNodeTests` class filter and confirm pass counts.
4. Execute checkpoint and promote proven IDs.

### Task 7: Port Test Wave T3 (8 IDs)

**Files:**
- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs`

**Steps:**
1. Set T3 IDs to `stub` (single update command).
2. Execute Per-Test Verification Loop for each ID.
3. Run class filter + full ImplBacklog filter.
4. Execute checkpoint and promote proven IDs.

### Task 8: Final Batch Verification and Closure

**Files:**
- Modify: `porting.db`
- Update generated report: `reports/current.md`

**Steps:**
1. Run final gates:
```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db
```
2. Resolve remaining `stub` items: promote with evidence or defer with reason.
3. Complete batch:
```bash
dotnet run --project tools/NatsNet.PortTracker -- batch complete 33 --db porting.db
```
4. Refresh report:
```bash
./reports/generate-report.sh
```