natsnet/docs/plans/2026-02-27-batch-33-js-cluster-streams-implementation-plan.md

Batch 33 JS Cluster Streams Implementation Plan

For Codex: REQUIRED SUB-SKILL: Use executeplan to implement this plan task-by-task.

Goal: Port and verify Batch 33 (JS Cluster Streams) from server/jetstream_cluster.go, with evidence-based status transitions for all 58 features and 22 tests.

Architecture: Implement stream/consumer cluster logic in three feature groups (20/20/18) across JetStream, NatsStream, JetStreamCluster data helpers, and NatsServer partials. Then port mapped tests in three waves (5/9/8) with method-level execution evidence. Every transition (deferred/not_started -> stub -> complete -> verified) is gated by stub scans, build/test gates, and checkpoint review.

Tech Stack: .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (porting.db)

Design doc: docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md

---

Batch 33 Scope

• Batch ID: 33
• Name: JS Cluster Streams
• Dependency: 32
• Go source: golang/nats-server/server/jetstream_cluster.go
• Features: 58 (1578-1635)
• Tests: 22 (1118,1214,1402,2144,2504,2616,2620,2622,2624,2627,2628,2630,2631,2634,2637,2638,2652,2657,2670,2671,2698,2699)

If dotnet is unavailable on PATH, use /usr/local/share/dotnet/dotnet in all commands.

Primary production files (expected touch set):

• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs

Primary test files (expected touch set):

• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs
• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs
• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs

---

MANDATORY VERIFICATION PROTOCOL

NON-NEGOTIABLE: Every Batch 33 feature/test ID must pass this protocol before promotion.

Dependency Preflight Gate (before any status changes)

dotnet run --project tools/NatsNet.PortTracker -- batch show 32 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch show 33 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db

Only if dependency is satisfied and Batch 33 is ready:

dotnet run --project tools/NatsNet.PortTracker -- batch start 33 --db porting.db

Capture baseline:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

Per-Feature Verification Loop (REQUIRED per feature ID)

1. Inspect mapping and Go span:

dotnet run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db

2. Claim only that ID as in-progress marker:

dotnet run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db

3. Add or update at least one focused test that fails for the intended behavior.
4. Run focused test and confirm failure is for the target behavior.
5. Implement minimal production logic for that feature ID.
6. Run Stub Detection Check.
7. Run Build Gate.
8. Run Test Gate (focused).
9. If green, mark that feature complete (promotion to verified only at checkpoint).

Per-Test Verification Loop (REQUIRED per test ID)

1. Inspect mapping and Go test source:

dotnet run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db

2. Claim only that test ID as stub.
3. Port behavior (Arrange/Act/Assert; no placeholder assertions).
4. Run the single test method:

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal

5. Confirm Passed: 1, Failed: 0 (not Passed: 0).
6. Run class-level filter for the touched class.
7. Run Stub Detection Check + Build Gate.
8. If green, mark test complete (promotion to verified only at checkpoint).

Stub Detection Check (REQUIRED after each feature/test loop)

Run only on changed C# files:

changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$changed_files" ]; then
  echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass|ShouldBe\\(true\\)|=>\\s*default;|=>\\s*null;|return\\s+null;\\s*$|return\\s+0;\\s*$|return\\s+false;\\s*$)"
fi

Any match in mapped methods/tests blocks promotion.

Build Gate (REQUIRED)

Run dotnet build dotnet/:

• after each feature loop,
• after each test loop,
• before every batch-update,
• at every task checkpoint.

Test Gate (REQUIRED)

Minimum gates:

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream"
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog"

Plus per-item focused gate:

• Per feature loop: run nearest impacted backlog or unit class filters.
• Per test loop: run single-method filter, then class filter.

Status Update Protocol (HARD LIMIT: 15 IDs max per batch-update)

• Never pass more than 15 IDs in one feature/test batch-update command.
• Never mark verified without loop evidence + task checkpoint evidence.
• Never update IDs outside the active task.
• For blocked IDs, use deferred override with explicit reason.

Templates:

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

Checkpoint Protocol Between Tasks (REQUIRED)

At the end of each task before moving on:

1. Run Stub Detection Check.
2. Run Build Gate.
3. Run targeted test filters for touched classes.
4. Run full unit suite:

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

5. Promote only proven IDs (complete and/or verified) in <=15 chunks.
6. Record evidence and commit checkpoint.

---

ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)

Forbidden Patterns (Features + Tests)

• throw new NotImplementedException() in mapped Batch 33 methods.
• Empty method body for mapped features.
• Placeholder comments in mapped code/tests (TODO, PLACEHOLDER, later).
• Fake-pass assertions (Assert.True(true), Assert.Pass(), trivial self-string assertions).
• Constant-return shortcuts for mapped methods without input/state use (return false;, return 0;, return null;, return string.Empty;).
• Catch-and-ignore blocks that suppress failures to make tests pass.

Hard Limits

• Feature group size max: ~20 IDs.
• feature batch-update max IDs: 15.
• test batch-update max IDs: 15.
• One active feature loop at a time.
• One active test loop at a time.
• No verified promotion without checkpoint evidence.
• Mandatory checkpoint between every task.

If You Get Stuck (REQUIRED)

1. Stop the current ID immediately.
2. Do not keep/add stubbed placeholder behavior.
3. Mark blocked item deferred with explicit reason:

dotnet run --project tools/NatsNet.PortTracker -- \
  feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db

dotnet run --project tools/NatsNet.PortTracker -- \
  test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db

4. Continue with next unblocked ID.

Deferred with reason is acceptable. Stubbed progress is not.

---

Feature Groups (max ~20 each)

Group A (20): monitor/snapshot/recovery primitives

IDs:
1578,1579,1580,1581,1582,1583,1584,1585,1586,1587,1588,1589,1590,1591,1592,1593,1594,1595,1596,1597

Group B (20): meta entries + raft group + stream monitor/apply + leader change

IDs:
1598,1599,1600,1601,1602,1603,1604,1605,1606,1607,1608,1609,1610,1611,1612,1613,1614,1615,1616,1617

Group C (18): advisories + stream/consumer assignment lifecycle

IDs:
1618,1619,1620,1621,1622,1623,1624,1625,1626,1627,1628,1629,1630,1631,1632,1633,1634,1635

Test Waves

Wave T1 (5): cluster/jwt/monitor/concurrency anchors

IDs:
1118,1214,1402,2144,2504

Wave T2 (9): raft election/term/recovery baseline

IDs:
2616,2620,2622,2624,2627,2628,2630,2631,2634

Wave T3 (8): raft replay/catchup/chain-of-blocks

IDs:
2637,2638,2652,2657,2670,2671,2698,2699

---

Task 1: Preflight and Baseline

Files:

• Read: docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md
• Read: golang/nats-server/server/jetstream_cluster.go

Steps:

1. Run dependency preflight gate and confirm Batch 33 can start.
2. Start Batch 33 only when ready.
3. Capture baseline build + unit test run.
4. Do not change any feature/test status before baseline evidence is captured.

Task 2: Implement Feature Group A (20 IDs)

Files:

• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs

Steps:

1. Set Group A to stub in two commands (15 + 5 IDs).
2. Run Per-Feature Verification Loop for each ID in Group A.
3. Apply Stub Detection + Build/Test gates after each ID.
4. At checkpoint, promote proven IDs to complete then verified in <=15 chunks.

Task 3: Implement Feature Group B (20 IDs)

Files:

• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs

Steps:

1. Set Group B to stub in two commands (15 + 5 IDs).
2. Execute Per-Feature Verification Loop for each Group B ID.
3. Run all required gates.
4. Execute checkpoint and chunked promotions (<=15).

Task 4: Implement Feature Group C (18 IDs)

Files:

• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs
• Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs

Steps:

1. Set Group C to stub in two commands (15 + 3 IDs).
2. Execute Per-Feature Verification Loop for each Group C ID.
3. Run all required gates.
4. Execute checkpoint and chunked promotions (<=15).

Task 5: Port Test Wave T1 (5 IDs)

Files:

• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs
• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs

Steps:

1. Set T1 IDs to stub (single update command).
2. Execute Per-Test Verification Loop for each ID.
3. Run class-level filters for each touched class.
4. Execute checkpoint and promote passing IDs in <=15 chunks.

Task 6: Port Test Wave T2 (9 IDs)

Files:

• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs

Steps:

1. Set T2 IDs to stub (single update command).
2. Execute Per-Test Verification Loop for each raft ID.
3. Run RaftNodeTests class filter and confirm pass counts.
4. Execute checkpoint and promote proven IDs.

Task 7: Port Test Wave T3 (8 IDs)

Files:

• Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs

Steps:

1. Set T3 IDs to stub (single update command).
2. Execute Per-Test Verification Loop for each ID.
3. Run class filter + full ImplBacklog filter.
4. Execute checkpoint and promote proven IDs.

Task 8: Final Batch Verification and Closure

Files:

• Modify: porting.db
• Update generated report: reports/current.md

Steps:

1. Run final gates:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db

2. Resolve remaining stub items: promote with evidence or defer with reason.
3. Complete batch:

dotnet run --project tools/NatsNet.PortTracker -- batch complete 33 --db porting.db

4. Refresh report:

./reports/generate-report.sh