From f8dce79ac0d07401673a206e8cb341f0c52d296c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 27 Feb 2026 17:01:31 -0500 Subject: [PATCH] Add batch plans for batches 31-36 (rounds 16-18) Generated design docs and implementation plans via Codex for: - Batch 31: Raft Part 2 - Batch 32: JS Cluster Meta - Batch 33: JS Cluster Streams - Batch 34: JS Cluster Consumers - Batch 35: JS Cluster Remaining - Batch 36: Stream Lifecycle All plans include mandatory verification protocol and anti-stub guardrails. Updated batches.md with file paths and planned status. --- docs/batches.md | 12 +- .../2026-02-27-batch-31-raft-part-2-design.md | 127 +++++ ...atch-31-raft-part-2-implementation-plan.md | 444 +++++++++++++++++ ...6-02-27-batch-32-js-cluster-meta-design.md | 137 ++++++ ...-32-js-cluster-meta-implementation-plan.md | 391 +++++++++++++++ ...2-27-batch-33-js-cluster-streams-design.md | 125 +++++ ...-js-cluster-streams-implementation-plan.md | 363 ++++++++++++++ ...27-batch-34-js-cluster-consumers-design.md | 149 ++++++ ...s-cluster-consumers-implementation-plan.md | 456 ++++++++++++++++++ ...27-batch-35-js-cluster-remaining-design.md | 135 ++++++ ...s-cluster-remaining-implementation-plan.md | 410 ++++++++++++++++ ...-02-27-batch-36-stream-lifecycle-design.md | 156 ++++++ ...36-stream-lifecycle-implementation-plan.md | 410 ++++++++++++++++ reports/current.md | 2 +- reports/report_c05d936.md | 37 ++ 15 files changed, 3347 insertions(+), 7 deletions(-) create mode 100644 docs/plans/2026-02-27-batch-31-raft-part-2-design.md create mode 100644 docs/plans/2026-02-27-batch-31-raft-part-2-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-32-js-cluster-meta-design.md create mode 100644 docs/plans/2026-02-27-batch-32-js-cluster-meta-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md create mode 100644 docs/plans/2026-02-27-batch-33-js-cluster-streams-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-34-js-cluster-consumers-design.md create mode 100644 docs/plans/2026-02-27-batch-34-js-cluster-consumers-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-35-js-cluster-remaining-design.md create mode 100644 docs/plans/2026-02-27-batch-35-js-cluster-remaining-implementation-plan.md create mode 100644 docs/plans/2026-02-27-batch-36-stream-lifecycle-design.md create mode 100644 docs/plans/2026-02-27-batch-36-stream-lifecycle-implementation-plan.md create mode 100644 reports/report_c05d936.md diff --git a/docs/batches.md b/docs/batches.md index 5ef37a5..c77fca5 100644 --- a/docs/batches.md +++ b/docs/batches.md @@ -45,12 +45,12 @@ | 28 | JetStream API | [design](plans/2026-02-27-batch-28-jetstream-api-design.md) | [plan](plans/2026-02-27-batch-28-jetstream-api-implementation-plan.md) | planned | | 29 | JetStream Batching | [design](plans/2026-02-27-batch-29-jetstream-batching-design.md) | [plan](plans/2026-02-27-batch-29-jetstream-batching-implementation-plan.md) | planned | | 30 | Raft Part 1 | [design](plans/2026-02-27-batch-30-raft-part-1-design.md) | [plan](plans/2026-02-27-batch-30-raft-part-1-implementation-plan.md) | planned | -| 31 | Raft Part 2 | | | not_planned | -| 32 | JS Cluster Meta | | | not_planned | -| 33 | JS Cluster Streams | | | not_planned | -| 34 | JS Cluster Consumers | | | not_planned | -| 35 | JS Cluster Remaining | | | not_planned | -| 36 | Stream Lifecycle | | | not_planned | +| 31 | Raft Part 2 | [design](plans/2026-02-27-batch-31-raft-part-2-design.md) | [plan](plans/2026-02-27-batch-31-raft-part-2-implementation-plan.md) | planned | +| 32 | JS Cluster Meta | [design](plans/2026-02-27-batch-32-js-cluster-meta-design.md) | [plan](plans/2026-02-27-batch-32-js-cluster-meta-implementation-plan.md) | planned | +| 33 | JS Cluster Streams | [design](plans/2026-02-27-batch-33-js-cluster-streams-design.md) | [plan](plans/2026-02-27-batch-33-js-cluster-streams-implementation-plan.md) | planned | +| 34 | JS Cluster Consumers | [design](plans/2026-02-27-batch-34-js-cluster-consumers-design.md) | [plan](plans/2026-02-27-batch-34-js-cluster-consumers-implementation-plan.md) | planned | +| 35 | JS Cluster Remaining | [design](plans/2026-02-27-batch-35-js-cluster-remaining-design.md) | [plan](plans/2026-02-27-batch-35-js-cluster-remaining-implementation-plan.md) | planned | +| 36 | Stream Lifecycle | [design](plans/2026-02-27-batch-36-stream-lifecycle-design.md) | [plan](plans/2026-02-27-batch-36-stream-lifecycle-implementation-plan.md) | planned | | 37 | Stream Messages | | | not_planned | | 38 | Consumer Lifecycle | | | not_planned | | 39 | Consumer Dispatch | | | not_planned | diff --git a/docs/plans/2026-02-27-batch-31-raft-part-2-design.md b/docs/plans/2026-02-27-batch-31-raft-part-2-design.md new file mode 100644 index 0000000..ceeb6d4 --- /dev/null +++ b/docs/plans/2026-02-27-batch-31-raft-part-2-design.md @@ -0,0 +1,127 @@ +# Batch 31 Raft Part 2 Design + +**Date:** 2026-02-27 +**Batch:** 31 (`Raft Part 2`) +**Scope:** 53 features + 19 unit tests +**Dependencies:** batch `30` (`Raft Part 1`) +**Go source:** `golang/nats-server/server/raft.go` + +## Problem + +Batch 31 covers the second Raft tranche in `raft.go` (roughly lines `3239-5038`), focused on catchup/snapshot transfer, append-entry processing, WAL consistency, quorum tracking, vote request/response handling, and leadership state transitions. The mapped test set (19 tests) is concentrated on candidate/leader transitions, quorum correctness, membership-change edge cases, and snapshot/catchup behavior. + +The design goal is to produce an execution-ready plan that enforces evidence-based status changes and prevents placeholder drift across both production features and tests. + +## Context Findings + +### Required command results + +- `batch show 31 --db porting.db` + - Status: `pending` + - Features: `53` (currently `deferred`) + - Tests: `19` (currently `deferred`) + - Depends on: `30` + - Go file: `server/raft.go` +- `batch list --db porting.db` + - Batch 31 is directly gated by Batch 30 and itself gates Batch 32 (`JS Cluster Meta`). +- `report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + - Deferred backlog remains large; verification discipline is required. + +### Feature and source mapping findings + +- Batch 31 feature IDs map in order to `raft.go` methods from: + - `sendSnapshotToFollower` through `updateLeader` (`2733-2750`) + - `processAppendEntry` through `setWriteErrLocked` (`2751-2777`) + - `isClosed` through `switchToLeader` (`2778-2796`) +- Existing .NET Raft surface is in: + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Current comments in `RaftTypes.cs` still describe algorithm methods as stubbed; Batch 31 must replace those gaps with concrete behavior and tests. + +### Test mapping findings + +- All 19 mapped tests are from `server/raft_test.go` and map to `RaftNodeTests` methods. +- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` does not currently exist, so Batch 31 planning should include creating it. +- The mapped tests are behavior-heavy; they cannot be verified using placeholder assertions. + +## Approaches + +### Approach A: Monolithic implementation of all 53 features and 19 tests in one pass + +- Pros: single sweep. +- Cons: high regression risk, weak traceability, hard to isolate failures. + +### Approach B (Recommended): Three feature groups (<=20 each) plus two test waves + +- Features are implemented in ordered method clusters, each with strict gates before status updates. +- Tests are ported in two behavioral waves (state/quorum first, then snapshot/membership edge cases). +- Pros: bounded scope, better failure isolation, cleaner status evidence. +- Cons: more checkpoint overhead. + +### Approach C: Test-first across all 19 tests, then fill feature gaps + +- Pros: quickly exposes missing behavior. +- Cons: expensive thrash because many tests depend on broad feature slices. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Architecture and file strategy + +- Keep Raft runtime behavior in `JetStream/RaftTypes.cs`, with optional split into partials if file size hurts reviewability: + - `RaftTypes.Catchup.cs` + - `RaftTypes.AppendProcessing.cs` + - `RaftTypes.Elections.cs` +- Keep test implementation in dedicated mapped backlog file: + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` +- Reuse existing support types (`IpQueue`, `Channel`, lock + `Interlocked`) and avoid introducing new infra unless required for deterministic testability. + +### 2. Feature slicing (max ~20 per group) + +- **Feature Group A (18): catchup/snapshot/commit foundations** + `2733,2734,2735,2736,2737,2738,2739,2740,2741,2742,2743,2744,2745,2746,2747,2748,2749,2750` +- **Feature Group B (18): append-entry processing and peer/WAL state** + `2751,2752,2753,2754,2755,2756,2758,2759,2760,2761,2765,2766,2767,2768,2769,2772,2776,2777` +- **Feature Group C (17): vote/RPC/state transitions** + `2778,2779,2780,2783,2784,2785,2786,2787,2788,2789,2790,2791,2792,2793,2794,2795,2796` + +### 3. Test slicing + +- **Test Wave T1 (10): state/quorum/election behavior** + `2626,2629,2635,2636,2663,2664,2667,2687,2690,2692` +- **Test Wave T2 (9): snapshot/catchup/membership-vote edge cases** + `2650,2651,2693,2694,2702,2704,2705,2712,2714` + +### 4. Verification model + +- Enforce per-feature and per-test loops (red/green + stub scan + build/test gates). +- Enforce status-update chunking (`<=15` IDs per `feature/test batch-update`). +- Enforce checkpoint protocol after every group/wave before proceeding. + +### 5. Stuck-item policy + +- A blocked item is not left as pseudo-implemented. +- If blocked, set `deferred` immediately with explicit reason via `--override`, then continue with next unblocked ID. + +## Risks and Mitigations + +- **Risk:** Batch 30 dependency incomplete blocks execution. + **Mitigation:** preflight dependency gate is mandatory; no Batch 31 status updates until Batch 30 is complete/ready. +- **Risk:** Large method `processAppendEntry` causes hidden regressions. + **Mitigation:** isolate with focused tests per behavior branch plus class-level gates. +- **Risk:** fake progress via placeholder methods/tests. + **Mitigation:** mandatory anti-stub scans and hard promotion gates. + +## Success Criteria + +- All 53 features are either `verified` with evidence or `deferred` with explicit blocker reason. +- All 19 tests are either `verified` with execution evidence or `deferred` with explicit blocker reason. +- No placeholder/stub patterns in touched production or test code. +- Batch-completion readiness is auditable through build/test outputs and chunked status updates. + +## Non-Goals + +- Executing implementation in this design doc. +- Implementing Batch 32+ scope. +- Building new distributed integration infrastructure beyond deterministic unit-level needs. diff --git a/docs/plans/2026-02-27-batch-31-raft-part-2-implementation-plan.md b/docs/plans/2026-02-27-batch-31-raft-part-2-implementation-plan.md new file mode 100644 index 0000000..228a1d9 --- /dev/null +++ b/docs/plans/2026-02-27-batch-31-raft-part-2-implementation-plan.md @@ -0,0 +1,444 @@ +# Batch 31 Raft Part 2 Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 31 (`Raft Part 2`) Raft behavior from `server/raft.go` with strict evidence gates for both features and tests. + +**Architecture:** Implement 53 mapped Raft methods in three dependency-ordered feature groups (max 20 IDs each), then port 19 mapped tests in two waves. Use per-item red/green loops, mandatory stub scans, and chunked tracker updates (`<=15` IDs per batch command) so status moves only with executable proof. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-31-raft-part-2-design.md` + +--- + +## Batch 31 Scope + +- Batch ID: `31` +- Name: `Raft Part 2` +- Dependency: `30` +- Go source: `golang/nats-server/server/raft.go` +- Features: `53` +- Tests: `19` + +Primary implementation files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Optional split (if needed for reviewability): + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Catchup.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.AppendProcessing.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Elections.cs` + +Primary test files: + +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` +- Modify (if shared helpers are needed): + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` + - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ImpltestsBacklogAssertions.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every feature ID and test ID in this batch must pass this protocol. + +### Preflight Dependency Gate (before any status change) + +1. Confirm dependency and readiness: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 30 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 31 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +2. Start only when Batch 31 is ready: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 31 --db porting.db +``` + +3. Capture baseline build/tests: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapping and Go span: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db +sed -n ',p' golang/nats-server/server/raft.go +``` + +2. Mark feature as in-progress (`stub` status in tracker only): + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` + +3. Add/adjust a focused failing test for this behavior. +4. Run focused test and confirm it fails for the expected reason. +5. Implement minimum production change to make it pass. +6. Run **Build Gate**. +7. Run **Test Gate** for touched tests. +8. Run **Stub Detection Check**. +9. If all pass, mark feature `complete` (not `verified` yet). + +### Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapping and Go test span: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db +sed -n ',p' golang/nats-server/server/raft_test.go +``` + +2. Mark test as in-progress (`stub` status in tracker only): + +```bash +dotnet run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` + +3. Implement a real Arrange/Act/Assert test that calls production code. +4. Run single-test filter: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~RaftNodeTests." \ + --verbosity normal +``` + +5. Confirm discovery + pass (`Passed: 1, Failed: 0`). +6. Run class-level filter for cumulative validation. +7. Run **Stub Detection Check**. +8. Mark test `complete` (promote to `verified` only at checkpoint). + +### Stub Detection Check (REQUIRED after each loop and each task) + +```bash +git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests \ + | rg "\.cs$" \ + | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass|=>\s*default;|=>\s*null;|return\s+null;\s*$|return\s+0;\s*$|return\s+false;\s*$)" +``` + +Any hit in mapped methods/tests blocks promotion. + +### Build Gate (REQUIRED) + +Run `dotnet build dotnet/`: + +- after each feature/test loop, +- before any batch status update, +- at each task checkpoint. + +### Test Gate (REQUIRED) + +Minimum required gates for touched Raft scope: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftTypes" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftNodeTests" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog" +``` + +Checkpoint and final gate: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Status Update Protocol (HARD LIMIT: <=15 IDs per batch-update) + +- Allowed flow: `deferred/not_started -> stub -> complete -> verified` +- Never include more than 15 IDs in one `feature/test batch-update` command. +- Never mark `verified` without passing Build Gate + Test Gate + Stub Detection Check. +- Apply updates only for IDs in the active task. + +Templates: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +At end of every task before starting the next: + +1. Run Stub Detection Check. +2. Run Build Gate. +3. Run focused Test Gate for touched classes. +4. Run full unit test suite. +5. Apply status updates for current task only (`<=15` IDs per command). +6. Commit checkpoint. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns (Features + Tests) + +Any of these in mapped/touched files is disallowed: + +- `throw new NotImplementedException()` +- Empty or no-op method bodies for mapped behaviors +- Placeholder comments (`// TODO`, `// PLACEHOLDER`, `// later`) +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`) +- Tests that only assert non-null/default without behavior validation +- Constant-return placeholders for complex logic (`return null;`, `return 0;`, `return false;`, `return string.Empty;`) +- Blanket `catch` blocks that swallow failures to force pass + +### Hard Limits + +- Max feature IDs per implementation group: `20` +- Max IDs per `feature/test batch-update`: `15` +- One active feature loop at a time +- One active test loop at a time +- No `verified` transition without full gate evidence +- Mandatory checkpoint between tasks + +### If You Get Stuck (REQUIRED) + +1. Stop work on that ID immediately. +2. Do not leave placeholder code or fake-pass assertions. +3. Mark item `deferred` with explicit reason: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status deferred --override "blocked: " --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- test update --status deferred --override "blocked: " --db porting.db +``` + +4. Add brief note in tracker/comment where needed. +5. Continue with next unblocked ID in the same task. + +Deferred-with-reason is valid. Stubs are not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (18) - Catchup/snapshot/commit foundations + +`2733,2734,2735,2736,2737,2738,2739,2740,2741,2742,2743,2744,2745,2746,2747,2748,2749,2750` + +### Group B (18) - Append processing, WAL, peer state + +`2751,2752,2753,2754,2755,2756,2758,2759,2760,2761,2765,2766,2767,2768,2769,2772,2776,2777` + +### Group C (17) - Vote/RPC/state transitions + +`2778,2779,2780,2783,2784,2785,2786,2787,2788,2789,2790,2791,2792,2793,2794,2795,2796` + +## Test Waves + +### T1 (10) - Election/quorum/state correctness + +`2626,2629,2635,2636,2663,2664,2667,2687,2690,2692` + +### T2 (9) - Catchup/snapshot/membership and vote retention + +`2650,2651,2693,2694,2702,2704,2705,2712,2714` + +--- + +### Task 1: Preflight and Baseline + +**Files:** + +- Read: `docs/plans/2026-02-27-batch-31-raft-part-2-design.md` +- Read: `golang/nats-server/server/raft.go` +- Read: `golang/nats-server/server/raft_test.go` + +**Step 1: Run Preflight Dependency Gate** + +Run all preflight commands in protocol. + +**Step 2: Start batch and capture baseline** + +Run `batch start 31`, `dotnet build`, and full unit tests. + +**Step 3: Checkpoint protocol** + +Complete checkpoint before feature work. + +--- + +### Task 2: Implement Feature Group A (18) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Catchup.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` + +**Step 1: Mark Group A IDs as `stub` in <=15-ID chunks** + +Chunk 1: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2733,2734,2735,2736,2737,2738,2739,2740,2741,2742,2743,2744,2745,2746,2747" --set-status stub --db porting.db --execute +``` + +Chunk 2: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2748,2749,2750" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for each Group A ID** + +**Step 3: Run task checkpoint gates** + +**Step 4: Promote eligible Group A IDs to `complete` then `verified` in <=15-ID chunks** + +--- + +### Task 3: Implement Feature Group B (18) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.AppendProcessing.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` + +**Step 1: Mark Group B IDs as `stub` in <=15-ID chunks** + +Chunk 1: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2751,2752,2753,2754,2755,2756,2758,2759,2760,2761,2765,2766,2767,2768,2769" --set-status stub --db porting.db --execute +``` + +Chunk 2: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2772,2776,2777" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for each Group B ID** + +**Step 3: Run task checkpoint gates** + +**Step 4: Promote eligible Group B IDs to `complete` then `verified` in <=15-ID chunks** + +--- + +### Task 4: Implement Feature Group C (17) + +**Files:** + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.Elections.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/RaftTypes.cs` +- Test: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs` + +**Step 1: Mark Group C IDs as `stub` in <=15-ID chunks** + +Chunk 1: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2778,2779,2780,2783,2784,2785,2786,2787,2788,2789,2790,2791,2792,2793,2794" --set-status stub --db porting.db --execute +``` + +Chunk 2: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "2795,2796" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Feature Verification Loop for each Group C ID** + +**Step 3: Run task checkpoint gates** + +**Step 4: Promote eligible Group C IDs to `complete` then `verified` in <=15-ID chunks** + +--- + +### Task 5: Port Test Wave T1 (10) + +**Files:** + +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` +- Modify (if helpers required): `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ImpltestsBacklogAssertions.cs` + +**Step 1: Mark T1 IDs as `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "2626,2629,2635,2636,2663,2664,2667,2687,2690,2692" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Test Verification Loop for each T1 ID** + +**Step 3: Run class-level + checkpoint gates** + +**Step 4: Promote eligible T1 IDs to `complete` then `verified`** + +--- + +### Task 6: Port Test Wave T2 (9) + +**Files:** + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Step 1: Mark T2 IDs as `stub`** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "2650,2651,2693,2694,2702,2704,2705,2712,2714" --set-status stub --db porting.db --execute +``` + +**Step 2: Execute Per-Test Verification Loop for each T2 ID** + +**Step 3: Run class-level + checkpoint gates** + +**Step 4: Promote eligible T2 IDs to `complete` then `verified`** + +--- + +### Task 7: Final Batch 31 Verification and Completion + +**Files:** + +- Modify: `porting.db` +- Modify: `reports/current.md` (via report script) + +**Step 1: Run final mandatory gates** + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +``` + +**Step 2: Resolve any remaining `stub` statuses** + +- Convert to `verified` with evidence, or +- Convert to `deferred` with explicit blocker reason. + +**Step 3: Complete the batch** + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 31 --db porting.db +``` + +**Step 4: Refresh report** + +```bash +./reports/generate-report.sh +``` + diff --git a/docs/plans/2026-02-27-batch-32-js-cluster-meta-design.md b/docs/plans/2026-02-27-batch-32-js-cluster-meta-design.md new file mode 100644 index 0000000..32bd26b --- /dev/null +++ b/docs/plans/2026-02-27-batch-32-js-cluster-meta-design.md @@ -0,0 +1,137 @@ +# Batch 32 JS Cluster Meta Design + +**Date:** 2026-02-27 +**Batch:** 32 (`JS Cluster Meta`) +**Scope:** 58 features + 36 unit tests +**Dependencies:** batches `27` (`JetStream Core`), `31` (`Raft Part 2`) +**Go source:** `golang/nats-server/server/jetstream_cluster.go` + +## Problem + +Batch 32 ports JetStream cluster metadata and control-plane behaviors from `jetstream_cluster.go`, including unsupported assignment handling, cluster leadership/currentness queries, meta-group setup hooks, assignment/health checks, inflight proposal tracking, meta-recovery flags, and orphan detection. + +The mapped tests are distributed across cluster/super-cluster/leaf/mqtt/concurrency/raft Go suites and require non-placeholder behavioral coverage. The design goal is to provide an implementation strategy that prevents fake progress and enforces evidence-based status transitions for both features and tests. + +## Context Findings + +### Required command outputs + +- `batch show 32 --db porting.db` + - Status: `pending` + - Features: `58` (`deferred`) + - Tests: `36` (`deferred`) + - Depends on: `27,31` + - Go file: `server/jetstream_cluster.go` +- `batch list --db porting.db` + - Batch 32 is in dependency chain `... -> 31 -> 32 -> 33/35`. +- `report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + +### Dependency state + +- `batch show 27` status: `pending` +- `batch show 31` status: `pending` +- `batch ready` does **not** include Batch 32 currently. + +Design implication: Batch 32 execution must start with an explicit dependency gate; no status changes before dependencies are ready. + +### Current .NET codebase state + +- Cluster data types already exist in `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs`. +- Most Batch 32 mapped method names are not yet implemented in .NET source. +- Backlog tests exist in `ImplBacklog`, but only one cluster class file currently exists: + - present: `JetStreamClusterTests2.Impltests.cs` + - missing and expected to be created during implementation: + - `JetStreamClusterTests1.Impltests.cs` + - `JetStreamClusterTests3.Impltests.cs` + - `JetStreamClusterTests4.Impltests.cs` + - `JetStreamClusterLongTests.Impltests.cs` + - `JetStreamSuperClusterTests.Impltests.cs` + +## Approaches + +### Approach A: Monolithic one-pass implementation (all 58 features + 36 tests together) + +- Pros: single pass, less planning overhead. +- Cons: high risk of regressions and undetected stubs; weak traceability for status updates. + +### Approach B (Recommended): Three feature groups (<=20 each) + three test waves + +- Implement feature groups in source-order clusters, each with strict build/test/stub gates before status updates. +- Port tests in behavior-based waves aligned to cluster-domain breadth. +- Pros: bounded risk, clear checkpoints, strong auditability. +- Cons: more checkpoint ceremony. + +### Approach C: Test-first all 36 tests before feature porting + +- Pros: surfaces missing behavior early. +- Cons: creates heavy thrash because many tests depend on broad feature slices not yet implemented. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Code architecture and file strategy + +- Keep cluster model + cluster helper logic in JetStream cluster files: + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Keep `JetStream`/`JsAccount` cluster-meta behaviors in JetStream engine file(s): + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + - optional split if needed for reviewability: `JetStream/JetStreamClusterMeta.cs` +- Keep server-facing API entry points in a server partial: + - create/modify `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterMeta.cs` +- Keep account-facing query helpers in account class: + - `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` + +### 2. Feature slicing (max ~20 each) + +- **Feature Group A (20): unsupported assignment + server cluster state basics** + IDs: `1520-1539` +- **Feature Group B (20): health/leadership queries + clustering enable + assignment checks** + IDs: `1540-1559` +- **Feature Group C (18): leader/consumer assignment internals + inflight tracking + recovery/orphans** + IDs: `1560-1577` + +This keeps each feature task under the required `~20` cap. + +### 3. Test slicing + +- **Wave T1 (13): core cluster behavior (cluster tests 1/2)** + IDs: `772,774,775,791,809,810,811,817,853,914,993,1014,1028` +- **Wave T2 (12): advanced cluster behavior (cluster tests 3/4 + long)** + IDs: `1060,1088,1098,1106,1109,1122,1128,1136,1194,1211,1212,1217` +- **Wave T3 (11): cross-domain coverage (leaf/super/mqtt/concurrency/raft hooks)** + IDs: `1406,1453,1454,1457,1465,1528,2225,2390,2459,2489,2689` + +### 4. Verification model (features + tests) + +- Mandatory per-feature red/green loop with build + focused tests + stub scan before promotion. +- Mandatory per-test loop (single-test pass evidence + class/wave pass evidence). +- Status updates only in chunks of `<=15 IDs` per `feature/test batch-update` command. +- Task checkpoints between groups/waves with full suite verification. + +### 5. Deferred policy + +If an item is blocked by missing infrastructure or unresolved dependency behavior, explicitly set it to `deferred` with `--override "blocked: "`. Do not leave stubs or fake-pass tests. + +## Risks and Mitigations + +- **Risk:** dependency batches 27/31 still pending. + **Mitigation:** enforce preflight dependency gate before any Batch 32 status transitions. +- **Risk:** broad cluster tests encourage placeholder assertions. + **Mitigation:** anti-stub guardrails + required assertion quality and per-test evidence. +- **Risk:** cross-file method ownership ambiguity (`NatsServer`, `JetStream`, `Account`). + **Mitigation:** fixed ownership map in plan and grouped implementation order by type. + +## Success Criteria + +- All 58 features are either `verified` with evidence or `deferred` with explicit blocker reason. +- All 36 tests are either `verified` with run evidence or `deferred` with explicit blocker reason. +- No forbidden stub patterns in touched production/test files. +- Batch 32 completion is auditable through build/test outputs and chunked status updates. + +## Non-Goals + +- Executing Batch 32 implementation in this document. +- Porting Batch 33/34/35 behaviors. +- Building new distributed integration infrastructure beyond what is needed for deterministic unit/backlog verification. diff --git a/docs/plans/2026-02-27-batch-32-js-cluster-meta-implementation-plan.md b/docs/plans/2026-02-27-batch-32-js-cluster-meta-implementation-plan.md new file mode 100644 index 0000000..0a35725 --- /dev/null +++ b/docs/plans/2026-02-27-batch-32-js-cluster-meta-implementation-plan.md @@ -0,0 +1,391 @@ +# Batch 32 JS Cluster Meta Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 32 (`JS Cluster Meta`) from `server/jetstream_cluster.go` with strict evidence gates for both features and tests. + +**Architecture:** Implement mapped cluster-meta methods in three source-ordered feature groups (`<=20` IDs each) across `NatsServer`, `JetStream`, `JetStreamCluster`, `RecoveryUpdates`, and `Account`. Then port mapped backlog tests in three waves. Every promotion (`stub -> complete -> verified`) is gated by build/test evidence, stub scans, and chunked tracker updates (`<=15` IDs per update). + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-32-js-cluster-meta-design.md` + +--- + +## Batch 32 Scope + +- Batch ID: `32` +- Name: `JS Cluster Meta` +- Dependencies: `27`, `31` +- Go source: `golang/nats-server/server/jetstream_cluster.go` +- Features: `58` +- Tests: `36` + +> If `dotnet` is not available on `PATH`, run commands with `/usr/local/share/dotnet/dotnet` instead. + +Primary source files: + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterMeta.cs` + +Primary test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every feature ID and test ID in Batch 32 must pass this protocol. + +### Dependency Preflight Gate (before any status change) + +1. Validate dependencies and readiness: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 31 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 32 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +2. Only when Batch 32 is ready, start it: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 32 --db porting.db +``` + +3. Capture clean baseline: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapped feature and Go span: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` + +2. Mark only the active feature as `stub` (tracker in-progress marker): + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` + +3. Add/adjust a focused test that fails for the intended behavior. +4. Run focused test; confirm expected failure. +5. Implement minimal production change for that feature. +6. Run **Stub Detection Check** (below). +7. Run **Build Gate** (below). +8. Run **Test Gate** (below). +9. If all pass, mark feature `complete`. Promote to `verified` only at task checkpoint. + +### Stub Detection Check (REQUIRED after each feature loop and each test loop) + +Run against changed C# files only: + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass|=>\s*default;|=>\s*null;|return\s+null;\s*$|return\s+0;\s*$|return\s+false;\s*$)" +fi +``` + +Any match in mapped methods/tests blocks promotion. + +### Build Gate (REQUIRED) + +Run `dotnet build dotnet/`: + +- after each feature loop, +- after each test loop, +- before every status batch-update, +- at every checkpoint between tasks. + +### Test Gate (REQUIRED) + +Minimum gates: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog" +``` + +For every mapped backlog test method: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." \ + --verbosity normal +``` + +Verify discovery and pass (`Passed: 1, Failed: 0`) before promotion. + +### Status Update Protocol (HARD LIMIT: <=15 IDs per batch-update) + +- Allowed transitions: `deferred/not_started -> stub -> complete -> verified` +- Never include more than `15` IDs in one update command. +- Never mark `verified` without passing stub scan + build gate + test gate. +- Never update IDs outside the currently active task. + +Templates: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +At the end of every task before starting the next: + +1. Run Stub Detection Check. +2. Run Build Gate. +3. Run targeted Test Gate for touched classes. +4. Run full unit tests: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +5. Apply status promotions (`complete`/`verified`) for that task only (`<=15` IDs per command). +6. Commit checkpoint. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns (Features + Tests) + +Any of these in mapped/touched files is disallowed: + +- `throw new NotImplementedException()` +- Empty method bodies for mapped feature methods +- Placeholder comments (`// TODO`, `// PLACEHOLDER`, `// later`) +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`) +- Trivial string/self assertions unrelated to production behavior +- Unconditional constant returns in mapped methods without state/input usage (`return false;`, `return 0;`, `return null;`, `return string.Empty;`) +- Catch-and-ignore blocks that suppress real failures just to pass tests + +### Hard Limits + +- Max features per implementation task group: `20` +- Max IDs per `feature batch-update`: `15` +- Max IDs per `test batch-update`: `15` +- One active feature loop at a time +- One active test loop at a time +- No `verified` status without checkpoint evidence +- Mandatory checkpoint between tasks + +### If You Get Stuck (REQUIRED) + +1. Stop work on that specific ID. +2. Do **not** keep or add placeholder/stub code. +3. Mark the item `deferred` immediately with explicit reason: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status deferred --override "blocked: " --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- test update --status deferred --override "blocked: " --db porting.db +``` + +4. Add a short note in tracker/test comment with concrete blocker details. +5. Continue with the next unblocked ID. + +Deferred-with-reason is valid progress. Stubs are not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (20) - Unsupported assignments + server/meta state entry points + +IDs: +`1520,1521,1522,1523,1524,1525,1526,1527,1528,1529,1530,1531,1532,1533,1534,1535,1536,1537,1538,1539` + +### Group B (20) - Health/leadership checks + clustering enable + assignment checks + +IDs: +`1540,1541,1542,1543,1544,1545,1546,1547,1548,1549,1550,1551,1552,1553,1554,1555,1556,1557,1558,1559` + +### Group C (18) - Stream/consumer leadership internals + inflight tracking + recovery/orphans + +IDs: +`1560,1561,1562,1563,1564,1565,1566,1567,1568,1569,1570,1571,1572,1573,1574,1575,1576,1577` + +## Test Waves + +### Wave T1 (13) - Core cluster tests (files 1/2) + +IDs: +`772,774,775,791,809,810,811,817,853,914,993,1014,1028` + +### Wave T2 (12) - Advanced cluster tests (files 3/4 + long) + +IDs: +`1060,1088,1098,1106,1109,1122,1128,1136,1194,1211,1212,1217` + +### Wave T3 (11) - Cross-domain integration-facing backlog tests + +IDs: +`1406,1453,1454,1457,1465,1528,2225,2390,2459,2489,2689` + +--- + +### Task 1: Preflight and Baseline + +**Files:** + +- Read: `docs/plans/2026-02-27-batch-32-js-cluster-meta-design.md` +- Read: `golang/nats-server/server/jetstream_cluster.go` + +**Step 1:** Run dependency preflight gate (27/31/32 + ready list). +**Step 2:** Start batch 32 only if ready. +**Step 3:** Capture baseline build and full test run. +**Step 4:** Apply checkpoint protocol before feature work. + +--- + +### Task 2: Implement Feature Group A (20) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterMeta.cs` + +**Step 1:** Set Group A to `stub` in two chunks (`15 + 5`). +**Step 2:** Execute per-feature verification loop for each ID. +**Step 3:** Run checkpoint protocol. +**Step 4:** Promote passing IDs to `complete` and `verified` in `<=15` chunks. + +--- + +### Task 3: Implement Feature Group B (20) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Create/Modify: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterMeta.cs` + +**Step 1:** Set Group B to `stub` in two chunks (`15 + 5`). +**Step 2:** Execute per-feature verification loop for each ID. +**Step 3:** Run checkpoint protocol. +**Step 4:** Promote passing IDs to `complete` and `verified` in `<=15` chunks. + +--- + +### Task 4: Implement Feature Group C (18) + +**Files:** + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + +**Step 1:** Set Group C to `stub` in two chunks (`15 + 3`). +**Step 2:** Execute per-feature verification loop for each ID. +**Step 3:** Run checkpoint protocol. +**Step 4:** Promote passing IDs to `complete` and `verified` in `<=15` chunks. + +--- + +### Task 5: Port Test Wave T1 (13) + +**Files:** + +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` + +**Step 1:** Set T1 test IDs to `stub` (single chunk of 13). +**Step 2:** Execute per-test verification loop for each ID (single-test proof required). +**Step 3:** Run class-level filters for `JetStreamClusterTests1` and `JetStreamClusterTests2`. +**Step 4:** Run checkpoint protocol and promote passing IDs in `<=15` chunk. + +--- + +### Task 6: Port Test Wave T2 (12) + +**Files:** + +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs` + +**Step 1:** Set T2 test IDs to `stub` (single chunk of 12). +**Step 2:** Execute per-test verification loop for each ID. +**Step 3:** Run class-level filters for touched classes. +**Step 4:** Run checkpoint protocol and promote passing IDs. + +--- + +### Task 7: Port Test Wave T3 (11) + +**Files:** + +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- Create/Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Step 1:** Set T3 test IDs to `stub` (single chunk of 11). +**Step 2:** Execute per-test verification loop for each ID. +**Step 3:** Run class-level filters for all touched backlog classes. +**Step 4:** Run checkpoint protocol and promote passing IDs. + +--- + +### Task 8: Final Batch 32 Verification and Closure + +**Files:** + +- Modify: `porting.db` +- Update: `reports/current.md` (generated) + +**Step 1:** Final gates: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +``` + +**Step 2:** Resolve any remaining `stub` statuses: + +- Promote to `verified` with evidence, or +- Set to `deferred` with explicit blocker reason. + +**Step 3:** Complete batch: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 32 --db porting.db +``` + +**Step 4:** Refresh report: + +```bash +./reports/generate-report.sh +``` + diff --git a/docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md b/docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md new file mode 100644 index 0000000..74031a5 --- /dev/null +++ b/docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md @@ -0,0 +1,125 @@ +# Batch 33 JS Cluster Streams Design + +**Date:** 2026-02-27 +**Batch:** 33 (`JS Cluster Streams`) +**Scope:** 58 features + 22 unit tests +**Dependency:** batch `32` (`JS Cluster Meta`) +**Go source:** `golang/nats-server/server/jetstream_cluster.go` + +## Problem + +Batch 33 ports JetStream cluster stream/consumer assignment execution paths from `server/jetstream_cluster.go`, covering cluster monitoring loops, metadata snapshots, raft-group creation, stream-entry application, leader-change advisories, and stream/consumer create-update-delete flows. + +The mapped tests are spread across JetStream cluster, monitor, JWT, concurrency, and raft suites. The design objective is to define a strict, auditable implementation path that avoids placeholder code and only advances tracker statuses with build/test evidence. + +## Context Findings + +### Required command outputs + +- `batch show 33 --db porting.db` + - Status: `pending` + - Features: `58` (all `deferred`) + - Tests: `22` (all `deferred`) + - Depends on: `32` + - Go file: `server/jetstream_cluster.go` +- `batch list --db porting.db` + - Batch chain includes `32 -> 33 -> 34` for JS cluster progression. +- `report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + +Note: in this environment, `dotnet` is not on `PATH`; use `/usr/local/share/dotnet/dotnet` when needed. + +### Current .NET state relevant to Batch 33 + +- Cluster data structures exist in `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs`. +- Core types exist in: + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` + - `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.*.cs` (partial server files) +- Backlog test coverage is mostly placeholder-level today; `JetStreamClusterTests2.Impltests.cs` is present, while several mapped classes (for example `JetStreamClusterTests3`, `JetStreamClusterLongTests`, `RaftNodeTests`) still need concrete batch coverage. + +## Clarified Constraints + +- Planning only in this session: no implementation execution. +- Mandatory guardrails from Batch 0 must be carried forward and adapted to features + tests. +- Feature work must be chunked into groups of at most ~20 features. +- Status updates must use `batch-update` chunks of max 15 IDs. + +## Approaches + +### Approach A: Monolithic pass (all 58 features + 22 tests) + +- Pros: fewer task boundaries. +- Cons: weak traceability and high risk of hidden stubs/regressions. + +### Approach B (Recommended): Three feature groups + three test waves with hard checkpoints + +- Pros: bounded scope per task, stronger verification evidence, easier rollback/debug. +- Cons: more command overhead and checkpoint ceremony. + +### Approach C: Test-heavy-first before major feature porting + +- Pros: early behavior signal. +- Cons: high churn because many mapped tests depend on stream/consumer cluster plumbing not yet ported. + +**Decision:** Approach B. + +## Proposed Design + +### 1. File ownership model + +- `JetStream` cluster stream orchestration methods in `JetStreamTypes.cs` or a new focused partial file (`JetStream.ClusterStreams.cs`). +- `NatsStream` raft/cluster helpers in `NatsStream.cs` or `NatsStream.Cluster.cs`. +- `RaftGroup`, `StreamAssignment`, `ConsumerAssignment`, and cluster helpers in `JetStreamClusterTypes.cs` (or focused partials if split improves reviewability). +- Server-facing operations and advisories in a new/updated server partial (`NatsServer.JetStreamClusterStreams.cs`). + +### 2. Feature slicing (max ~20 each) + +- **Feature Group A (20 IDs):** cluster monitor + snapshot/recovery primitives + IDs: `1578-1597` +- **Feature Group B (20 IDs):** meta-entry application + raft-group/stream monitoring + leader-change core + IDs: `1598-1617` +- **Feature Group C (18 IDs):** advisory + stream assignment/process lifecycle + consumer assignment/process lifecycle + IDs: `1618-1635` + +### 3. Test slicing + +- **Test Wave T1 (5 IDs):** cluster long-path + JWT/monitor/concurrency anchors + IDs: `1118,1214,1402,2144,2504` +- **Test Wave T2 (9 IDs):** raft elections and term behavior (early raft set) + IDs: `2616,2620,2622,2624,2627,2628,2630,2631,2634` +- **Test Wave T3 (8 IDs):** raft replay/catchup/chain-of-blocks paths + IDs: `2637,2638,2652,2657,2670,2671,2698,2699` + +### 4. Verification architecture + +- Per-feature loop: `feature show` -> focused failing test -> minimal implementation -> stub scan -> build gate -> targeted test gate -> status transition. +- Per-test loop: `test show` -> Go behavioral port -> single-test run evidence -> class-level run -> status transition. +- Checkpoint after every feature group and test wave, including full unit suite run. + +### 5. Deferred handling model + +If blocked by missing dependency behavior/infrastructure, immediately mark item `deferred` with explicit reason via `--override`; do not leave stubs in source or tests. + +## Risks and Mitigations + +- **Dependency risk:** Batch 32 is prerequisite. + **Mitigation:** block all Batch 33 status transitions until dependency preflight confirms readiness. +- **Stub-risk in backlog tests:** existing placeholder-style tests can produce false progress. + **Mitigation:** required stub scan + assertion-quality checks + single-test execution evidence. +- **Ownership ambiguity risk:** methods span `JetStream`, `NatsStream`, `JetStreamCluster`, `NatsServer`. + **Mitigation:** explicit file ownership map and grouped tasking by domain. + +## Success Criteria + +- All 58 features are either `verified` with evidence or `deferred` with explicit blocker reason. +- All 22 tests are either `verified` with evidence or `deferred` with explicit blocker reason. +- No forbidden stub patterns in touched files. +- Batch progress is auditable from command outputs and chunked status updates. + +## Non-Goals + +- Executing the implementation in this document. +- Extending scope into Batch 34/35. +- Building full distributed integration harness beyond mapped unit/backlog verification needs. diff --git a/docs/plans/2026-02-27-batch-33-js-cluster-streams-implementation-plan.md b/docs/plans/2026-02-27-batch-33-js-cluster-streams-implementation-plan.md new file mode 100644 index 0000000..880d249 --- /dev/null +++ b/docs/plans/2026-02-27-batch-33-js-cluster-streams-implementation-plan.md @@ -0,0 +1,363 @@ +# Batch 33 JS Cluster Streams Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 33 (`JS Cluster Streams`) from `server/jetstream_cluster.go`, with evidence-based status transitions for all 58 features and 22 tests. + +**Architecture:** Implement stream/consumer cluster logic in three feature groups (`20/20/18`) across `JetStream`, `NatsStream`, `JetStreamCluster` data helpers, and `NatsServer` partials. Then port mapped tests in three waves (`5/9/8`) with method-level execution evidence. Every transition (`deferred/not_started -> stub -> complete -> verified`) is gated by stub scans, build/test gates, and checkpoint review. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md` + +--- + +## Batch 33 Scope + +- Batch ID: `33` +- Name: `JS Cluster Streams` +- Dependency: `32` +- Go source: `golang/nats-server/server/jetstream_cluster.go` +- Features: `58` (`1578-1635`) +- Tests: `22` (`1118,1214,1402,2144,2504,2616,2620,2622,2624,2627,2628,2630,2631,2634,2637,2638,2652,2657,2670,2671,2698,2699`) + +If `dotnet` is unavailable on `PATH`, use `/usr/local/share/dotnet/dotnet` in all commands. + +Primary production files (expected touch set): + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs` + +Primary test files (expected touch set): + +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 33 feature/test ID must pass this protocol before promotion. + +### Dependency Preflight Gate (before any status changes) + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch show 32 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch show 33 --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Only if dependency is satisfied and Batch 33 is ready: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch start 33 --db porting.db +``` + +Capture baseline: + +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapping and Go span: +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` +2. Claim only that ID as in-progress marker: +```bash +dotnet run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` +3. Add or update at least one focused test that fails for the intended behavior. +4. Run focused test and confirm failure is for the target behavior. +5. Implement minimal production logic for that feature ID. +6. Run `Stub Detection Check`. +7. Run `Build Gate`. +8. Run `Test Gate` (focused). +9. If green, mark that feature `complete` (promotion to `verified` only at checkpoint). + +### Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapping and Go test source: +```bash +dotnet run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` +2. Claim only that test ID as `stub`. +3. Port behavior (Arrange/Act/Assert; no placeholder assertions). +4. Run the single test method: +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` +5. Confirm `Passed: 1, Failed: 0` (not `Passed: 0`). +6. Run class-level filter for the touched class. +7. Run `Stub Detection Check` + `Build Gate`. +8. If green, mark test `complete` (promotion to `verified` only at checkpoint). + +### Stub Detection Check (REQUIRED after each feature/test loop) + +Run only on changed C# files: + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass|ShouldBe\\(true\\)|=>\\s*default;|=>\\s*null;|return\\s+null;\\s*$|return\\s+0;\\s*$|return\\s+false;\\s*$)" +fi +``` + +Any match in mapped methods/tests blocks promotion. + +### Build Gate (REQUIRED) + +Run `dotnet build dotnet/`: + +- after each feature loop, +- after each test loop, +- before every `batch-update`, +- at every task checkpoint. + +### Test Gate (REQUIRED) + +Minimum gates: + +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream" +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog" +``` + +Plus per-item focused gate: + +- Per feature loop: run nearest impacted backlog or unit class filters. +- Per test loop: run single-method filter, then class filter. + +### Status Update Protocol (HARD LIMIT: 15 IDs max per `batch-update`) + +- Never pass more than `15` IDs in one `feature/test batch-update` command. +- Never mark `verified` without loop evidence + task checkpoint evidence. +- Never update IDs outside the active task. +- For blocked IDs, use deferred override with explicit reason. + +Templates: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +dotnet run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +At the end of each task before moving on: + +1. Run `Stub Detection Check`. +2. Run `Build Gate`. +3. Run targeted test filters for touched classes. +4. Run full unit suite: +```bash +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` +5. Promote only proven IDs (`complete` and/or `verified`) in `<=15` chunks. +6. Record evidence and commit checkpoint. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns (Features + Tests) + +- `throw new NotImplementedException()` in mapped Batch 33 methods. +- Empty method body for mapped features. +- Placeholder comments in mapped code/tests (`TODO`, `PLACEHOLDER`, `later`). +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`, trivial self-string assertions). +- Constant-return shortcuts for mapped methods without input/state use (`return false;`, `return 0;`, `return null;`, `return string.Empty;`). +- Catch-and-ignore blocks that suppress failures to make tests pass. + +### Hard Limits + +- Feature group size max: `~20` IDs. +- `feature batch-update` max IDs: `15`. +- `test batch-update` max IDs: `15`. +- One active feature loop at a time. +- One active test loop at a time. +- No `verified` promotion without checkpoint evidence. +- Mandatory checkpoint between every task. + +### If You Get Stuck (REQUIRED) + +1. Stop the current ID immediately. +2. Do not keep/add stubbed placeholder behavior. +3. Mark blocked item `deferred` with explicit reason: + +```bash +dotnet run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db + +dotnet run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +4. Continue with next unblocked ID. + +Deferred with reason is acceptable. Stubbed progress is not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (20): monitor/snapshot/recovery primitives + +IDs: +`1578,1579,1580,1581,1582,1583,1584,1585,1586,1587,1588,1589,1590,1591,1592,1593,1594,1595,1596,1597` + +### Group B (20): meta entries + raft group + stream monitor/apply + leader change + +IDs: +`1598,1599,1600,1601,1602,1603,1604,1605,1606,1607,1608,1609,1610,1611,1612,1613,1614,1615,1616,1617` + +### Group C (18): advisories + stream/consumer assignment lifecycle + +IDs: +`1618,1619,1620,1621,1622,1623,1624,1625,1626,1627,1628,1629,1630,1631,1632,1633,1634,1635` + +## Test Waves + +### Wave T1 (5): cluster/jwt/monitor/concurrency anchors + +IDs: +`1118,1214,1402,2144,2504` + +### Wave T2 (9): raft election/term/recovery baseline + +IDs: +`2616,2620,2622,2624,2627,2628,2630,2631,2634` + +### Wave T3 (8): raft replay/catchup/chain-of-blocks + +IDs: +`2637,2638,2652,2657,2670,2671,2698,2699` + +--- + +### Task 1: Preflight and Baseline + +**Files:** +- Read: `docs/plans/2026-02-27-batch-33-js-cluster-streams-design.md` +- Read: `golang/nats-server/server/jetstream_cluster.go` + +**Steps:** +1. Run dependency preflight gate and confirm Batch 33 can start. +2. Start Batch 33 only when ready. +3. Capture baseline build + unit test run. +4. Do not change any feature/test status before baseline evidence is captured. + +### Task 2: Implement Feature Group A (20 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` + +**Steps:** +1. Set Group A to `stub` in two commands (`15 + 5` IDs). +2. Run Per-Feature Verification Loop for each ID in Group A. +3. Apply Stub Detection + Build/Test gates after each ID. +4. At checkpoint, promote proven IDs to `complete` then `verified` in `<=15` chunks. + +### Task 3: Implement Feature Group B (20 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs` + +**Steps:** +1. Set Group B to `stub` in two commands (`15 + 5` IDs). +2. Execute Per-Feature Verification Loop for each Group B ID. +3. Run all required gates. +4. Execute checkpoint and chunked promotions (`<=15`). + +### Task 4: Implement Feature Group C (18 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterStreams.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` + +**Steps:** +1. Set Group C to `stub` in two commands (`15 + 3` IDs). +2. Execute Per-Feature Verification Loop for each Group C ID. +3. Run all required gates. +4. Execute checkpoint and chunked promotions (`<=15`). + +### Task 5: Port Test Wave T1 (5 IDs) + +**Files:** +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterLongTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs` + +**Steps:** +1. Set T1 IDs to `stub` (single update command). +2. Execute Per-Test Verification Loop for each ID. +3. Run class-level filters for each touched class. +4. Execute checkpoint and promote passing IDs in `<=15` chunks. + +### Task 6: Port Test Wave T2 (9 IDs) + +**Files:** +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Steps:** +1. Set T2 IDs to `stub` (single update command). +2. Execute Per-Test Verification Loop for each raft ID. +3. Run `RaftNodeTests` class filter and confirm pass counts. +4. Execute checkpoint and promote proven IDs. + +### Task 7: Port Test Wave T3 (8 IDs) + +**Files:** +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Steps:** +1. Set T3 IDs to `stub` (single update command). +2. Execute Per-Test Verification Loop for each ID. +3. Run class filter + full ImplBacklog filter. +4. Execute checkpoint and promote proven IDs. + +### Task 8: Final Batch Verification and Closure + +**Files:** +- Modify: `porting.db` +- Update generated report: `reports/current.md` + +**Steps:** +1. Run final gates: +```bash +dotnet build dotnet/ +dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +dotnet run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +``` +2. Resolve remaining `stub` items: promote with evidence or defer with reason. +3. Complete batch: +```bash +dotnet run --project tools/NatsNet.PortTracker -- batch complete 33 --db porting.db +``` +4. Refresh report: +```bash +./reports/generate-report.sh +``` + diff --git a/docs/plans/2026-02-27-batch-34-js-cluster-consumers-design.md b/docs/plans/2026-02-27-batch-34-js-cluster-consumers-design.md new file mode 100644 index 0000000..70d8018 --- /dev/null +++ b/docs/plans/2026-02-27-batch-34-js-cluster-consumers-design.md @@ -0,0 +1,149 @@ +# Batch 34 JS Cluster Consumers Design + +**Date:** 2026-02-27 +**Batch:** 34 (`JS Cluster Consumers`) +**Scope:** 58 features + 160 unit tests +**Dependency:** batch `33` (`JS Cluster Streams`) +**Go source:** `golang/nats-server/server/jetstream_cluster.go` + +## Problem + +Batch 34 ports JetStream cluster consumer operations from `server/jetstream_cluster.go` (lines ~5935-8744), including consumer assignment/inflight reconciliation, replicated ack processing, leader-change handling, peer-group placement logic, clustered stream request handling, and stream/consumer mutation encoding/decoding. + +The mapped test set is broad (160 tests across 29 test classes), so the design must enforce strict evidence gates and avoid fake progress through placeholder implementations. + +## Context Findings + +### Required command outputs + +- `batch show 34 --db porting.db` + - Status: `pending` + - Features: `58` (all `deferred`) + - Tests: `160` (all `deferred`) + - Depends on: `33` + - Go file: `server/jetstream_cluster.go` +- `batch list --db porting.db` + - Batch chain includes `33 -> 34 -> 38` for JS cluster consumer progression. +- `report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + +Environment note: `dotnet` was not on `PATH` in this shell; commands need `/usr/local/share/dotnet/dotnet` fallback. + +### Mapped feature ownership (from `porting.db`) + +- `JetStreamCluster`: 19 +- `JetStreamEngine`: 13 +- `NatsServer`: 13 +- `NatsConsumer`: 7 +- `SelectPeerError`: 4 +- `JsAccount`: 1 +- `Account`: 1 + +### Mapped test distribution (top classes) + +- `ServerOptionsTests` (28), `JwtProcessorTests` (20), `WebSocketHandlerTests` (14), `LeafNodeHandlerTests` (11), `JetStreamEngineTests` (11), `JetStreamClusterTests1` (10), plus 23 additional classes. + +## Clarified Constraints + +- Planning only in this session; no implementation execution. +- Batch 0 guardrail rigor is mandatory and must be adapted for **features + tests**. +- Feature work must be sliced into groups with max ~20 feature IDs. +- Status updates must use `feature/test batch-update` chunks of max 15 IDs. +- If blocked, mark `deferred` with explicit reason; do not write stubs. + +## Approaches + +### Approach A: Single large implementation pass + +- Pros: low planning overhead. +- Cons: poor auditability, high regression/stub risk, hard to isolate failures. + +### Approach B (Recommended): Feature-first 3 groups, then 5 test waves, each with hard checkpoint gates + +- Pros: bounded scope, auditable status transitions, faster root-cause isolation. +- Cons: more CLI/test command overhead. + +### Approach C: Test-first across all 160 before feature completion + +- Pros: immediate behavior pressure. +- Cons: high churn because many tests depend on not-yet-ported consumer cluster paths. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Architecture and File Ownership + +Production code is split by behavior boundary instead of one monolithic file: + +- `JetStream` consumer orchestration: + - expected: `JetStream/JetStream.ClusterConsumers.cs` (create) or `JetStreamTypes.cs` (modify) +- `NatsConsumer` cluster hooks: + - expected: `JetStream/NatsConsumer.Cluster.cs` (create) or `NatsConsumer.cs` (modify) +- `JetStreamCluster` placement + encoding/decoding: + - expected: `JetStream/JetStreamCluster.Consumers.cs` (create) or `JetStreamClusterTypes.cs` (modify) +- `NatsServer` clustered request/advisory endpoints: + - expected: `NatsServer.JetStreamClusterConsumers.cs` (create) as partial server extension +- `Account` limits selection helper: + - expected: `Accounts/Account.JetStream.cs` (create) or `Accounts/Account.cs` (modify) + +### 2. Feature Slicing (max ~20 IDs each) + +- **Group A (20 IDs):** `1636-1655` + Consumer assignment/inflight lookup, consumer raft-node helpers, monitor/apply entries, ack decode, leader advisory primitives. +- **Group B (20 IDs):** `1656-1675` + Assignment result processors, updates subscription lifecycle, leader-change flow, peer remap/selection foundation, tier/limits checks, base clustered stream request helpers. +- **Group C (18 IDs):** `1676-1693` + Clustered stream update/delete/purge/restore/list, consumer/message delete requests, and assignment/purge/message encode-decode helpers. + +### 3. Test Slicing + +- **Wave T1 (37 IDs):** JetStream cluster/consumer behavior core (`JetStreamClusterTests1/2/3/4`, `JetStreamEngineTests`, `NatsConsumerTests`) +- **Wave T2 (39 IDs):** config/reload/options surface (`ServerOptionsTests`, `ConfigCheckTests`, `ConfigReloaderTests`, `NatsServerTests`) +- **Wave T3 (33 IDs):** JWT/auth/cert/account validations (`JwtProcessorTests`, `JetStreamJwtTests`, `AuthCalloutTests`, `AuthHandlerTests`, `CertificateStoreWindowsTests`, `AccountTests`) +- **Wave T4 (32 IDs):** transport + route + leaf/websocket (`WebSocketHandlerTests`, `LeafNodeHandlerTests`, `LeafNodeProxyTests`, `RouteHandlerTests`, `GatewayHandlerTests`) +- **Wave T5 (19 IDs):** remaining integration-oriented regressions (`MqttHandlerTests`, `JetStreamLeafNodeTests`, `JetStreamSuperClusterTests`, `MessageTracerTests`, `MonitoringHandlerTests`, `EventsHandlerTests`, `JetStreamFileStoreTests`) + +### 4. Verification Model + +- Per-feature loop and per-test loop are mandatory. +- Every loop requires: + - stub detection scan + - build gate + - targeted test gate +- Checkpoint required between all tasks before any `verified` promotion. +- Status transitions are evidence-driven only: + - `deferred/not_started -> stub -> complete -> verified` + +### 5. Failure and Deferral Strategy + +If blocked by missing infra/dependency behavior: + +1. Stop the current item. +2. Do not introduce placeholder logic or fake-pass tests. +3. Mark item `deferred` with explicit reason via `--override`. +4. Continue with next unblocked ID. + +## Risks and Mitigations + +- **Dependency readiness risk (Batch 33):** + Mitigation: hard preflight gate before starting Batch 34. +- **Wide test blast radius (160 tests / 29 classes):** + Mitigation: wave-based execution and strict checkpoints. +- **Stub regression risk in ported methods/tests:** + Mitigation: non-negotiable anti-stub scans and hard limits. +- **Ownership ambiguity across partial classes:** + Mitigation: explicit file ownership map and method-to-class grouping. + +## Success Criteria + +- All 58 features are `verified` with evidence or `deferred` with explicit blocker reason. +- All 160 tests are `verified` with evidence or `deferred` with explicit blocker reason. +- No forbidden stub patterns remain in touched production or test files. +- Status updates are auditable and chunked (`<=15` IDs per `batch-update` call). + +## Non-Goals + +- Executing implementation in this planning session. +- Expanding scope beyond Batch 34. +- Building new infrastructure outside existing batch-mapped feature/test needs. diff --git a/docs/plans/2026-02-27-batch-34-js-cluster-consumers-implementation-plan.md b/docs/plans/2026-02-27-batch-34-js-cluster-consumers-implementation-plan.md new file mode 100644 index 0000000..3ff827e --- /dev/null +++ b/docs/plans/2026-02-27-batch-34-js-cluster-consumers-implementation-plan.md @@ -0,0 +1,456 @@ +# Batch 34 JS Cluster Consumers Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 34 (`JS Cluster Consumers`) from `server/jetstream_cluster.go`, with auditable status transitions for all 58 features and 160 tests. + +**Architecture:** Implement consumer-cluster behavior in three bounded feature groups (`20/20/18`) across `JetStream`, `JetStreamCluster`, `NatsConsumer`, `NatsServer`, and `Account`. Then port mapped tests in five waves (`37/39/33/32/19`) with strict per-item verification evidence, anti-stub enforcement, and checkpoint gates between tasks. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-34-js-cluster-consumers-design.md` + +--- + +## Batch 34 Scope + +- Batch ID: `34` +- Name: `JS Cluster Consumers` +- Dependency: `33` +- Go source: `golang/nats-server/server/jetstream_cluster.go` +- Features: `58` (`1636-1693`) +- Tests: `160` (IDs listed in wave sections below) + +If `dotnet` is not on `PATH`, use: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Primary production files (expected touch set): + +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStream.ClusterConsumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamCluster.Consumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Cluster.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterConsumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.JetStream.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.cs` +- Modify (as needed): `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` + +Primary test files (expected touch set): + +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ServerOptionsTests.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigCheckTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthCalloutTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/CertificateStoreWindowsTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeProxyTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MessageTracerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 34 feature/test ID must pass this protocol before promotion. + +### Dependency Preflight Gate (before any status changes) + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 33 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 34 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Only when dependency is satisfied and batch is ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 34 --db porting.db +``` + +Baseline evidence: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapping and Go span: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` +2. Claim only that feature ID: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` +3. Add/update at least one focused test case that fails for the intended behavior. +4. Run focused test and confirm failure is for the target behavior. +5. Implement minimal production logic for that feature ID. +6. Run **Stub Detection Check** (below). +7. Run **Build Gate**. +8. Run **Test Gate** (focused/class filters). +9. If green, mark feature `complete` (promote to `verified` only at checkpoint). + +### Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapping and Go source: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` +2. Claim only that test ID: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` +3. Port full Arrange/Act/Assert behavior (no placeholders). +4. Run single test method: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` +5. Confirm summary contains `Passed: 1, Failed: 0` (not `Passed: 0`). +6. Run class-level filter for the touched class. +7. Run **Stub Detection Check** + **Build Gate**. +8. If green, mark test `complete` (promote to `verified` only at checkpoint). + +### Stub Detection Check (REQUIRED after every feature/test loop) + +Run against changed C# files: + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\.True\(true\)|Assert\.Pass|ShouldBe\(true\)|=>\s*default;|=>\s*null;|return\s+null;\s*$|return\s+0;\s*$|return\s+false;\s*$)" +fi +``` + +Any match in mapped methods/tests blocks promotion. + +### Build Gate (REQUIRED) + +Run: + +```bash +$DOTNET build dotnet/ +``` + +Required timing: + +- after each feature loop +- after each test loop +- before every `batch-update` +- at every task checkpoint + +### Test Gate (REQUIRED) + +Minimum gates: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog" +``` + +Plus per-item focused gates: + +- feature loop: nearest impacted class filters +- test loop: method filter, then class filter + +### Status Update Protocol (HARD LIMIT: 15 IDs max per `batch-update`) + +- Never pass more than `15` IDs in one `feature batch-update` or `test batch-update`. +- Never promote `verified` without loop evidence + checkpoint evidence. +- Never update IDs outside the active task. +- For blocked IDs, keep `deferred` and provide explicit `--override` reason. + +Templates: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +Before starting the next task: + +1. Run **Stub Detection Check**. +2. Run **Build Gate**. +3. Run targeted class filters for all touched files. +4. Run full unit suite: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` +5. Promote proven IDs only, in `<=15` chunks. +6. Commit checkpoint before moving on. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +These rules apply to both production features and tests. + +### Forbidden Patterns + +- `throw new NotImplementedException()` in any mapped Batch 34 method/test. +- Empty method bodies for mapped features. +- Placeholder markers in mapped code (`TODO`, `PLACEHOLDER`, `later`). +- Fake pass assertions (`Assert.True(true)`, `Assert.Pass()`, one-line tautologies). +- Constant-return shortcuts that ignore inputs/state (`return false;`, `return 0;`, `return null;`, `return string.Empty;`). +- Catch-and-ignore patterns that suppress failures to force green tests. + +### Hard Limits + +- Feature task groups: max `~20` IDs. +- `feature batch-update`: max `15` IDs per call. +- `test batch-update`: max `15` IDs per call. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between all tasks. +- No `verified` promotions without checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +1. Stop on the current ID immediately. +2. Do not add placeholder logic or fake assertions. +3. Mark blocked item `deferred` with explicit reason: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +4. Continue with the next unblocked ID. + +Deferred-with-reason is correct. Stubbed/fake-pass progress is not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (20): consumer assignment + consumer raft hooks + ack/leader foundations + +IDs: +`1636,1637,1638,1639,1640,1641,1642,1643,1644,1645,1646,1647,1648,1649,1650,1651,1652,1653,1654,1655` + +### Group B (20): assignment-result processing + peer selection + clustered request prep + +IDs: +`1656,1657,1658,1659,1660,1661,1662,1663,1664,1665,1666,1667,1668,1669,1670,1671,1672,1673,1674,1675` + +### Group C (18): clustered update/delete/list + encode/decode helpers + +IDs: +`1676,1677,1678,1679,1680,1681,1682,1683,1684,1685,1686,1687,1688,1689,1690,1691,1692,1693` + +## Test Waves + +### Wave T1 (37): JetStream cluster consumer core + +IDs: +`757,761,778,802,813,818,834,881,883,900,911,929,963,964,986,1003,1059,1108,1151,1193,1208,1231,1233,1237,1290,1342,1553,1576,1582,1587,1599,1627,1633,1662,1663,1674,1747` + +### Wave T2 (39): options/config/reload/server validations + +IDs: +`271,272,273,2515,2534,2535,2536,2543,2548,2554,2558,2559,2562,2564,2568,2570,2573,2574,2575,2577,2578,2579,2580,2581,2582,2584,2591,2592,2594,2595,2596,2723,2740,2763,2765,2777,2785,2889,2892` + +### Wave T3 (33): JWT/auth/cert/account validations + +IDs: +`103,126,136,139,147,151,155,156,157,1386,1388,1391,1397,1834,1854,1856,1859,1862,1863,1865,1866,1868,1869,1870,1871,1876,1877,1878,1879,1880,1882,1885,1886` + +### Wave T4 (32): websocket/leaf/route/gateway behavior + +IDs: +`655,1904,1905,1910,1913,1914,1920,1931,1937,1938,1941,1969,1972,1979,2824,2829,2845,2849,3074,3081,3090,3091,3092,3094,3096,3100,3101,3107,3108,3123,3129,3133` + +### Wave T5 (19): remaining JS-related integration regressions + +IDs: +`342,599,1405,1408,1422,1430,1439,1441,1444,1448,2146,2172,2173,2176,2180,2186,2264,2332,2352` + +--- + +### Task 1: Preflight and Baseline + +**Files:** +- Read: `docs/plans/2026-02-27-batch-34-js-cluster-consumers-design.md` +- Read: `golang/nats-server/server/jetstream_cluster.go` + +**Steps:** +1. Run dependency preflight commands and ensure Batch 34 is startable. +2. Start Batch 34. +3. Capture baseline build and test results. +4. Do not update statuses beyond `stub` markers before baseline evidence is captured. + +### Task 2: Implement Feature Group A (20 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStream.ClusterConsumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsConsumer.Cluster.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamCluster.Consumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterConsumers.cs` + +**Steps:** +1. Set Group A IDs to `stub` using two commands (`15 + 5` IDs). +2. Run Per-Feature Verification Loop for each Group A ID. +3. Execute required gates after every ID. +4. At checkpoint, promote proven IDs (`complete`, then `verified`) in `<=15` chunks. + +### Task 3: Implement Feature Group B (20 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStream.ClusterConsumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamCluster.Consumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterConsumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.JetStream.cs` + +**Steps:** +1. Set Group B IDs to `stub` using two commands (`15 + 5` IDs). +2. Run Per-Feature Verification Loop for each Group B ID. +3. Execute required gates after every ID. +4. At checkpoint, promote proven IDs in `<=15` chunks. + +### Task 4: Implement Feature Group C (18 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterConsumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamCluster.Consumers.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStream.ClusterConsumers.cs` + +**Steps:** +1. Set Group C IDs to `stub` using two commands (`15 + 3` IDs). +2. Run Per-Feature Verification Loop for each Group C ID. +3. Execute required gates after every ID. +4. At checkpoint, promote proven IDs in `<=15` chunks. + +### Task 5: Port Test Wave T1 (37 IDs) + +**Files:** +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests4.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` + +**Steps:** +1. Set T1 IDs to `stub` in `<=15` ID chunks. +2. Run Per-Test Verification Loop for each ID. +3. Run class filters per touched class. +4. Execute checkpoint and promote proven IDs in `<=15` chunks. + +### Task 6: Port Test Wave T2 (39 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ServerOptionsTests.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigCheckTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsServerTests.Impltests.cs` + +**Steps:** +1. Set T2 IDs to `stub` in `<=15` ID chunks. +2. Run Per-Test Verification Loop for each ID. +3. Run class filters per touched class. +4. Execute checkpoint and promote proven IDs in `<=15` chunks. + +### Task 7: Port Test Wave T3 (33 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamJwtTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthCalloutTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AuthHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/CertificateStoreWindowsTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/AccountTests.Impltests.cs` + +**Steps:** +1. Set T3 IDs to `stub` in `<=15` ID chunks. +2. Run Per-Test Verification Loop for each ID. +3. Run class filters per touched class. +4. Execute checkpoint and promote proven IDs in `<=15` chunks. + +### Task 8: Port Test Wave T4 (32 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/WebSocketHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeProxyTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RouteHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs` + +**Steps:** +1. Set T4 IDs to `stub` in `<=15` ID chunks. +2. Run Per-Test Verification Loop for each ID. +3. Run class filters per touched class. +4. Execute checkpoint and promote proven IDs in `<=15` chunks. + +### Task 9: Port Test Wave T5 (19 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MqttHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamLeafNodeTests.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamSuperClusterTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MessageTracerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/MonitoringHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` + +**Steps:** +1. Set T5 IDs to `stub` in `<=15` ID chunks. +2. Run Per-Test Verification Loop for each ID. +3. Run class filters per touched class. +4. Execute checkpoint and promote proven IDs in `<=15` chunks. + +### Task 10: Final Batch Verification and Closure + +**Files:** +- Modify: `porting.db` +- Modify: `reports/current.md` (via generator) + +**Steps:** +1. Run final gates (`stub scan`, `build`, full unit tests). +2. Run audits and reconcile status mismatches: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +``` +3. Ensure all Batch 34 IDs are `verified` or explicit `deferred` with reason. +4. Complete batch: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 34 --db porting.db +``` +5. Generate report: +```bash +./reports/generate-report.sh +``` + diff --git a/docs/plans/2026-02-27-batch-35-js-cluster-remaining-design.md b/docs/plans/2026-02-27-batch-35-js-cluster-remaining-design.md new file mode 100644 index 0000000..b6bad6d --- /dev/null +++ b/docs/plans/2026-02-27-batch-35-js-cluster-remaining-design.md @@ -0,0 +1,135 @@ +# Batch 35 JS Cluster Remaining Design + +**Date:** 2026-02-27 +**Batch:** 35 (`JS Cluster Remaining`) +**Scope:** 57 features + 49 unit tests +**Dependency:** batch `32` (`JS Cluster Meta`) +**Go source:** `golang/nats-server/server/jetstream_cluster.go` + +## Problem + +Batch 35 covers the remaining JetStream cluster behavior in `server/jetstream_cluster.go` (roughly lines `8766-10866`): delete-range and assignment encoding/decoding, stream snapshot/catchup processing, cluster info assembly, catchup throttling counters, and sync subject helpers. It also includes 49 tests, mostly `RaftNodeTests`, that validate catchup/truncation and snapshot correctness. + +The plan must prevent false progress: no placeholder feature ports and no fake-pass tests. + +## Context Findings + +### Required command outputs + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 35 --db porting.db` + - Status: `pending` + - Features: `57` (all `deferred`) + - Tests: `49` (all `deferred`) + - Depends on: `32` + - Go file: `server/jetstream_cluster.go` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` + - Confirms chain around this area: `32 -> 33/34/35`. +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + +Environment note: `dotnet` was not on `PATH` in this shell; use `/usr/local/share/dotnet/dotnet`. + +### Feature ownership distribution (from `porting.db`) + +- `NatsStream`: 27 +- `JetStreamCluster`: 19 +- `NatsServer`: 8 +- `JetStreamEngine`: 3 + +### Test distribution (from `porting.db`) + +- `RaftNodeTests`: 42 +- `JetStreamClusterTests1`: 6 +- `JetStreamBatchingTests`: 1 + +## Constraints and Success Criteria + +- Planning only; no implementation execution in this session. +- Reuse Batch 0 rigor, but for **features + tests**. +- Feature tasks must be grouped in chunks of max ~20 features. +- Status updates must use batch-update chunks of at most 15 IDs. +- Blocked work must be marked `deferred` with explicit reason, never stubbed. + +Success means all 57 feature IDs and 49 test IDs are either: +- promoted with verification evidence (`complete/verified`), or +- kept `deferred` with specific blocker notes. + +## Approaches + +### Approach A: Monolithic pass (all features, then all tests) + +- Pros: simple sequencing. +- Cons: high risk, poor auditability, hard to isolate regressions. + +### Approach B (Recommended): Three feature groups plus four test waves with hard gates + +- Pros: bounded scope, clearer rollback points, aligns with max-20 feature grouping and max-15 status updates. +- Cons: more command overhead. + +### Approach C: Test-first for all 49 tests before feature completion + +- Pros: immediate behavior pressure. +- Cons: high churn because most tests depend on unported catchup/snapshot internals. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Code Organization + +Use behavior-focused files while keeping existing class ownership: + +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` + - assignment/sync subject encode/decode helpers and cluster utility functions. +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` + - snapshot and catchup state transitions plus inbound cluster message processing. +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + - offline/online cluster info and alternate-stream assembly. +- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` and/or new partial `NatsServer.JetStreamClusterRemaining.cs` + - clustered consumer request path and gcb accounting helpers. + +### 2. Feature Grouping (max ~20) + +- **Group A (20 IDs):** `1694-1713` + Delete-range, consumer assignment, stream/batch encode-decode, snapshot support/state capture, clustered inbound message entry point. +- **Group B (20 IDs):** `1714-1733` + Delete trace, sync request calculation, snapshot delete handling, catchup peer lifecycle, snapshot/catchup processing, stream sync handler, cluster info base methods. +- **Group C (17 IDs):** `1734-1750` + Cluster info checks, stream alternates/info request handling, gcb accounting/kick channel, run-catchup loop, sync subject helper family. + +### 3. Test Wave Grouping + +- **Wave T1 (7 IDs):** `730,846,847,848,890,891,893` (`JetStreamBatchingTests` + `JetStreamClusterTests1`) +- **Wave T2 (14 IDs):** `2640,2641,2643,2644,2645,2646,2647,2648,2649,2653,2655,2656,2658,2659` +- **Wave T3 (14 IDs):** `2660,2661,2662,2665,2666,2668,2669,2673,2676,2677,2678,2679,2680,2681` +- **Wave T4 (14 IDs):** `2682,2683,2684,2685,2686,2688,2691,2696,2697,2703,2715,2716,2717,2719` + +### 4. Verification Strategy (Design-Level) + +- Every feature follows a per-feature loop with focused test evidence. +- Every test follows method-level then class-level verification. +- Stub detection runs after each loop and before status promotions. +- Build and targeted/full test gates are mandatory before checkpoint status updates. +- Checkpoints occur between every task boundary. + +### 5. Deferral Strategy + +If infrastructure or dependency behavior blocks a feature/test: + +1. Stop work on that ID. +2. Do not add placeholder implementation/assertions. +3. Mark `deferred` with explicit reason via `--override`. +4. Continue with next unblocked item. + +## Risks and Mitigations + +- **Dependency readiness risk (Batch 32):** enforce preflight before `batch start 35`. +- **Raft-heavy test concentration:** split `RaftNodeTests` into three equal waves and checkpoint each wave. +- **Stub regression under volume:** hard anti-stub scans and strict status chunking (`<=15`). +- **Class ownership drift:** keep methods in mapped classes only (`JetStreamCluster`, `NatsStream`, `JetStreamEngine`, `NatsServer`). + +## Non-Goals + +- Executing the implementation in this session. +- Expanding scope beyond Batch 35 mappings. +- Changing batch dependencies/order in PortTracker. diff --git a/docs/plans/2026-02-27-batch-35-js-cluster-remaining-implementation-plan.md b/docs/plans/2026-02-27-batch-35-js-cluster-remaining-implementation-plan.md new file mode 100644 index 0000000..f4962d2 --- /dev/null +++ b/docs/plans/2026-02-27-batch-35-js-cluster-remaining-implementation-plan.md @@ -0,0 +1,410 @@ +# Batch 35 JS Cluster Remaining Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 35 (`JS Cluster Remaining`) from `server/jetstream_cluster.go`, with auditable status transitions for all 57 features and 49 tests. + +**Architecture:** Execute three bounded feature groups (`20/20/17`) mapped to `JetStreamCluster`, `NatsStream`, `JetStreamEngine`, and `NatsServer`, then execute four test waves (`7/14/14/14`) with strict method-level verification evidence. Every promotion is gated by stub scans, build/test gates, and checkpoint protocol between tasks. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-35-js-cluster-remaining-design.md` + +--- + +## Batch 35 Scope + +- Batch ID: `35` +- Name: `JS Cluster Remaining` +- Dependency: `32` +- Go source: `golang/nats-server/server/jetstream_cluster.go` +- Features: `57` (`1694-1750`) +- Tests: `49` (`730,846,847,848,890,891,893,2640,2641,2643,2644,2645,2646,2647,2648,2649,2653,2655,2656,2658,2659,2660,2661,2662,2665,2666,2668,2669,2673,2676,2677,2678,2679,2680,2681,2682,2683,2684,2685,2686,2688,2691,2696,2697,2703,2715,2716,2717,2719`) + +If `dotnet` is not on `PATH`, use: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Primary production files (expected touch set): + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify or Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterRemaining.cs` +- Modify (if not using partial file): `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` + +Primary test files (expected touch set): + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBatchingTests.Impltests.cs` +- Modify or Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify or Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every Batch 35 feature/test ID must pass this protocol before promotion. + +### Dependency Preflight Gate (before any status changes) + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 32 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 35 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Only if Batch 35 is ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 35 --db porting.db +``` + +Capture baseline evidence: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapping and Go line span: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` +2. Mark only that feature as in-progress: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` +3. Add or update at least one focused test that fails for intended behavior. +4. Run focused test; verify failure reason matches target behavior. +5. Implement minimal feature logic (no placeholder code). +6. Run **Stub Detection Check**. +7. Run **Build Gate**. +8. Run **Test Gate** (method/class filters for impacted area). +9. If green, mark feature `complete`; defer `verified` promotion until task checkpoint. + +### Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapping and Go source: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` +2. Mark only that test as in-progress: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` +3. Port full Arrange/Act/Assert behavior. +4. Run single test method: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` +5. Confirm summary includes `Passed: 1, Failed: 0` (never `Passed: 0`). +6. Run class-level filter for touched class. +7. Run **Stub Detection Check** + **Build Gate**. +8. If green, mark test `complete`; defer `verified` promotion until task checkpoint. + +### Stub Detection Check (REQUIRED after every feature/test loop) + +Run on changed C# files only: + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass|ShouldBe\\(true\\)|=>\\s*default;|=>\\s*null;|return\\s+null;\\s*$|return\\s+0;\\s*$|return\\s+false;\\s*$)" +fi +``` + +Any match in mapped methods/tests blocks promotion. + +### Build Gate (REQUIRED) + +Run: + +```bash +$DOTNET build dotnet/ +``` + +Required timing: + +- after each feature loop +- after each test loop +- before every `batch-update` +- at every checkpoint + +### Test Gate (REQUIRED) + +Minimum gates: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStream" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftNodeTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~ImplBacklog" +``` + +Per-item focused gates are mandatory: + +- Feature loop: nearest impacted class filters. +- Test loop: method filter, then class filter. + +### Status Update Protocol (HARD LIMIT: 15 IDs max per `batch-update`) + +- Never pass more than `15` IDs in one `feature batch-update` or `test batch-update` call. +- Never promote `verified` without loop evidence and checkpoint evidence. +- Never update IDs outside active task scope. +- For blocked items, keep `deferred` and provide explicit `--override` reason. + +Templates: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +Before starting the next task: + +1. Run **Stub Detection Check**. +2. Run **Build Gate**. +3. Run class filters for all touched classes. +4. Run full unit suite: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` +5. Promote only proven IDs in `<=15` chunks. +6. Commit checkpoint before next task. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +These apply to features and tests. + +### Forbidden Patterns + +- `throw new NotImplementedException()` in any mapped Batch 35 feature/test. +- Empty method bodies for mapped features. +- Placeholder comments in mapped paths (`TODO`, `PLACEHOLDER`, `later`). +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`, tautological assertions). +- Constant-return shortcuts in mapped features that ignore inputs/state (`return false;`, `return 0;`, `return null;`, `return string.Empty;`). +- Catch-and-ignore blocks that suppress failures to force green results. + +### Hard Limits + +- Feature task group size max: `~20` IDs. +- `feature batch-update` max IDs: `15`. +- `test batch-update` max IDs: `15`. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between every task. +- No `verified` promotion without checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +1. Stop on the current ID immediately. +2. Do not add placeholder implementation or fake assertions. +3. Mark blocked item `deferred` with explicit reason. +4. Move to next unblocked ID. + +Feature deferral command: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db +``` + +Test deferral command: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +Deferred with reason is correct behavior. Stubbed/fake-pass progress is not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (20): delete-range/assignment codecs + snapshot capability foundations + +IDs: +`1694,1695,1696,1697,1698,1699,1700,1701,1702,1703,1704,1705,1706,1707,1708,1709,1710,1711,1712,1713` + +### Group B (20): snapshot delete/catchup peer lifecycle + sync/catchup processing + cluster info base + +IDs: +`1714,1715,1716,1717,1718,1719,1720,1721,1722,1723,1724,1725,1726,1727,1728,1729,1730,1731,1732,1733` + +### Group C (17): cluster stream info, gcb controls, runCatchup, and sync subject helpers + +IDs: +`1734,1735,1736,1737,1738,1739,1740,1741,1742,1743,1744,1745,1746,1747,1748,1749,1750` + +## Test Waves + +### Wave T1 (7): batching + cluster-meta-recovery/offline strict decoding + +IDs: +`730,846,847,848,890,891,893` + +### Wave T2 (14): raft catchup/truncate/snapshot baseline + +IDs: +`2640,2641,2643,2644,2645,2646,2647,2648,2649,2653,2655,2656,2658,2659` + +### Wave T3 (14): raft health/quorum/leader-change/snapshot replay + +IDs: +`2660,2661,2662,2665,2666,2668,2669,2673,2676,2677,2678,2679,2680,2681` + +### Wave T4 (14): raft startup/rollback/install snapshot and peer replay + +IDs: +`2682,2683,2684,2685,2686,2688,2691,2696,2697,2703,2715,2716,2717,2719` + +--- + +### Task 1: Preflight and Baseline + +**Files:** +- Read: `docs/plans/2026-02-27-batch-35-js-cluster-remaining-design.md` +- Read: `golang/nats-server/server/jetstream_cluster.go` + +**Steps:** +1. Run dependency preflight commands. +2. Start Batch 35 only if dependency checks pass. +3. Capture baseline build/test evidence. +4. Do not promote any ID before baseline evidence exists. + +### Task 2: Implement Feature Group A (20 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterRemaining.cs` + +**Steps:** +1. Move Group A IDs to `stub` in two `batch-update` calls (`15 + 5`). +2. Execute Per-Feature Verification Loop for each ID. +3. Run mandatory gates after each ID. +4. Apply checkpoint protocol. +5. Promote proven Group A IDs to `complete/verified` in `<=15` chunks. +6. Commit checkpoint. + +### Task 3: Implement Feature Group B (20 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` + +**Steps:** +1. Move Group B IDs to `stub` in two `batch-update` calls (`15 + 5`). +2. Execute Per-Feature Verification Loop for each ID. +3. Run mandatory gates after each ID. +4. Apply checkpoint protocol. +5. Promote proven Group B IDs to `complete/verified` in `<=15` chunks. +6. Commit checkpoint. + +### Task 4: Implement Feature Group C (17 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.JetStreamClusterRemaining.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamClusterTypes.cs` + +**Steps:** +1. Move Group C IDs to `stub` in two `batch-update` calls (`15 + 2`). +2. Execute Per-Feature Verification Loop for each ID. +3. Run mandatory gates after each ID. +4. Apply checkpoint protocol. +5. Promote proven Group C IDs to `complete/verified` in `<=15` chunks. +6. Commit checkpoint. + +### Task 5: Port Test Wave T1 (7 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamBatchingTests.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` + +**Steps:** +1. Move T1 IDs to `stub` in one call. +2. Execute Per-Test Verification Loop for each ID. +3. Run class-level filters for touched classes. +4. Apply checkpoint protocol. +5. Promote proven T1 IDs to `complete/verified` in `<=15` chunks. +6. Commit checkpoint. + +### Task 6: Port Test Wave T2 (14 IDs) + +**Files:** +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Steps:** +1. Move T2 IDs to `stub` in one call. +2. Execute Per-Test Verification Loop for each ID. +3. Run class-level `RaftNodeTests` filter. +4. Apply checkpoint protocol. +5. Promote proven T2 IDs to `complete/verified` in `<=15` chunks. +6. Commit checkpoint. + +### Task 7: Port Test Wave T3 (14 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Steps:** +1. Move T3 IDs to `stub` in one call. +2. Execute Per-Test Verification Loop for each ID. +3. Run class-level `RaftNodeTests` filter. +4. Apply checkpoint protocol. +5. Promote proven T3 IDs to `complete/verified` in `<=15` chunks. +6. Commit checkpoint. + +### Task 8: Port Test Wave T4 (14 IDs) + +**Files:** +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs` + +**Steps:** +1. Move T4 IDs to `stub` in one call. +2. Execute Per-Test Verification Loop for each ID. +3. Run class-level `RaftNodeTests` filter. +4. Apply checkpoint protocol. +5. Promote proven T4 IDs to `complete/verified` in `<=15` chunks. +6. Commit checkpoint. + +### Task 9: Final Batch 35 Closure + +**Files:** +- Modify: `porting.db` +- Modify: `reports/current.md` + +**Steps:** +1. Run final audit checks: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db +``` +2. Resolve any audit mismatches (or explicit override reasons). +3. Complete batch: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 35 --db porting.db +``` +4. Generate updated report: +```bash +./reports/generate-report.sh +``` +5. Final regression gate: +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` +6. Commit final Batch 35 closure. diff --git a/docs/plans/2026-02-27-batch-36-stream-lifecycle-design.md b/docs/plans/2026-02-27-batch-36-stream-lifecycle-design.md new file mode 100644 index 0000000..aaab4e8 --- /dev/null +++ b/docs/plans/2026-02-27-batch-36-stream-lifecycle-design.md @@ -0,0 +1,156 @@ +# Batch 36 Stream Lifecycle Design + +**Date:** 2026-02-27 +**Batch:** 36 (`Stream Lifecycle`) +**Scope:** 92 features + 53 unit tests +**Dependencies:** Batches `8, 11, 12, 13, 14, 15, 28` +**Go source:** `golang/nats-server/server/stream.go` (focus through ~line 4600) + +## Problem + +Batch 36 is the lifecycle/core control plane for JetStream streams: stream creation, assignment, config validation/update, purge/delete flows, mirror/source setup and message processing, and internal subscriptions. This batch is a gating dependency for Batch 37 (`Stream Messages`) and Batch 38 (`Consumer Lifecycle`), so fake progress here creates downstream breakage. + +The current .NET surface is materially under-ported for this area: + +- Batch 36 features are all `deferred`. +- Most mapped methods are missing from source classes. +- Existing `ImplBacklog` tests for related classes are largely placeholder-style and need replacement with behavior checks. + +## Context Findings + +### Required command outputs + +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db` + - Status: `pending` + - Features: `92` (`3196-3293` with gaps) + - Tests: `53` + - Depends on: `8,11,12,13,14,15,28` + - Go file: `server/stream.go` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch list --db porting.db` + - Confirms chain: `36 -> 37,38` +- `/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db` + - Overall progress: `1924/6942 (27.7%)` + +Environment note: `dotnet` is not on `PATH` in this shell. Use `/usr/local/share/dotnet/dotnet` explicitly. + +### Feature ownership split (from `porting.db`) + +- `NatsStream`: 76 +- `Account`: 4 +- `PersistModeType`: 3 +- `StreamSourceInfo`: 2 +- `StreamSource`: 2 +- `JsAccount`: 2 +- `NatsServer`: 1 +- `InMsg`: 1 +- `ExternalStream`: 1 + +### Test split (from `porting.db`) + +- `JetStreamEngineTests`: 35 +- `NatsConsumerTests`: 4 +- `JetStreamClusterTests1`: 3 +- `JetStreamClusterTests2`: 4 +- `JetStreamClusterTests3`: 3 +- `ConcurrencyTests1`: 2 +- `JetStreamFileStoreTests`: 1 +- `StorageEngineTests`: 1 + +Additional finding: `JetStreamClusterTests1.Impltests.cs` and `JetStreamClusterTests3.Impltests.cs` do not currently exist and must be created. + +## Approaches + +### Approach A: Single-file port in existing files only + +Port all methods directly into existing files (`NatsStream.cs`, `Account.cs`, `JetStreamTypes.cs`, etc.) without structural splits. + +- Pros: minimal file churn. +- Cons: high merge risk, hard reviewability, and poor fault isolation for 76 `NatsStream` methods. + +### Approach B (Recommended): Lifecycle-focused partial-file decomposition + staged verification + +Keep mapped owning classes intact but split implementation by concern into new partial files for stream lifecycle/mirror/source/subscription paths. Execute in bounded feature groups and test waves with strict evidence gates. + +- Pros: manageable review units, clearer ownership, better checkpointing and rollback, aligns with anti-stub guardrails. +- Cons: requires adding partial declarations where classes are currently non-partial. + +### Approach C: Test-first broad wave before feature completion + +Attempt to port all 53 tests up front to drive implementation. + +- Pros: fast behavior pressure. +- Cons: most tests depend on missing lifecycle internals and will churn heavily; expensive red-state noise. + +**Decision:** Approach B. + +## Proposed Design + +### 1. Code organization strategy + +Use concern-oriented partial files while preserving mapped class ownership: + +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` (retain core type; convert to `partial`) +- Create `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Lifecycle.cs` +- Create `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Mirror.cs` +- Create `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Source.cs` +- Create `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Subscriptions.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` (convert to `partial`) +- Create `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamLifecycle.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` (for `JsAccount`, convert class to `partial` if split) +- Create `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JsAccount.StreamLifecycle.cs` +- `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.cs` or `NatsServer.*.cs` partial for `CheckStreamCfg` +- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs` and `StreamTypes.cs` for `PersistModeType`, `ExternalStream.Domain`, `StreamSource`/`StreamSourceInfo`, `InMsg` + +### 2. Functional decomposition + +- **Config and creation path:** `PersistModeType`, `ExternalStream.Domain`, `Account.AddStream*`, `StreamSource.ComposeIName/SetIndexName`, `NatsServer.CheckStreamCfg`, `JsAccount.ConfigUpdateCheck`, `NatsStream.Update*`. +- **State/advisory path:** leader checks, sequence counters (`CLFS`), created time, create/update/delete advisories, purge/delete/erase behavior. +- **Mirror path:** mirror setup/retry/cancel, inbound mirror flow-control handling, lag tracking, retry backoff. +- **Source path:** source setup/retry/cancel, shared source message pump, source headers, start-sequence reconstruction. +- **Subscription/internal path:** stream/direct/mirror-direct subscriptions, source consumer stop/unsubscribe, batching cleanup, internal subscribe helpers. + +### 3. Test design + +Port test IDs in waves tied to implemented behavior: + +- Wave 1: file store + cluster/meta + consumer API basics (15 tests) +- Wave 2-4: `JetStreamEngineTests` heavy stream lifecycle scenarios (12/12/14) + +Test files: + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/StorageEngineTests.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` + +### 4. Verification and risk controls (design-level) + +- Per-feature and per-test loops are mandatory before status promotion. +- Stub scans are mandatory after each loop and at each checkpoint. +- Build and test gates are mandatory before every status batch update. +- Status updates are chunked to max 15 IDs. +- If blocked: remain `deferred` with explicit reason, never placeholder implementation. + +## Feature Grouping (for implementation plan) + +- Group A (20): IDs `3196-3219` (actual mapped IDs, 20 total) +- Group B (20): IDs `3220-3240` (actual mapped IDs, 20 total) +- Group C (20): IDs `3241-3261` (actual mapped IDs, 20 total) +- Group D (20): IDs `3262-3281` (actual mapped IDs, 20 total) +- Group E (12): IDs `3282-3293` (actual mapped IDs, 12 total) + +## Constraints + +- Planning only in this session; no implementation execution. +- Must preserve .NET standards (`xUnit 3`, `Shouldly`, `NSubstitute`, nullable, naming conventions). +- Must avoid fake-pass tests and placeholder feature bodies. + +## Non-Goals + +- Executing Batch 36 implementation in this session. +- Expanding scope beyond Batch 36 mapped features/tests. +- Building new integration infrastructure not required by mapped unit tests. diff --git a/docs/plans/2026-02-27-batch-36-stream-lifecycle-implementation-plan.md b/docs/plans/2026-02-27-batch-36-stream-lifecycle-implementation-plan.md new file mode 100644 index 0000000..ee31b2d --- /dev/null +++ b/docs/plans/2026-02-27-batch-36-stream-lifecycle-implementation-plan.md @@ -0,0 +1,410 @@ +# Batch 36 Stream Lifecycle Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task. + +**Goal:** Port and verify Batch 36 (`Stream Lifecycle`) from `server/stream.go` so all 92 mapped features and 53 mapped tests are either genuinely implemented/verified or explicitly deferred with documented blockers. + +**Architecture:** Implement in five bounded feature groups (`20/20/20/20/12`) aligned to `NatsStream` lifecycle domains (config/update, mirror, source, subscriptions) plus supporting `Account`, `NatsServer`, `JsAccount`, and stream type helpers. Execute four test waves with strict per-test evidence and anti-stub enforcement before any status promotion. + +**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`) + +**Design doc:** `docs/plans/2026-02-27-batch-36-stream-lifecycle-design.md` + +--- + +## Batch 36 Scope + +- Batch ID: `36` +- Name: `Stream Lifecycle` +- Dependencies: `8,11,12,13,14,15,28` +- Go source: `golang/nats-server/server/stream.go` +- Features: `92` (`3196-3293` mapped IDs) +- Tests: `53` + +If `dotnet` is not on `PATH`, use: + +```bash +DOTNET=/usr/local/share/dotnet/dotnet +``` + +Primary production files (expected): + +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Lifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Mirror.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Source.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Subscriptions.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JsAccount.StreamLifecycle.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamLifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.StreamLifecycle.cs` + +Primary test files (expected): + +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/StorageEngineTests.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` + +--- + +## MANDATORY VERIFICATION PROTOCOL + +> **NON-NEGOTIABLE:** Every feature/test in Batch 36 must pass this loop. No shortcut status updates. + +### Dependency Preflight (before any status change) + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 8 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 11 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 12 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 13 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 14 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 15 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 28 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db +``` + +Start only when ready: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch start 36 --db porting.db +``` + +Capture baseline: + +```bash +$DOTNET build dotnet/ +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Per-Feature Verification Loop (REQUIRED per feature ID) + +1. Inspect mapping and Go context: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature show --db porting.db +``` +2. Mark claim-in-progress: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- feature update --status stub --db porting.db +``` +3. Add/adjust focused test(s) for that feature behavior (red first when practical). +4. Implement minimal real logic (no placeholder bodies). +5. Run **Stub Detection Check** on touched files. +6. Run **Build Gate**. +7. Run targeted **Test Gate** for affected classes. +8. If green, promote feature to `complete` (or `verified` only with checkpoint evidence). +9. Record evidence before touching next feature. + +### Per-Test Verification Loop (REQUIRED per test ID) + +1. Inspect mapping and Go test location: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test show --db porting.db +``` +2. Mark claim-in-progress: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- test update --status stub --db porting.db +``` +3. Port full Arrange/Act/Assert behavior against production code. +4. Run the single test method: +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \ + --filter "FullyQualifiedName~." --verbosity normal +``` +5. Confirm output shows `Passed: 1, Failed: 0` (never `Passed: 0`). +6. Run class-level filter for touched test class. +7. Run **Stub Detection Check** + **Build Gate**. +8. Promote test to `complete` (or `verified` only with checkpoint evidence). + +### Stub Detection Check (REQUIRED after every loop) + +Run on changed C# files in source + tests: + +```bash +changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true) +if [ -n "$changed_files" ]; then + echo "$changed_files" | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass|ShouldContain\\(\"Should\"\\)|var goFile = \"server/|JetStreamVersioning\\.GetRequiredApiLevel\\(new Dictionary\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$)" +fi +``` + +Any match in mapped Batch 36 methods/tests blocks promotion until resolved. + +### Build Gate (REQUIRED) + +```bash +$DOTNET build dotnet/ +``` + +Run: + +- after every feature loop +- after every test loop +- before any batch-update +- at every task checkpoint + +### Test Gate (REQUIRED) + +Minimum class/domain gates per touched area: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamEngineTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamClusterTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~NatsConsumerTests" +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~StorageEngineTests|FullyQualifiedName~JetStreamFileStoreTests|FullyQualifiedName~ConcurrencyTests1" +``` + +Checkpoint/full gate: + +```bash +$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ +``` + +### Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update) + +- Never pass more than `15` IDs per `feature batch-update` or `test batch-update` call. +- Never promote `verified` without loop evidence + checkpoint gate. +- Never update IDs outside current task scope. +- For blocked work, keep `deferred` with explicit reason. + +Templates: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature batch-update --ids "" --set-status --db porting.db --execute + +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test batch-update --ids "" --set-status --db porting.db --execute +``` + +### Checkpoint Protocol Between Tasks (REQUIRED) + +Before starting the next task: + +1. Run **Stub Detection Check** on current diff. +2. Run **Build Gate**. +3. Run class filters for all touched classes. +4. Run full unit test gate. +5. Promote only proven IDs in `<=15` chunks. +6. Commit checkpoint before the next task. + +--- + +## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE) + +### Forbidden Patterns + +These are forbidden in mapped Batch 36 features/tests: + +- `throw new NotImplementedException()` +- Empty or no-op mapped method bodies +- Placeholder comments (`TODO`, `PLACEHOLDER`, `later`) +- Fake-pass assertions (`Assert.True(true)`, `Assert.Pass()`, tautological string checks) +- Template test bodies that only assert literals, including patterns such as: + - `var goFile = "server/..."` + - `"".ShouldContain("Should")` + - `JetStreamVersioning.GetRequiredApiLevel(new Dictionary()).ShouldBe(string.Empty)` as the core assertion +- Constant-return shortcuts in mapped feature methods that ignore state/input (`return default;`, `return null;`, hardcoded primitives) unless directly required by Go behavior and covered by tests +- Catch-and-ignore blocks used to force green tests + +### Hard Limits + +- Feature groups must stay at `~20` IDs max. +- `feature batch-update` maximum IDs per call: `15`. +- `test batch-update` maximum IDs per call: `15`. +- One active feature loop at a time. +- One active test loop at a time. +- Mandatory checkpoint between every task. +- No `verified` promotion without checkpoint evidence. + +### If You Get Stuck (REQUIRED) + +1. Stop on the current ID immediately. +2. Do **not** add placeholder code or fake assertions. +3. Mark blocked item `deferred` with explicit reason. +4. Move to next unblocked ID. + +Feature deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + feature update --status deferred --override "blocked: " --db porting.db +``` + +Test deferral: + +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- \ + test update --status deferred --override "blocked: " --db porting.db +``` + +`deferred` with a concrete blocker is correct. Stubbed progress is not. + +--- + +## Feature Groups (max ~20 each) + +### Group A (20): Stream creation, assignment, leadership primitives + +IDs: +`3196,3197,3198,3200,3201,3202,3203,3204,3205,3206,3207,3208,3209,3210,3211,3213,3214,3216,3217,3219` + +### Group B (20): Limits, dedupe, advisories, config validation/update entry points + +IDs: +`3220,3221,3222,3223,3224,3225,3226,3227,3228,3229,3230,3231,3232,3233,3234,3235,3237,3238,3239,3240` + +### Group C (20): Update internals, purge/delete, mirror processing/retry foundation + +IDs: +`3241,3242,3244,3245,3246,3247,3248,3249,3250,3251,3252,3253,3254,3255,3256,3257,3258,3259,3260,3261` + +### Group D (20): Source consumer setup, flow control, source header/sequence recovery + +IDs: +`3262,3263,3264,3265,3266,3267,3268,3269,3270,3271,3272,3273,3274,3275,3276,3277,3278,3279,3280,3281` + +### Group E (12): Direct/mirror-direct subscriptions, internal subscribe/unsubscribe cleanup + +IDs: +`3282,3283,3284,3285,3286,3287,3288,3289,3290,3291,3292,3293` + +## Test Waves + +### Wave T1 (15): file-store + cluster + consumer API basics + +IDs: +`548,849,869,902,951,953,955,990,1032,1034,1125,1227,1262,1315,1317` + +### Wave T2 (12): stream create/update/publish baseline + +IDs: +`1468,1470,1471,1474,1475,1478,1480,1481,1482,1483,1504,1512` + +### Wave T3 (12): limits, perf-sensitive behavior, republish cases + +IDs: +`1532,1544,1554,1555,1564,1565,1614,1623,1635,1636,1637,1638` + +### Wave T4 (14): republish tail, direct get, versioning, concurrency/storage + +IDs: +`1639,1640,1644,1645,1646,1653,1656,1657,1669,1741,1743,2387,2402,2952` + +--- + +### Task 1: Preflight and Baseline + +**Files:** +- Read: `docs/plans/2026-02-27-batch-36-stream-lifecycle-design.md` +- Read: `golang/nats-server/server/stream.go` + +**Steps:** +1. Run dependency preflight commands. +2. Start Batch 36 only if ready. +3. Capture baseline build/test evidence. +4. Do not promote any IDs before baseline evidence exists. + +### Task 2: Implement Feature Group A (20 IDs) + +**Files:** +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamLifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Lifecycle.cs` + +**Steps:** +1. Move Group A IDs to `stub` using `15 + 5` update chunks. +2. Execute Per-Feature Verification Loop per ID. +3. Run mandatory gates after each ID. +4. Promote proven IDs (`complete/verified`) in `<=15` chunks. +5. Run Checkpoint Protocol. + +### Task 3: Implement Feature Group B (20 IDs) + Test Wave T1 (15 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/NatsServer.StreamLifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JsAccount.StreamLifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Lifecycle.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs` +- Modify/Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests3.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/NatsConsumerTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs` + +**Steps:** +1. Implement Group B features with per-feature loop and gates. +2. Port Wave T1 tests with per-test loop and gates. +3. Apply status updates in max-15 chunks only. +4. Run Checkpoint Protocol. + +### Task 4: Implement Feature Group C (20 IDs) + Test Wave T2 (12 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Lifecycle.cs` +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Mirror.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` + +**Steps:** +1. Implement Group C feature IDs with per-feature loop. +2. Port Wave T2 tests. +3. Run stub/build/test gates and checkpoint. +4. Promote only proven IDs in `<=15` chunks. + +### Task 5: Implement Feature Group D (20 IDs) + Test Wave T3 (12 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Source.cs` +- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` + +**Steps:** +1. Implement Group D features with per-feature loop. +2. Port Wave T3 tests. +3. Run mandatory gates and checkpoint. +4. Promote only proven IDs in `<=15` chunks. + +### Task 6: Implement Feature Group E (12 IDs) + Test Wave T4 (14 IDs) + +**Files:** +- Modify/Create: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Subscriptions.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs` +- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/StorageEngineTests.Impltests.cs` + +**Steps:** +1. Implement Group E feature IDs. +2. Port Wave T4 tests. +3. Run mandatory gates and checkpoint. +4. Promote proven IDs in `<=15` chunks. + +### Task 7: Batch 36 Final Closure + +**Files:** +- Modify: `porting.db` +- Generate: `reports/current.md` (via report script) + +**Steps:** +1. Run final stub scan on all touched source/test files. +2. Run full build and full unit suite. +3. Validate batch state: +```bash +$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db +$DOTNET run --project tools/NatsNet.PortTracker -- report summary --db porting.db +``` +4. Ensure every Batch 36 ID is `complete/verified/n_a` or `deferred` with explicit blocker notes. +5. Generate report: +```bash +./reports/generate-report.sh +``` + +--- + +**Execution note:** Planning only. Do not execute this plan in this session. diff --git a/reports/current.md b/reports/current.md index a6eb769..ca13c83 100644 --- a/reports/current.md +++ b/reports/current.md @@ -1,6 +1,6 @@ # NATS .NET Porting Status Report -Generated: 2026-02-27 21:33:11 UTC +Generated: 2026-02-27 22:01:32 UTC ## Modules (12 total) diff --git a/reports/report_c05d936.md b/reports/report_c05d936.md new file mode 100644 index 0000000..ca13c83 --- /dev/null +++ b/reports/report_c05d936.md @@ -0,0 +1,37 @@ +# NATS .NET Porting Status Report + +Generated: 2026-02-27 22:01:32 UTC + +## Modules (12 total) + +| Status | Count | +|--------|-------| +| verified | 12 | + +## Features (3673 total) + +| Status | Count | +|--------|-------| +| deferred | 2377 | +| n_a | 24 | +| stub | 1 | +| verified | 1271 | + +## Unit Tests (3257 total) + +| Status | Count | +|--------|-------| +| deferred | 2640 | +| n_a | 187 | +| verified | 430 | + +## Library Mappings (36 total) + +| Status | Count | +|--------|-------| +| mapped | 36 | + + +## Overall Progress + +**1924/6942 items complete (27.7%)**