dohertj2/natsnet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a841b553f2c2d58f266c08a2452d6864d5b7f42f
natsnet/docs/plans/2026-02-27-batch-29-jetstream-batching-implementation-plan.md
Joseph Doherty c05d93618e Add batch plans for batches 23-30 (rounds 12-15) 
Generated design docs and implementation plans via Codex for:
- Batch 23: Routes
- Batch 24: Leaf Nodes
- Batch 25: Gateways
- Batch 26: WebSocket
- Batch 27: JetStream Core
- Batch 28: JetStream API
- Batch 29: JetStream Batching
- Batch 30: Raft Part 1

All plans include mandatory verification protocol and anti-stub guardrails.
Updated batches.md with file paths and planned status.
2026-02-27 16:33:10 -05:00

16 KiB
Raw Blame History

Batch 29 JetStream Batching Implementation Plan

For Codex: REQUIRED SUB-SKILL: Use executeplan to implement this plan task-by-task.

Goal: Port and verify Batch 29 JetStream batching behavior (12 features, 3 tests) from server/jetstream_batching.go and mapped raft_test.go cases without introducing stubs or unverifiable tracker updates.

Architecture: Execute in two feature groups and one test group. Group A ports lifecycle/store primitives (1508-1514), Group B ports staged/apply/header checks (1515-1519, including the large checkMsgHeadersPreClusteredProposal surface), then mapped Raft tests are ported/verified. Every ID follows strict per-item evidence gates before status promotion.

Tech Stack: .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (porting.db)

Design doc: docs/plans/2026-02-27-batch-29-jetstream-batching-design.md

Batch 29 Scope

  • Batch ID: 29
  • Name: JetStream Batching
  • Dependency: 27
  • Go source: golang/nats-server/server/jetstream_batching.go
  • Features: 1508-1519 (12 total)
  • Tests: 2654,2674,2718

Primary implementation files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.cs
  • Optional split (recommended for readability of feature 1519):
    • Create/Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.HeaderChecks.cs
  • Optional supporting updates only if required by compile/runtime behavior:
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs

Primary test files:

  • Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeTests.cs
  • Modify or Create focused batching tests as needed:
    • dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingCoreTests.cs
  • Existing deferred batching integration placeholders (do not convert to fake unit tests):
    • dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingTests.cs

MANDATORY VERIFICATION PROTOCOL

NON-NEGOTIABLE: Applies to every feature ID and test ID in Batch 29.

Preflight Dependency Gate (REQUIRED before Task 1 coding)

  1. Verify dependency and readiness: 
    dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch show 29 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
  2. Start the batch only when dependency checks pass: 
    dotnet run --project tools/NatsNet.PortTracker -- batch start 29 --db porting.db
  3. Baseline gates: 
    dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

Per-Feature Verification Loop (REQUIRED for each feature ID)

For each feature in current task group:

  1. Inspect mapping and Go span: 
    dotnet run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
sed -n '<go_start>,<go_end>p' golang/nats-server/server/jetstream_batching.go
  2. Mark that feature as stub before editing: 
    dotnet run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
  3. Implement mapped behavior in .NET (no placeholders).
  4. Run Build Gate.
  5. Run Test Gate (focused tests for changed behavior).
  6. Run Stub Detection Check on touched files.
  7. If gates are green and no stubs are detected, promote feature to complete: 
    dotnet run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status complete --db porting.db
  8. Promote feature IDs to verified only after task-level checkpoint passes.

Per-Test Verification Loop (REQUIRED for each test ID)

  1. Inspect mapped test details and Go source: 
    dotnet run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
sed -n '<go_start>,<go_end>p' golang/nats-server/server/raft_test.go
  2. Mark test stub before editing: 
    dotnet run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
  3. Implement real Arrange/Act/Assert test calling production code.
  4. Run single-test command: 
    dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.JetStream.RaftNodeTests.<MethodName>" \
  --verbosity normal
  5. Confirm summary includes Passed: 1, Failed: 0 (not Passed: 0).
  6. Run class-level test gate: 
    dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.JetStream.RaftNodeTests"
  7. Run Stub Detection Check.
  8. Promote test to complete, then verified at checkpoint when evidence is complete.

Stub Detection Check (REQUIRED after every feature/test loop and each task)

Run against touched code:

git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests \
  | rg "\.cs$" \
  | xargs rg -n "(NotImplementedException|// TODO|// PLACEHOLDER|Assert\\.True\\(true\\)|Assert\\.Pass|throw new Exception\\(\"TODO\"\\)|=>\\s*default;|=>\\s*null;)"

Any match in touched methods means the item is not eligible for complete or verified.

Build Gate (REQUIRED)

dotnet build dotnet/ is mandatory:

  • after each feature implementation loop
  • after each test implementation loop
  • before any batch-update
  • at every task checkpoint

Test Gate (REQUIRED)

Minimum per-task gates:

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamBatching"
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftNodeTests"

Checkpoint and final gates:

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update)

  • Allowed path: deferred/not_started -> stub -> complete -> verified
  • Use batch-update with at most 15 IDs per call.
  • Never update IDs outside the current task group.
  • Never mark verified without corresponding build/test/stub-scan evidence.
  • If audit rejects status change, use --override "<specific reason>" only with explicit evidence.

Templates:

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

Checkpoint Protocol Between Tasks (REQUIRED)

At the end of each task before starting the next task:

  1. Run Stub Detection Check.
  2. Run Build Gate.
  3. Run task-relevant Test Gate.
  4. Run full unit suite: 
    dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
  5. Update status IDs for current task only (max 15 IDs per command).
  6. Commit checkpoint changes.

ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)

Forbidden Patterns (Features + Tests)

Any of these in touched methods means the work is a stub and must not be promoted:

  • throw new NotImplementedException()
  • Empty mapped method bodies
  • Placeholder comments: // TODO, // PLACEHOLDER, // later
  • Fake-pass assertions: Assert.True(true), Assert.Pass()
  • Tests that never call production code in ZB.MOM.NatsNet.Server
  • Constant-return placeholders for non-trivial logic (return null;, return 0;, return false;, return string.Empty;) without Go-equivalent behavior
  • Catch-all exception swallowing used to force pass behavior

Hard Limits

  • Max feature IDs per group: ~20
  • Max IDs per feature/test batch-update: 15
  • One active feature loop at a time
  • No verified promotion unless build + targeted tests + stub scan are green
  • One checkpoint commit per task minimum
  • Feature 1519 must not be merged as partial/no-op behavior

If You Get Stuck (MANDATORY)

  1. Stop work on the blocked ID immediately.
  2. Do not leave placeholder code or fake tests.
  3. Mark the blocked item deferred with specific reason: 
    dotnet run --project tools/NatsNet.PortTracker -- feature update <id> --status deferred --override "blocked: <specific reason>" --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- test update <id> --status deferred --override "blocked: <specific reason>" --db porting.db
  4. Move to the next unblocked ID in the same task.
  5. Deferred-with-reason is correct behavior; stubs are not.

Feature/Test Grouping (max ~20 per group)

Group A (7 features): Batch lifecycle + store creation

1508,1509,1510,1511,1512,1513,1514

Group B (5 features): Staged/apply state + pre-proposal header checks

1515,1516,1517,1518,1519

Group C (3 tests): Raft-node behavioral tests mapped to batch 29

2654,2674,2718

Task 1: Preflight and Batch Start

Files:

  • Read: docs/standards/dotnet-standards.md
  • Read: docs/plans/2026-02-27-batch-29-jetstream-batching-design.md
  • Read: golang/nats-server/server/jetstream_batching.go
  • Read: golang/nats-server/server/raft_test.go (mapped lines)

Step 1: Dependency/readiness checks

Run:

dotnet run --project tools/NatsNet.PortTracker -- batch show 27 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch show 29 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db

Expected:

  • Batch 29 is startable, dependency 27 satisfied.

Step 2: Start batch

Run:

dotnet run --project tools/NatsNet.PortTracker -- batch start 29 --db porting.db

Expected:

  • Batch transitions to in-progress.

Step 3: Baseline build/test

Run:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

Expected:

  • Known baseline captured before edits.

Step 4: Checkpoint protocol and commit

Task 2: Implement Group A Features (1508-1514)

Files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.cs
  • Optional modify if needed by compile/runtime wiring:
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs
  • Test: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingCoreTests.cs

Step 1: Mark Group A as stub (single chunk, 7 IDs <= 15)

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1508,1509,1510,1511,1512,1513,1514" --set-status stub --db porting.db --execute

Step 2: Execute Per-Feature Verification Loop for each ID in Group A

Expected:

  • Lifecycle/store methods are behaviorally implemented and individually gated.

Step 3: Task-level gates + checkpoint protocol

Run:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamBatching"

Step 4: Promote eligible Group A IDs to complete (blocked IDs stay deferred)

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<eligible subset of 1508-1514>" --set-status complete --db porting.db --execute

Step 5: After checkpoint evidence, promote eligible Group A IDs to verified

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<eligible subset up to 7 ids>" --set-status verified --db porting.db --execute

Task 3: Implement Group B Features (1515-1519)

Files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.cs
  • Optional create/modify to isolate large method:
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamBatching.HeaderChecks.cs
  • Optional supporting updates if required by dependencies:
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamTypes.cs
    • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/JetStreamErrors.cs
  • Test: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamBatchingCoreTests.cs

Step 1: Mark Group B as stub (single chunk, 5 IDs <= 15)

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1515,1516,1517,1518,1519" --set-status stub --db porting.db --execute

Step 2: Execute Per-Feature Verification Loop for each ID in Group B

Notes:

  • Treat 1519 as its own micro-loop: implement incrementally but only promote when full gate passes.
  • Validate error-path parity (duplicate msg ID, expected sequence checks, schedule/rollup validation, discard-new limits).

Step 3: Task-level gates + checkpoint protocol

Run:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamBatching"
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftTypesTests"

Step 4: Promote eligible Group B IDs to complete (blocked IDs stay deferred)

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<eligible subset of 1515-1519>" --set-status complete --db porting.db --execute

Step 5: After checkpoint evidence, promote eligible Group B IDs to verified

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<eligible subset up to 5 ids>" --set-status verified --db porting.db --execute

Task 4: Implement Group C Tests (2654,2674,2718)

Files:

  • Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftNodeTests.cs
  • Optional modify if helper setup reuse is needed:
    • dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/RaftTypesTests.cs
  • Source references:
    • golang/nats-server/server/raft_test.go

Step 1: Mark tests as stub

dotnet run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "2654,2674,2718" --set-status stub --db porting.db --execute

Step 2: Execute Per-Test Verification Loop for each test ID

Expected:

  • Each mapped test method is real, discovered, and passing individually.

Step 3: Task-level gates + checkpoint protocol

Run:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~RaftNodeTests"

Step 4: Promote eligible tests to complete

dotnet run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<eligible subset of 2654,2674,2718>" --set-status complete --db porting.db --execute

Step 5: After checkpoint evidence, promote eligible tests to verified

dotnet run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<eligible subset up to 3 ids>" --set-status verified --db porting.db --execute

Task 5: Batch 29 Final Verification and Closeout

Files:

  • Modify: porting.db
  • Generate: reports/current.md (via report script)

Step 1: Final mandatory gates

Run:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
dotnet run --project tools/NatsNet.PortTracker -- batch show 29 --db porting.db
dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db

Expected:

  • All Batch 29 IDs are verified or deferred with explicit reasons.

Step 2: Complete batch

Run:

dotnet run --project tools/NatsNet.PortTracker -- batch complete 29 --db porting.db

Step 3: Generate report + commit

./reports/generate-report.sh
git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream \
       dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream \
       porting.db reports/
git commit -m "feat(batch29): port jetstream batching and mapped raft tests"
Reference in New Issue View Git Blame Copy Permalink