dohertj2/natsnet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
34bc4e1137d25d899614727b8e2065eb7cafae56
natsnet/docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md
Joseph Doherty 8a126c4932 Add batch plans for batches 37-41 (rounds 19-21) 
Generated design docs and implementation plans via Codex for:
- Batch 37: Stream Messages
- Batch 38: Consumer Lifecycle
- Batch 39: Consumer Dispatch
- Batch 40: MQTT Server/JSA
- Batch 41: MQTT Client/IO

All plans include mandatory verification protocol and anti-stub guardrails.
Updated batches.md with file paths and planned status.
All 42 batches (0-41) now have design docs and implementation plans.
2026-02-27 17:27:51 -05:00

16 KiB
Raw Blame History

Batch 37 Stream Messages Implementation Plan

For Codex: REQUIRED SUB-SKILL: Use executeplan to implement this plan task-by-task.

Goal: Port and verify Batch 37 (Stream Messages) from server/stream.go so all 86 mapped features and 13 mapped tests are either genuinely implemented/verified or explicitly deferred with concrete blockers.

Architecture: Implement Batch 37 in five feature groups (18/16/16/17/19) using NatsStream partials plus message carrier helpers (InMsg, CMsg, JsPubMsg, JsOutQ) and account restore support. Execute three test waves with strict per-feature/per-test evidence loops and mandatory anti-stub gates before any status promotion.

Tech Stack: .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (porting.db)

Design doc: docs/plans/2026-02-27-batch-37-stream-messages-design.md

Batch 37 Scope

  • Batch ID: 37
  • Name: Stream Messages
  • Dependency: 36
  • Go source: golang/nats-server/server/stream.go
  • Features: 86 (3294-3387 mapped IDs)
  • Tests: 13

If dotnet is not on PATH, use:

DOTNET=/usr/local/share/dotnet/dotnet

Expected production files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs
  • Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs

Expected test files:

  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs
  • Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs
  • Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs

MANDATORY VERIFICATION PROTOCOL

NON-NEGOTIABLE: Every Batch 37 feature and test must pass this loop. No shortcut status updates.

Dependency Preflight (before any status change)

$DOTNET run --project tools/NatsNet.PortTracker -- batch show 36 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch show 37 --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- batch ready --db porting.db

Start only when Batch 37 is ready:

$DOTNET run --project tools/NatsNet.PortTracker -- batch start 37 --db porting.db

Capture baseline:

$DOTNET build dotnet/
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

What Counts as a Real Port

A feature is real only if all are true:

  1. Mapped method/type exists with non-placeholder behavior.
  2. Logic uses actual state/inputs (not hardcoded defaults).
  3. At least one behavioral test exercises that code path.
  4. No forbidden stub patterns are present.

A test is real only if all are true:

  1. Arrange/Act/Assert is present.
  2. It calls production code in ZB.MOM.NatsNet.Server.*.
  3. Assertions validate behavior from Act output/effects.
  4. It is not a template/literal stub.

Per-Feature Verification Loop (REQUIRED per feature ID)

  1. Inspect mapping and Go context:
$DOTNET run --project tools/NatsNet.PortTracker -- feature show <feature_id> --db porting.db
  1. Claim work:
$DOTNET run --project tools/NatsNet.PortTracker -- feature update <feature_id> --status stub --db porting.db
  1. Add or update focused behavioral test(s) for that feature.
  2. Implement minimal real logic for the mapped method/type.
  3. Run Stub Detection Check on changed files.
  4. Run Build Gate.
  5. Run targeted Test Gate (specific class/filter for touched area).
  6. Promote only proven feature IDs to complete (or verified only after checkpoint evidence).

Per-Test Verification Loop (REQUIRED per test ID)

  1. Inspect mapping and Go test location:
$DOTNET run --project tools/NatsNet.PortTracker -- test show <test_id> --db porting.db
  1. Claim work:
$DOTNET run --project tools/NatsNet.PortTracker -- test update <test_id> --status stub --db porting.db
  1. Port full behavior (no template assertions).
  2. Run single test method:
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~<ClassName>.<MethodName>" --verbosity normal
  1. Confirm Passed: 1, Failed: 0 (never Passed: 0).
  2. Run touched class filter.
  3. Run Stub Detection Check + Build Gate.
  4. Promote only proven test IDs to complete (or verified only after checkpoint evidence).

Stub Detection Check (REQUIRED after every feature/test loop)

changed_files=$(git diff --name-only -- dotnet/src/ZB.MOM.NatsNet.Server dotnet/tests/ZB.MOM.NatsNet.Server.Tests | rg "\\.cs$" || true)
if [ -n "$changed_files" ]; then
  echo "$changed_files" | xargs rg -n "(NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass\\(\\)|// TODO|// PLACEHOLDER|var goFile = \"server/|\\.ShouldContain\\(\"Should\"\\)|GetRequiredApiLevel\\(new Dictionary<string, string>\\)\\.ShouldBe\\(string\\.Empty\\)|=>\\s*default;|return\\s+default;|return\\s+null;\\s*$)"
fi

Any match blocks status promotion until fixed or deferred with reason.

Build Gate (REQUIRED)

$DOTNET build dotnet/

Run this:

  • after each feature loop
  • after each test loop
  • before every batch-update
  • at every task checkpoint

Test Gate (REQUIRED)

Minimum targeted gates by touched domain:

$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamEngineTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamClusterTests1|FullyQualifiedName~JetStreamClusterTests2"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~JetStreamFileStoreTests|FullyQualifiedName~RaftNodeTests"
$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --filter "FullyQualifiedName~GatewayHandlerTests|FullyQualifiedName~JwtProcessorTests|FullyQualifiedName~LeafNodeHandlerTests|FullyQualifiedName~ConcurrencyTests1"

Checkpoint/full gate:

$DOTNET test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

Status Update Protocol (HARD LIMIT: max 15 IDs per batch-update)

  • Never pass more than 15 IDs per feature batch-update or test batch-update.
  • Never update IDs outside current task scope.
  • Never promote to verified without checkpoint evidence.
  • For blocked items, keep deferred with explicit reason.

Use:

$DOTNET run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

$DOTNET run --project tools/NatsNet.PortTracker -- \
  test batch-update --ids "<max 15 ids>" --set-status <stub|complete|verified> --db porting.db --execute

Checkpoint Protocol Between Tasks (REQUIRED)

Before starting the next task:

  1. Run Stub Detection Check on current diff.
  2. Run Build Gate.
  3. Run targeted class test filters for touched classes.
  4. Run full unit test gate.
  5. Apply status updates in <=15 ID chunks only.
  6. Commit checkpoint before next task.

ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)

Forbidden Patterns

Forbidden in mapped Batch 37 feature/test work:

  • throw new NotImplementedException()
  • Empty mapped method bodies or no-op method shims
  • Placeholder comments (TODO, PLACEHOLDER, later)
  • Fake-pass assertions (Assert.True(true), Assert.Pass())
  • Template assertions that do not validate behavior:
    • var goFile = "server/..."
    • "<MethodName>".ShouldContain("Should")
    • GetRequiredApiLevel(new Dictionary<string, string>()).ShouldBe(string.Empty) as core assertion
  • Constant-return shortcuts in mapped methods without behavioral justification:
    • => default;
    • return default;
    • return null;
  • Catch-and-ignore patterns that hide failures

Hard Limits

  • Max feature group size: ~20 IDs.
  • Max IDs per feature batch-update: 15.
  • Max IDs per test batch-update: 15.
  • One active feature loop at a time.
  • One active test loop at a time.
  • Mandatory checkpoint between every task.
  • No verified promotion without full checkpoint evidence.

If You Get Stuck (REQUIRED)

  1. Stop on the current ID immediately.
  2. Do not write placeholder logic or fake assertions.
  3. Mark item deferred with concrete blocker.
  4. Continue with next unblocked item.

Feature deferral:

$DOTNET run --project tools/NatsNet.PortTracker -- \
  feature update <feature_id> --status deferred --override "blocked: <specific reason>" --db porting.db

Test deferral:

$DOTNET run --project tools/NatsNet.PortTracker -- \
  test update <test_id> --status deferred --override "blocked: <specific reason>" --db porting.db

deferred with explicit reason is correct. Stubbed progress is not.

Feature Groups (max ~20 each)

Group A (18): Dedupe and header extraction foundation

IDs:
3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3310,3311,3312

Group B (16): Scheduling/batch headers and direct-get ingress path

IDs:
3314,3315,3316,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330

Group C (16): Message carriers, publish out queue, and internal loop hooks

IDs:
3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3345,3346,3347

Group D (17): Consumer registry and interest-state operations

IDs:
3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3364,3366,3367,3368

Group E (19): Pre-ack, snapshot/restore, monitor/replication checks

IDs:
3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387

Test Waves

Wave T1 (5): direct-get and rollup/mirror semantics

IDs:
828,954,987,1642,1643

Wave T2 (4): snapshot/restore and raft recovery paths

IDs:
383,2617,2672,2695

Wave T3 (4): gateway/jwt/leaf/perf regression coverage

IDs:
635,1892,1961,2426

Task 1: Preflight, Baseline, and Batch Start

Files:

  • Read: docs/plans/2026-02-27-batch-37-stream-messages-design.md
  • Read: golang/nats-server/server/stream.go

Steps:

  1. Run dependency preflight and verify Batch 36 completion.
  2. Start Batch 37.
  3. Capture baseline build/test evidence.
  4. Do not change any status without baseline logs.

Task 2: Implement Feature Group A (18 IDs)

Files:

  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessageHeaders.cs

Steps:

  1. Move Group A IDs to stub in 15 + 3 chunks.
  2. Run Per-Feature Verification Loop for each ID.
  3. Run required stub/build/test gates per ID.
  4. Promote proven IDs in <=15 chunks.
  5. Run checkpoint protocol.

Task 3: Implement Feature Group B (16 IDs)

Files:

  • Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.DirectGet.cs
  • Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.MessagePipeline.cs

Steps:

  1. Move Group B IDs to stub in 15 + 1 chunks.
  2. Execute Per-Feature Verification Loop per ID.
  3. Run mandatory gates.
  4. Promote proven IDs in <=15 chunks.
  5. Run checkpoint protocol.

Task 4: Implement Feature Group C (16 IDs)

Files:

  • Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs
  • Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.cs
  • Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StreamTypes.MessageCarriers.cs

Steps:

  1. Move Group C IDs to stub in 15 + 1 chunks.
  2. Execute Per-Feature Verification Loop per ID.
  3. Ensure JsOutQ exists with mapped methods.
  4. Run mandatory gates.
  5. Promote proven IDs in <=15 chunks.
  6. Run checkpoint protocol.

Task 5: Implement Feature Group D (17 IDs)

Files:

  • Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.Consumers.cs

Steps:

  1. Move Group D IDs to stub in 15 + 2 chunks.
  2. Execute Per-Feature Verification Loop per ID.
  3. Run mandatory gates.
  4. Promote proven IDs in <=15 chunks.
  5. Run checkpoint protocol.

Task 6: Implement Feature Group E (19 IDs)

Files:

  • Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/NatsStream.SnapshotMonitor.cs
  • Modify/Create: dotnet/src/ZB.MOM.NatsNet.Server/Accounts/Account.StreamRestore.cs

Steps:

  1. Move Group E IDs to stub in 15 + 4 chunks.
  2. Execute Per-Feature Verification Loop per ID.
  3. Run mandatory gates.
  4. Promote proven IDs in <=15 chunks.
  5. Run checkpoint protocol.

Task 7: Port Test Wave T1 (5 IDs)

Files:

  • Modify/Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests1.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamClusterTests2.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamEngineTests.Impltests.cs

Steps:

  1. Move Wave T1 IDs to stub.
  2. Execute Per-Test Verification Loop per ID.
  3. Run mandatory stub/build/class/full test gates.
  4. Promote proven IDs in one <=15 batch.
  5. Run checkpoint protocol.

Task 8: Port Test Wave T2 (4 IDs)

Files:

  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs
  • Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/RaftNodeTests.Impltests.cs

Steps:

  1. Move Wave T2 IDs to stub.
  2. Execute Per-Test Verification Loop per ID.
  3. Run mandatory gates.
  4. Promote proven IDs in one <=15 batch.
  5. Run checkpoint protocol.

Task 9: Port Test Wave T3 (4 IDs)

Files:

  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/LeafNodeHandlerTests.Impltests.cs
  • Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs

Steps:

  1. Move Wave T3 IDs to stub.
  2. Execute Per-Test Verification Loop per ID.
  3. Run mandatory gates.
  4. Promote proven IDs in one <=15 batch.
  5. Run checkpoint protocol.

Task 10: Batch Closeout

Files:

  • Modify: porting.db
  • Modify: reports/current.md (via report script output)

Steps:

  1. Run final stub detection on current diff.
  2. Run final build gate and full test gate.
  3. Run dry-run audits for features and tests:
$DOTNET run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db
$DOTNET run --project tools/NatsNet.PortTracker -- audit --type tests --db porting.db
  1. Complete batch:
$DOTNET run --project tools/NatsNet.PortTracker -- batch complete 37 --db porting.db
  1. Regenerate summary report:
./reports/generate-report.sh

Plan complete and saved to docs/plans/2026-02-27-batch-37-stream-messages-implementation-plan.md. Two execution options:

  1. Subagent-Driven (this session) - I dispatch fresh subagent per task, review between tasks, fast iteration.
  2. Parallel Session (separate) - Open new session with executeplan, batch execution with checkpoints.

Which approach?

Reference in New Issue View Git Blame Copy Permalink