natsnet/docs/plans/2026-02-27-batch-15-msgblock-consumerfilestore-implementation-plan.md

Batch 15 MsgBlock + ConsumerFileStore Implementation Plan

For Codex: REQUIRED SUB-SKILL: Use executeplan to implement this plan task-by-task.

Goal: Implement and verify Batch 15 (MsgBlock + ConsumerFileStore) with real MessageBlock/ConsumerFileStore behavior parity from server/filestore.go, then close feature/test statuses with evidence-backed updates.

Architecture: Execute seven feature groups (max 20 IDs each) and five test waves in a strict vertical loop: read Go intent, implement minimal real C#, build, run related tests, and only then update statuses. Apply mandatory anti-stub checks and checkpoint gates between tasks to prevent false-positive progress.

Tech Stack: .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (porting.db)

---

I'm using writeplan to create the implementation plan.

Design doc: docs/plans/2026-02-27-batch-15-msgblock-consumerfilestore-design.md

Batch Inputs

• Batch: 15 (MsgBlock + ConsumerFileStore)
• Dependency: Batch 14
• Features: 121 (all currently deferred)
• Tests: 89 (all currently deferred)
• Go source: golang/nats-server/server/filestore.go

Feature groups (max ~20 each):

• Group 1 (20): 951,952,953,954,971,973,977,981,982,983,984,985,986,994,1000,1001,1002,1003,1004,1025
• Group 2 (20): 1026,1032,1049,1050,1051,1052,1053,1055,1056,1058,1059,1060,1061,1062,1063,1064,1065,1067,1068,1069
• Group 3 (20): 1070,1071,1072,1073,1074,1075,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1096,1097,1098,1103
• Group 4 (20): 1104,1105,1106,1107,1112,1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1123,1124,1125,1126,1127
• Group 5 (20): 1128,1129,1130,1134,1135,1136,1156,1157,1159,1173,1174,1175,1176,1185,1186,1187,1188,1190,1191,1192
• Group 6 (20): 1194,1218,1219,1220,1221,1222,1223,1231,1234,1236,1237,1238,1240,1241,1245,1246,1247,1250,1253,1258
• Group 7 (1): 1259

Test waves:

• Wave 1 (20, filestore core): 368,373,382,394,399,403,404,405,407,411,416,428,445,448,449,451,458,459,460,472
• Wave 2 (20, filestore compaction/recovery): 473,474,478,479,482,486,497,499,500,502,527,533,534,535,544,546,547,550,555,557
• Wave 3 (18, filestore tombstone/index edges): 559,562,570,571,573,574,577,578,579,580,581,582,583,584,591,595,596,597
• Wave 4 (7, concurrency + cross-module smoke dependencies): 2422,2505,2510,333,603,604,2762
• Wave 5 (24, JWT-linked tests likely dependency-sensitive): 1809,1810,1811,1812,1813,1814,1815,1816,1817,1818,1820,1824,1826,1827,1828,1829,1842,1843,1844,1845,1848,1850,1851,1852

Primary files in scope:

• dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs
• dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs
• dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs
• dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs
• dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs
• dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs
• dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs
• dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs
• dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs
• dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs
• dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs

---

MANDATORY VERIFICATION PROTOCOL

NON-NEGOTIABLE: Every feature/test transition and status update in this plan must satisfy all gates below.

Per-Feature Verification Loop (required for every feature ID)

1. Read feature mapping:

   /usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- feature show <FEATURE_ID> --db porting.db
   

2. Read Go source at mapped location in golang/nats-server/server/filestore.go (include nearby helpers and lock assumptions).
3. Implement the C# port in the mapped .NET file with real behavior (no placeholder logic).
4. Build immediately:

   /usr/local/share/dotnet/dotnet build dotnet/
   

5. Run related tests (single method or narrow class filter).
6. Only after green build + related tests, add ID to complete/verified candidate list.

Stub Detection Check (required after each feature group and each test wave)

Run all scans before any status update:

# Production placeholder markers
grep -R -n -E "NotImplementedException|TODO|PLACEHOLDER" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream

# Empty/suspicious method bodies on core files
grep -R -n -E "^[[:space:]]*(public|private|internal|protected).*[)]\s*\{\s*\}$|=>\s*default;|=>\s*null;" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs

# Test placeholders/always-pass patterns
grep -R -n -E "NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass|// TODO|// PLACEHOLDER|ShouldBe\\(true\\);" \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog

Any hit must be fixed or intentionally deferred with explicit reason before continuing.

Build Gate (required after each feature group)

After each feature group, before status updates:

/usr/local/share/dotnet/dotnet build dotnet/

Build must complete with zero errors.

Test Gate (required before marking features verified)

Before moving any feature IDs to verified, all related tests in the active wave/class set must pass.

Minimum required commands:

/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.JetStreamFileStoreTests" \
  --verbosity normal

/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.ConcurrencyTests2|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.EventsHandlerTests|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.GatewayHandlerTests|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.ConfigReloaderTests|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.JwtProcessorTests" \
  --verbosity normal

Status Update Protocol (required)

• Max 15 IDs per feature batch-update or test batch-update command.
• Required evidence per update batch:
  • Go source reviewed for each ID.
  • Build gate passed.
  • Related test gate passed.
  • Stub detection check clean.
• Keep evidence logs in /tmp/batch15-evidence/.
• Never mark IDs verified without recorded evidence.

Example command pattern:

/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "951,952,953,954,971" --set-status complete --db porting.db --execute

Checkpoint Protocol Between Tasks (required)

After every feature group + test wave pair:

1. Full build:

   /usr/local/share/dotnet/dotnet build dotnet/
   

2. Full unit test run:

   /usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --verbosity normal
   

3. Commit checkpoint:

   git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream \
     dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog \
     porting.db
   git commit -m "feat(batch15): complete group <N> msgblock/consumerfilestore"
   

---

ANTI-STUB GUARDRAILS

Forbidden Patterns

The following are forbidden in Batch 15 scope (production or tests):

• throw new NotImplementedException(...)
• Empty mapped method bodies ({ }) for mapped APIs
• TODO/PLACEHOLDER markers in implemented methods/tests
• Assert.True(true) or equivalent always-pass assertions
• Test methods with no production act step
• “Name-only” or constant-only assertions that do not verify behavior

Hard Limits

• Max ~20 features per implementation group (fixed at 20/20/20/20/20/20/1)
• Max 15 IDs per status update command
• Max one feature group per status-update cycle
• Zero stub-scan hits before complete or verified
• No feature may move to verified before related tests pass
• Mandatory checkpoint commit between tasks

If You Get Stuck (mandatory behavior)

1. Do not stub and do not fake-pass tests.
2. Keep blocked IDs deferred.
3. Record precise blocker reason (infra missing, dependency batch not complete, non-batch surface required, etc.).
4. Continue with unblocked IDs in the same group.

Example:

/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \
  feature update <ID> --status deferred --db porting.db \
  --override "blocked: requires <specific dependency/infra> for non-stub implementation"

Use same pattern for tests.

---

Task 1: Preflight Dependency Gate and Evidence Setup

Files:

• Modify: porting.db
• Create: /tmp/batch15-evidence/

Step 1: Confirm Batch 14 dependency and batch state

/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 15 --db porting.db
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db

Expected: Batch 15 not startable until Batch 14 is complete.

Step 2: Create evidence folder

mkdir -p /tmp/batch15-evidence

Step 3: Start Batch 15 only after dependency is satisfied

/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch start 15 --db porting.db

Task 2: Implement Feature Group 1 (20 IDs)

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs
• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs
• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs

Step 1: Port IDs 951,952,953,954,971,973,977,981,982,983

Step 2: Port IDs 984,985,986,994,1000,1001,1002,1003,1004,1025

Step 3: Apply mandatory verification protocol (per-feature loop + stub detection + build gate).

Step 4: Update Group 1 feature statuses in chunks <=15 (stub -> complete).

Task 3: Port and Verify Test Wave 1 (20 IDs)

Files:

• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs

Step 1: Port first 10 tests from Wave 1; run individual-method verification.

Step 2: Port second 10 tests; run individual-method verification.

Step 3: Run class-level JetStreamFileStoreTests gate + stub detection.

Step 4: Mark passing Wave 1 tests verified in chunks <=15.

Task 4: Group 1 Checkpoint

Files:

• Modify: porting.db

Step 1: Run Checkpoint Protocol (build + full unit tests + commit).

Step 2: Promote Group 1 features to verified only if Wave 1 gate is green.

Task 5: Implement Feature Group 2 (20 IDs)

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs

Step 1: Port IDs 1026,1032,1049,1050,1051,1052,1053,1055,1056,1058.

Step 2: Port IDs 1059,1060,1061,1062,1063,1064,1065,1067,1068,1069.

Step 3: Run per-feature verification loop and group build gate.

Step 4: Update statuses in chunks <=15.

Task 6: Port and Verify Test Wave 2 (20 IDs)

Files:

• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs

Step 1: Port first 10 tests in Wave 2.

Step 2: Port second 10 tests in Wave 2.

Step 3: Run class gate + stub detection + assertion depth review.

Step 4: Mark verified tests (<=15 IDs per command).

Task 7: Group 2 Checkpoint

Files:

• Modify: porting.db

Step 1: Run Checkpoint Protocol.

Step 2: Promote Group 2 features to verified if Wave 2 tests are green.

Task 8: Implement Feature Group 3 (20 IDs)

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs

Step 1: Port IDs 1070,1071,1072,1073,1074,1075,1086,1087,1088,1089.

Step 2: Port IDs 1090,1091,1092,1093,1094,1095,1096,1097,1098,1103.

Step 3: Apply mandatory verification protocol.

Step 4: Update statuses (<=15 IDs per update).

Task 9: Port and Verify Test Wave 3 (18 IDs)

Files:

• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs

Step 1: Port first 9 tests in Wave 3.

Step 2: Port last 9 tests in Wave 3.

Step 3: Run class gate + stub detection.

Step 4: Update test statuses for passing IDs.

Task 10: Group 3 Checkpoint

Files:

• Modify: porting.db

Step 1: Run Checkpoint Protocol.

Step 2: Promote Group 3 features to verified after Wave 3 gate.

Task 11: Implement Feature Group 4 (20 IDs)

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs
• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs

Step 1: Port IDs 1104,1105,1106,1107,1112,1113,1114,1115,1116,1117.

Step 2: Port IDs 1118,1119,1120,1121,1122,1123,1124,1125,1126,1127.

Step 3: Run per-feature loop and group build/test gates.

Step 4: Update feature statuses in <=15 chunks.

Task 12: Port and Verify Test Wave 4 (7 IDs)

Files:

• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs

Step 1: Port concurrency-focused tests (2422,2505,2510) and run each 3 times for flake check.

Step 2: Attempt 333,603,604,2762; if blocked by non-Batch-15 dependencies, keep deferred with reasons.

Step 3: Run cross-class test gate and status updates with evidence.

Task 13: Group 4 Checkpoint

Files:

• Modify: porting.db

Step 1: Run Checkpoint Protocol.

Step 2: Promote Group 4 features to verified where gates passed.

Task 14: Implement Feature Group 5 (20 IDs)

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs

Step 1: Port IDs 1128,1129,1130,1134,1135,1136,1156,1157,1159,1173.

Step 2: Port IDs 1174,1175,1176,1185,1186,1187,1188,1190,1191,1192.

Step 3: Run mandatory verification protocol and update status in <=15 chunks.

Task 15: Port and Evaluate Test Wave 5 (24 JWT-linked IDs)

Files:

• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs

Step 1: For each test ID, verify whether dependency feature(s) are in Batch 15 scope.

Step 2: Port only tests backed by implemented dependencies.

Step 3: For blocked tests, keep deferred with explicit reason (dependency outside Batch 15).

Step 4: Run class gate and update only evidenced IDs.

Task 16: Group 5 Checkpoint

Files:

• Modify: porting.db

Step 1: Run Checkpoint Protocol.

Step 2: Promote Group 5 features to verified only with passing related test coverage.

Task 17: Implement Feature Group 6 (20 IDs)

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs
• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs

Step 1: Port IDs 1194,1218,1219,1220,1221,1222,1223,1231,1234,1236.

Step 2: Port IDs 1237,1238,1240,1241,1245,1246,1247,1250,1253,1258.

Step 3: Run per-feature loop, group build gate, and related test gate.

Step 4: Update statuses in <=15 chunks.

Task 18: ConsumerFileStore Regression Sweep

Files:

• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs

Step 1: Add/adjust tests that specifically exercise consumer state flush, encryption, and delete/flush coordination.

Step 2: Run narrow filters for consumer-focused tests and then class-level gates.

Step 3: Update remaining Group 6 feature/test statuses with evidence.

Task 19: Group 6 Checkpoint

Files:

• Modify: porting.db

Step 1: Run Checkpoint Protocol.

Step 2: Commit Group 6 completion.

Task 20: Implement Group 7 + Final Batch Closure

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs
• Modify: porting.db
• Modify: reports/current.md

Step 1: Implement feature 1259 (StoreCompression.Decompress) with full per-feature verification loop.

Step 2: Run mandatory stub detection checks across full batch scope.

Step 3: Run final full build and full unit test suite.

Step 4: Perform status cleanup (complete -> verified) in <=15-ID chunks with evidence.

Step 5: Attempt batch closure and generate report.

/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch complete 15 --db porting.db
./reports/generate-report.sh

Step 6: Final commit.

git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog \
  porting.db reports/current.md

git commit -m "feat(batch15): complete msgblock and consumer file store port"