natsnet/docs/plans/2026-02-27-batch-15-msgblock-consumerfilestore-implementation-plan.md

# Batch 15 MsgBlock + ConsumerFileStore Implementation Plan

> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.

**Goal:** Implement and verify Batch 15 (`MsgBlock + ConsumerFileStore`) with real `MessageBlock`/`ConsumerFileStore` behavior parity from `server/filestore.go`, then close feature/test statuses with evidence-backed updates.

**Architecture:** Execute seven feature groups (max 20 IDs each) and five test waves in a strict vertical loop: read Go intent, implement minimal real C#, build, run related tests, and only then update statuses. Apply mandatory anti-stub checks and checkpoint gates between tasks to prevent false-positive progress.

**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)

---

I'm using `writeplan` to create the implementation plan.

**Design doc:** `docs/plans/2026-02-27-batch-15-msgblock-consumerfilestore-design.md`

## Batch Inputs

- Batch: `15` (`MsgBlock + ConsumerFileStore`)
- Dependency: Batch `14`
- Features: `121` (all currently `deferred`)
- Tests: `89` (all currently `deferred`)
- Go source: `golang/nats-server/server/filestore.go`

Feature groups (max ~20 each):

- Group 1 (`20`): `951,952,953,954,971,973,977,981,982,983,984,985,986,994,1000,1001,1002,1003,1004,1025`
- Group 2 (`20`): `1026,1032,1049,1050,1051,1052,1053,1055,1056,1058,1059,1060,1061,1062,1063,1064,1065,1067,1068,1069`
- Group 3 (`20`): `1070,1071,1072,1073,1074,1075,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1096,1097,1098,1103`
- Group 4 (`20`): `1104,1105,1106,1107,1112,1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1123,1124,1125,1126,1127`
- Group 5 (`20`): `1128,1129,1130,1134,1135,1136,1156,1157,1159,1173,1174,1175,1176,1185,1186,1187,1188,1190,1191,1192`
- Group 6 (`20`): `1194,1218,1219,1220,1221,1222,1223,1231,1234,1236,1237,1238,1240,1241,1245,1246,1247,1250,1253,1258`
- Group 7 (`1`): `1259`

Test waves:

- Wave 1 (`20`, filestore core): `368,373,382,394,399,403,404,405,407,411,416,428,445,448,449,451,458,459,460,472`
- Wave 2 (`20`, filestore compaction/recovery): `473,474,478,479,482,486,497,499,500,502,527,533,534,535,544,546,547,550,555,557`
- Wave 3 (`18`, filestore tombstone/index edges): `559,562,570,571,573,574,577,578,579,580,581,582,583,584,591,595,596,597`
- Wave 4 (`7`, concurrency + cross-module smoke dependencies): `2422,2505,2510,333,603,604,2762`
- Wave 5 (`24`, JWT-linked tests likely dependency-sensitive): `1809,1810,1811,1812,1813,1814,1815,1816,1817,1818,1820,1824,1826,1827,1828,1829,1842,1843,1844,1845,1848,1850,1851,1852`

Primary files in scope:

- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs`
- `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`

---

## MANDATORY VERIFICATION PROTOCOL

> **NON-NEGOTIABLE:** Every feature/test transition and status update in this plan must satisfy all gates below.

### Per-Feature Verification Loop (required for every feature ID)

1. Read feature mapping:
   ```bash
   /usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- feature show <FEATURE_ID> --db porting.db
   ```
2. Read Go source at mapped location in `golang/nats-server/server/filestore.go` (include nearby helpers and lock assumptions).
3. Implement the C# port in the mapped .NET file with real behavior (no placeholder logic).
4. Build immediately:
   ```bash
   /usr/local/share/dotnet/dotnet build dotnet/
   ```
5. Run related tests (single method or narrow class filter).
6. Only after green build + related tests, add ID to `complete`/`verified` candidate list.

### Stub Detection Check (required after each feature group and each test wave)

Run all scans before any status update:

```bash
# Production placeholder markers
grep -R -n -E "NotImplementedException|TODO|PLACEHOLDER" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream

# Empty/suspicious method bodies on core files
grep -R -n -E "^[[:space:]]*(public|private|internal|protected).*[)]\s*\{\s*\}$|=>\s*default;|=>\s*null;" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs

# Test placeholders/always-pass patterns
grep -R -n -E "NotImplementedException|Assert\\.True\\(true\\)|Assert\\.Pass|// TODO|// PLACEHOLDER|ShouldBe\\(true\\);" \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog
```

Any hit must be fixed or intentionally deferred with explicit reason before continuing.

### Build Gate (required after each feature group)

After each feature group, before status updates:

```bash
/usr/local/share/dotnet/dotnet build dotnet/
```

Build must complete with zero errors.

### Test Gate (required before marking features verified)

Before moving any feature IDs to `verified`, all related tests in the active wave/class set must pass.

Minimum required commands:

```bash
/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.JetStreamFileStoreTests" \
  --verbosity normal

/usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.ConcurrencyTests1|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.ConcurrencyTests2|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.EventsHandlerTests|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.GatewayHandlerTests|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.ConfigReloaderTests|FullyQualifiedName~ZB.MOM.NatsNet.Server.Tests.ImplBacklog.JwtProcessorTests" \
  --verbosity normal
```

### Status Update Protocol (required)

- Max `15` IDs per `feature batch-update` or `test batch-update` command.
- Required evidence per update batch:
  - Go source reviewed for each ID.
  - Build gate passed.
  - Related test gate passed.
  - Stub detection check clean.
- Keep evidence logs in `/tmp/batch15-evidence/`.
- Never mark IDs `verified` without recorded evidence.

Example command pattern:

```bash
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "951,952,953,954,971" --set-status complete --db porting.db --execute
```

### Checkpoint Protocol Between Tasks (required)

After every feature group + test wave pair:

1. Full build:
   ```bash
   /usr/local/share/dotnet/dotnet build dotnet/
   ```
2. Full unit test run:
   ```bash
   /usr/local/share/dotnet/dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ --verbosity normal
   ```
3. Commit checkpoint:
   ```bash
   git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream \
     dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog \
     porting.db
   git commit -m "feat(batch15): complete group <N> msgblock/consumerfilestore"
   ```

---

## ANTI-STUB GUARDRAILS

### Forbidden Patterns

The following are forbidden in Batch 15 scope (production or tests):

- `throw new NotImplementedException(...)`
- Empty mapped method bodies (`{ }`) for mapped APIs
- `TODO`/`PLACEHOLDER` markers in implemented methods/tests
- `Assert.True(true)` or equivalent always-pass assertions
- Test methods with no production act step
- “Name-only” or constant-only assertions that do not verify behavior

### Hard Limits

- Max ~20 features per implementation group (fixed at `20/20/20/20/20/20/1`)
- Max `15` IDs per status update command
- Max one feature group per status-update cycle
- Zero stub-scan hits before `complete` or `verified`
- No feature may move to `verified` before related tests pass
- Mandatory checkpoint commit between tasks

### If You Get Stuck (mandatory behavior)

1. Do not stub and do not fake-pass tests.
2. Keep blocked IDs `deferred`.
3. Record precise blocker reason (`infra missing`, `dependency batch not complete`, `non-batch surface required`, etc.).
4. Continue with unblocked IDs in the same group.

Example:

```bash
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- \
  feature update <ID> --status deferred --db porting.db \
  --override "blocked: requires <specific dependency/infra> for non-stub implementation"
```

Use same pattern for tests.

---

### Task 1: Preflight Dependency Gate and Evidence Setup

**Files:**
- Modify: `porting.db`
- Create: `/tmp/batch15-evidence/`

**Step 1: Confirm Batch 14 dependency and batch state**

```bash
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch show 15 --db porting.db
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch ready --db porting.db
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- report summary --db porting.db
```

Expected: Batch 15 not startable until Batch 14 is complete.

**Step 2: Create evidence folder**

```bash
mkdir -p /tmp/batch15-evidence
```

**Step 3: Start Batch 15 only after dependency is satisfied**

```bash
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch start 15 --db porting.db
```

### Task 2: Implement Feature Group 1 (20 IDs)

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs`

**Step 1:** Port IDs `951,952,953,954,971,973,977,981,982,983`

**Step 2:** Port IDs `984,985,986,994,1000,1001,1002,1003,1004,1025`

**Step 3:** Apply mandatory verification protocol (per-feature loop + stub detection + build gate).

**Step 4:** Update Group 1 feature statuses in chunks <=15 (`stub -> complete`).

### Task 3: Port and Verify Test Wave 1 (20 IDs)

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`

**Step 1:** Port first 10 tests from Wave 1; run individual-method verification.

**Step 2:** Port second 10 tests; run individual-method verification.

**Step 3:** Run class-level `JetStreamFileStoreTests` gate + stub detection.

**Step 4:** Mark passing Wave 1 tests `verified` in chunks <=15.

### Task 4: Group 1 Checkpoint

**Files:**
- Modify: `porting.db`

**Step 1:** Run Checkpoint Protocol (`build + full unit tests + commit`).

**Step 2:** Promote Group 1 features to `verified` only if Wave 1 gate is green.

### Task 5: Implement Feature Group 2 (20 IDs)

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`

**Step 1:** Port IDs `1026,1032,1049,1050,1051,1052,1053,1055,1056,1058`.

**Step 2:** Port IDs `1059,1060,1061,1062,1063,1064,1065,1067,1068,1069`.

**Step 3:** Run per-feature verification loop and group build gate.

**Step 4:** Update statuses in chunks <=15.

### Task 6: Port and Verify Test Wave 2 (20 IDs)

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`

**Step 1:** Port first 10 tests in Wave 2.

**Step 2:** Port second 10 tests in Wave 2.

**Step 3:** Run class gate + stub detection + assertion depth review.

**Step 4:** Mark verified tests (<=15 IDs per command).

### Task 7: Group 2 Checkpoint

**Files:**
- Modify: `porting.db`

**Step 1:** Run Checkpoint Protocol.

**Step 2:** Promote Group 2 features to `verified` if Wave 2 tests are green.

### Task 8: Implement Feature Group 3 (20 IDs)

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`

**Step 1:** Port IDs `1070,1071,1072,1073,1074,1075,1086,1087,1088,1089`.

**Step 2:** Port IDs `1090,1091,1092,1093,1094,1095,1096,1097,1098,1103`.

**Step 3:** Apply mandatory verification protocol.

**Step 4:** Update statuses (<=15 IDs per update).

### Task 9: Port and Verify Test Wave 3 (18 IDs)

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`

**Step 1:** Port first 9 tests in Wave 3.

**Step 2:** Port last 9 tests in Wave 3.

**Step 3:** Run class gate + stub detection.

**Step 4:** Update test statuses for passing IDs.

### Task 10: Group 3 Checkpoint

**Files:**
- Modify: `porting.db`

**Step 1:** Run Checkpoint Protocol.

**Step 2:** Promote Group 3 features to `verified` after Wave 3 gate.

### Task 11: Implement Feature Group 4 (20 IDs)

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs`

**Step 1:** Port IDs `1104,1105,1106,1107,1112,1113,1114,1115,1116,1117`.

**Step 2:** Port IDs `1118,1119,1120,1121,1122,1123,1124,1125,1126,1127`.

**Step 3:** Run per-feature loop and group build/test gates.

**Step 4:** Update feature statuses in <=15 chunks.

### Task 12: Port and Verify Test Wave 4 (7 IDs)

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/EventsHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/GatewayHandlerTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConfigReloaderTests.Impltests.cs`

**Step 1:** Port concurrency-focused tests (`2422,2505,2510`) and run each 3 times for flake check.

**Step 2:** Attempt `333,603,604,2762`; if blocked by non-Batch-15 dependencies, keep deferred with reasons.

**Step 3:** Run cross-class test gate and status updates with evidence.

### Task 13: Group 4 Checkpoint

**Files:**
- Modify: `porting.db`

**Step 1:** Run Checkpoint Protocol.

**Step 2:** Promote Group 4 features to `verified` where gates passed.

### Task 14: Implement Feature Group 5 (20 IDs)

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`

**Step 1:** Port IDs `1128,1129,1130,1134,1135,1136,1156,1157,1159,1173`.

**Step 2:** Port IDs `1174,1175,1176,1185,1186,1187,1188,1190,1191,1192`.

**Step 3:** Run mandatory verification protocol and update status in <=15 chunks.

### Task 15: Port and Evaluate Test Wave 5 (24 JWT-linked IDs)

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JwtProcessorTests.Impltests.cs`

**Step 1:** For each test ID, verify whether dependency feature(s) are in Batch 15 scope.

**Step 2:** Port only tests backed by implemented dependencies.

**Step 3:** For blocked tests, keep `deferred` with explicit reason (dependency outside Batch 15).

**Step 4:** Run class gate and update only evidenced IDs.

### Task 16: Group 5 Checkpoint

**Files:**
- Modify: `porting.db`

**Step 1:** Run Checkpoint Protocol.

**Step 2:** Promote Group 5 features to `verified` only with passing related test coverage.

### Task 17: Implement Feature Group 6 (20 IDs)

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs`

**Step 1:** Port IDs `1194,1218,1219,1220,1221,1222,1223,1231,1234,1236`.

**Step 2:** Port IDs `1237,1238,1240,1241,1245,1246,1247,1250,1253,1258`.

**Step 3:** Run per-feature loop, group build gate, and related test gate.

**Step 4:** Update statuses in <=15 chunks.

### Task 18: ConsumerFileStore Regression Sweep

**Files:**
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`

**Step 1:** Add/adjust tests that specifically exercise consumer state flush, encryption, and delete/flush coordination.

**Step 2:** Run narrow filters for consumer-focused tests and then class-level gates.

**Step 3:** Update remaining Group 6 feature/test statuses with evidence.

### Task 19: Group 6 Checkpoint

**Files:**
- Modify: `porting.db`

**Step 1:** Run Checkpoint Protocol.

**Step 2:** Commit Group 6 completion.

### Task 20: Implement Group 7 + Final Batch Closure

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs`
- Modify: `porting.db`
- Modify: `reports/current.md`

**Step 1:** Implement feature `1259` (`StoreCompression.Decompress`) with full per-feature verification loop.

**Step 2:** Run mandatory stub detection checks across full batch scope.

**Step 3:** Run final full build and full unit test suite.

**Step 4:** Perform status cleanup (`complete -> verified`) in <=15-ID chunks with evidence.

**Step 5:** Attempt batch closure and generate report.

```bash
/usr/local/share/dotnet/dotnet run --project tools/NatsNet.PortTracker -- batch complete 15 --db porting.db
./reports/generate-report.sh
```

**Step 6:** Final commit.

```bash
git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog \
  porting.db reports/current.md

git commit -m "feat(batch15): complete msgblock and consumer file store port"
```