natsnet/docs/plans/2026-02-27-batch-13-filestore-read-query-plan.md

Batch 13 FileStore Read/Query Implementation Plan

For Codex: REQUIRED SUB-SKILL: Use executeplan to implement this plan task-by-task.

Goal: Implement and verify Batch 13 FileStore read/query helpers (1006,1007,1008,1009,1010,1011,1014,1015) from filestore.go without introducing placeholder logic.

Architecture: Port the helper methods into JetStreamFileStore with Go-equivalent lock and index behavior (_psim, _bim, _blks), add focused unit tests for helper-level correctness, and keep current public delegation stable until later batches wire these helpers into public query paths.

Tech Stack: .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (porting.db)

Design doc: docs/plans/2026-02-27-batch-13-filestore-read-query-design.md

---

Batch Facts

• Batch ID: 13
• Name: FileStore Read/Query
• Dependencies: Batch 12
• Features: 8
• Tests mapped in batch: 0
• Go source: golang/nats-server/server/filestore.go

Feature IDs:

• Group A: 1006,1007,1008
• Group B: 1009,1010,1011
• Group C: 1014,1015

---

MANDATORY VERIFICATION PROTOCOL

NON-NEGOTIABLE. Every feature task must follow this protocol.

Per-Feature Verification Loop (REQUIRED for every feature ID)

For each feature (1006 through 1015 in this batch):

1. Read the exact Go method body and nearby comments from golang/nats-server/server/filestore.go.
2. Add/adjust failing unit tests that assert the method behavior (or behavior exercised through a helper entry point).
3. Run the targeted test filter and verify it fails for the expected reason.
4. Implement minimal C# code to pass the failing test.
5. Run targeted tests again and confirm pass.
6. Run dotnet build dotnet/ and confirm build success.
7. Add feature ID to the evidence table only if steps 1-6 are complete.

Stub Detection Check (REQUIRED after each feature group)

Run stub scans on all files touched in the group before any status update:

grep -n -E "(NotImplementedException|TODO|PLACEHOLDER)" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs

grep -n -E "^\s*(public|private|internal|protected).*\{\s*\}$" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs

If any matches are found, fix code or defer the impacted feature. Do not mark complete/verified.

Build Gate (REQUIRED after each feature group)

dotnet build dotnet/

Must pass with 0 errors before feature status changes in that group.

Test Gate (REQUIRED before verified)

Run related tests after each group and before any verified update:

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests|FullyQualifiedName~JetStreamFileStoreTests|FullyQualifiedName~StorageEngineTests.FileStoreMultiLastSeqsAndLoadLastMsgWithLazySubjectState_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreAllLastSeqs_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreMultiLastSeqs_ShouldSucceed"

All selected tests must pass. Any failure blocks verified.

Status Update Protocol (REQUIRED)

• Maximum 15 IDs per feature batch-update call.
• Evidence required per ID: Go lines reviewed, targeted test names, pass output, build pass.
• Transition order: deferred/not_started -> stub -> complete -> verified.
• Never mark verified before both Build Gate and Test Gate pass.
• If audit disagrees, use --override "<specific evidence>" only with concrete reason.

Status update commands for this batch:

# Claim work
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1011,1014-1015" --set-status stub --db porting.db --execute

# Group completion (run per group with only that group's IDs)
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<group ids>" --set-status complete --db porting.db --execute

# Group verification (after test gate)
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<group ids>" --set-status verified --db porting.db --execute

Checkpoint Protocol (REQUIRED between tasks)

Between Task 2, Task 3, and Task 4:

1. Run full build:

   dotnet build dotnet/
   

2. Run full unit test project:

   dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
   

3. Confirm pass/fail/skip summary and capture it in notes.
4. Commit checkpoint before moving to next task.

---

ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)

These rules apply to both production features and tests in this plan.

Forbidden Patterns

Do not leave any of these in touched code:

• throw new NotImplementedException(...)
• // TODO or // PLACEHOLDER in implemented method bodies
• Empty method bodies (including one-line { } placeholders)
• Test methods with no assertions
• Tests that only assert non-null for complex query behavior

Hard Limits

• Max 15 feature IDs per status batch update.
• Max 1 feature group per status-update cycle.
• Mandatory build + related test gate before verified.
• Mandatory checkpoint commit between feature groups.

If You Get Stuck

If a feature cannot be completed without missing infrastructure:

1. Leave or move that feature to deferred (never ship a stub).
2. Record the blocker using --override with specific reason.
3. Continue with remaining features in the same group.
4. Do not force complete/verified to unblock the batch.

Example:

dotnet run --project tools/NatsNet.PortTracker -- \
  feature update <id> --status deferred --db porting.db \
  --override "blocked: requires <missing dependency> for non-stub implementation"

---

Task 1: Batch Setup and Baseline

Files:

• Modify: porting.db
• Read: golang/nats-server/server/filestore.go
• Read: docs/plans/2026-02-27-batch-13-filestore-read-query-design.md

Step 1: Confirm batch readiness

Run:

dotnet run --project tools/NatsNet.PortTracker -- batch show 13 --db porting.db

Expected: Batch 13 displayed with dependency on Batch 12.

Step 2: Start batch

Run:

dotnet run --project tools/NatsNet.PortTracker -- batch start 13 --db porting.db

Expected: Batch status changes to in-progress (or equivalent accepted state).

Step 3: Claim all 8 features as stub

Run:

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1011,1014-1015" --set-status stub --db porting.db --execute

Expected: All 8 features updated to stub.

Step 4: Run baseline build and unit tests

Run:

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/

Expected: Baseline captured before feature changes.

Step 5: Commit baseline checkpoint

git add porting.db docs/plans/2026-02-27-batch-13-filestore-read-query-design.md docs/plans/2026-02-27-batch-13-filestore-read-query-plan.md
git commit -m "plan(batch13): establish filestore read/query baseline and protocol"

---

Task 2: Group A Features (1006, 1007, 1008) Skip-First-Block Helpers

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs
• Create: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs
• Optional Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs (only if helper accessors are required)

Step 1: Read Go source slice for Group A

Read:

sed -n '3241,3307p' golang/nats-server/server/filestore.go

Expected: Exact behavior for checkSkipFirstBlock, checkSkipFirstBlockMulti, selectSkipFirstBlock.

Step 2: Write failing tests for block-selection behavior

Add tests covering:

• filter="" and filter=">" returns bi + 1.
• Literal filter with no psim match returns ErrStoreEOF.
• stop <= currentBlockIndex returns ErrStoreEOF.
• start > currentBlockIndex jumps to selected block index.
• Multi-filter path uses SimpleSublist intersection semantics.

Step 3: Run targeted tests to verify failure

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.CheckSkipFirstBlock|FullyQualifiedName~JetStreamFileStoreReadQueryTests.SelectSkipFirstBlock"

Expected: Failing tests indicating missing or incorrect behavior.

Step 4: Implement Group A methods in FileStore.cs

Implement logic equivalent to Go:

private (int Next, Exception? Error) SelectSkipFirstBlock(int bi, uint start, uint stop)
{
    var mbi = _blks[bi].Index;
    if (stop <= mbi)
        return (-1, StoreErrors.ErrStoreEOF);
    if (start > mbi && _bim.TryGetValue(start, out var mb) && mb != null)
    {
        var (ni, _) = SelectMsgBlockWithIndex(mb.Last.Seq);
        return (ni, null);
    }
    return (bi + 1, null);
}

Use the same boundary rules for CheckSkipFirstBlock and multi-filter variant.

Step 5: Run targeted tests to green

Run:

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.CheckSkipFirstBlock|FullyQualifiedName~JetStreamFileStoreReadQueryTests.SelectSkipFirstBlock"

Expected: Pass.

Step 6: Run Stub Detection Check, Build Gate, Test Gate

Run all commands from MANDATORY VERIFICATION PROTOCOL.

Step 7: Update statuses for Group A

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1008" --set-status complete --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1008" --set-status verified --db porting.db --execute

Expected: IDs 1006-1008 set to verified with recorded evidence.

Step 8: Run Checkpoint Protocol and commit

git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs \
  porting.db
git commit -m "feat(batch13): port filestore skip-first-block query helpers"

---

Task 3: Group B Features (1009, 1010, 1011) Filtered Pending Helpers

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs

Step 1: Read Go source slice for Group B

sed -n '3308,3503p' golang/nats-server/server/filestore.go

Expected: Behavior for numFilteredPending*, stale fblk correction, optional last-seq logic.

Step 2: Write failing tests for filtered pending

Add tests for:

• All-subject short-circuit (filter="" and filter=">").
• No-match returns zeroed SimpleState.
• Literal and wildcard totals from psim.
• NumFilteredPendingNoLast leaves Last == 0.
• Stale Psi.Fblk is corrected after first miss path.

Step 3: Run targeted tests to verify failure

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.NumFilteredPending"

Expected: Fail until implementation is in place.

Step 4: Implement Group B methods

Port Go behavior:

• NumFilteredPending delegates to with-last variant (includeLast=true).
• NumFilteredPendingNoLast delegates with includeLast=false.
• NumFilteredPendingWithLast:
  • Calculates totals from _psim.
  • Resolves first via start block then forward scan.
  • Schedules Psi.Fblk correction under write lock if stale.
  • Computes last only when requested.

Step 5: Run targeted + related tests

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.NumFilteredPending|FullyQualifiedName~JetStreamFileStoreTests"

Expected: Pass.

Step 6: Run Stub Detection Check, Build Gate, Test Gate

Run all commands from MANDATORY VERIFICATION PROTOCOL.

Step 7: Update statuses for Group B

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1009-1011" --set-status complete --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1009-1011" --set-status verified --db porting.db --execute

Step 8: Run Checkpoint Protocol and commit

git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs \
  porting.db
git commit -m "feat(batch13): port filestore filtered-pending query helpers"

---

Task 4: Group C Features (1014, 1015) Last-Sequence and Filter-All Helpers

Files:

• Modify: dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs
• Modify: dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs

Step 1: Read Go source slice for Group C

sed -n '3504,3571p' golang/nats-server/server/filestore.go

Expected: allLastSeqsLocked and filterIsAll behavior details.

Step 2: Write failing tests for Group C

Add tests for:

• AllLastSeqsLocked returns sorted last sequence list.
• No messages or no subject tracking returns empty/nil equivalent.
• Lazy lastNeedsUpdate path recalculates before collecting sequence.
• FilterIsAll true for reordered equivalent subject set.
• FilterIsAll false for count mismatch or non-subset match.

Step 3: Run targeted tests to verify failure

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.AllLastSeqsLocked|FullyQualifiedName~JetStreamFileStoreReadQueryTests.FilterIsAll"

Expected: Fail until implementation is complete.

Step 4: Implement Group C methods

Port behavior:

• Reverse block walk collecting per-subject last sequence once.
• Keep subs set to avoid duplicates.
• Sort final sequence array before return.
• FilterIsAll compares sorted filters to sorted configured subjects using subset-match check.

Step 5: Run targeted + regression tests

dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.AllLastSeqsLocked|FullyQualifiedName~JetStreamFileStoreReadQueryTests.FilterIsAll|FullyQualifiedName~StorageEngineTests.FileStoreMultiLastSeqsAndLoadLastMsgWithLazySubjectState_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreAllLastSeqs_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreMultiLastSeqs_ShouldSucceed"

Expected: Pass.

Step 6: Run Stub Detection Check, Build Gate, Test Gate

Run all commands from MANDATORY VERIFICATION PROTOCOL.

Step 7: Update statuses for Group C

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1014-1015" --set-status complete --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1014-1015" --set-status verified --db porting.db --execute

Step 8: Run Checkpoint Protocol and commit

git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs \
  porting.db
git commit -m "feat(batch13): port filestore all-last-seqs and filter-is-all helpers"

---

Task 5: Batch Completion and Handoff

Files:

• Modify: porting.db
• Optional Modify: reports/current.md (if report script updates it)

Step 1: Full verification sweep

dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.IntegrationTests/

Expected: Green build and tests (or clearly documented pre-existing failures).

Step 2: Audit features

dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db

Expected: Batch 13 IDs classified consistently with implementation.

Step 3: Complete batch

dotnet run --project tools/NatsNet.PortTracker -- batch complete 13 --db porting.db

Expected: Batch 13 completes if all required statuses are valid.

Step 4: Refresh report and inspect newly unblocked work

./reports/generate-report.sh
dotnet run --project tools/NatsNet.PortTracker -- dependency ready --db porting.db

Expected: Updated report and visibility into next available batches.

---

Evidence Log Template (Use During Execution)

For each feature ID, capture:

• Go reference: file + line range reviewed.
• Test cases added/updated.
• Targeted test command + pass summary.
• Build command + pass summary.
• Stub scan output status.
• PortTracker status command(s) executed.

This evidence is required before any verified transition.