natsnet/docs/plans/2026-02-27-batch-13-filestore-read-query-plan.md

# Batch 13 FileStore Read/Query Implementation Plan

> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.

**Goal:** Implement and verify Batch 13 FileStore read/query helpers (`1006,1007,1008,1009,1010,1011,1014,1015`) from `filestore.go` without introducing placeholder logic.

**Architecture:** Port the helper methods into `JetStreamFileStore` with Go-equivalent lock and index behavior (`_psim`, `_bim`, `_blks`), add focused unit tests for helper-level correctness, and keep current public delegation stable until later batches wire these helpers into public query paths.

**Tech Stack:** .NET 10, C# latest, xUnit 3, Shouldly, NSubstitute, PortTracker CLI, SQLite (`porting.db`)

**Design doc:** `docs/plans/2026-02-27-batch-13-filestore-read-query-design.md`

---

## Batch Facts

- Batch ID: `13`
- Name: `FileStore Read/Query`
- Dependencies: Batch `12`
- Features: `8`
- Tests mapped in batch: `0`
- Go source: `golang/nats-server/server/filestore.go`

Feature IDs:

- Group A: `1006,1007,1008`
- Group B: `1009,1010,1011`
- Group C: `1014,1015`

---

## MANDATORY VERIFICATION PROTOCOL

> **NON-NEGOTIABLE. Every feature task must follow this protocol.**

### Per-Feature Verification Loop (REQUIRED for every feature ID)

For each feature (`1006` through `1015` in this batch):

1. Read the exact Go method body and nearby comments from `golang/nats-server/server/filestore.go`.
2. Add/adjust failing unit tests that assert the method behavior (or behavior exercised through a helper entry point).
3. Run the targeted test filter and verify it fails for the expected reason.
4. Implement minimal C# code to pass the failing test.
5. Run targeted tests again and confirm pass.
6. Run `dotnet build dotnet/` and confirm build success.
7. Add feature ID to the evidence table only if steps 1-6 are complete.

### Stub Detection Check (REQUIRED after each feature group)

Run stub scans on all files touched in the group before any status update:

```bash
grep -n -E "(NotImplementedException|TODO|PLACEHOLDER)" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs

grep -n -E "^\s*(public|private|internal|protected).*\{\s*\}$" \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs
```

If any matches are found, fix code or defer the impacted feature. Do not mark complete/verified.

### Build Gate (REQUIRED after each feature group)

```bash
dotnet build dotnet/
```

Must pass with `0` errors before feature status changes in that group.

### Test Gate (REQUIRED before `verified`)

Run related tests after each group and before any `verified` update:

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests|FullyQualifiedName~JetStreamFileStoreTests|FullyQualifiedName~StorageEngineTests.FileStoreMultiLastSeqsAndLoadLastMsgWithLazySubjectState_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreAllLastSeqs_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreMultiLastSeqs_ShouldSucceed"
```

All selected tests must pass. Any failure blocks `verified`.

### Status Update Protocol (REQUIRED)

- Maximum `15` IDs per `feature batch-update` call.
- Evidence required per ID: Go lines reviewed, targeted test names, pass output, build pass.
- Transition order: `deferred/not_started -> stub -> complete -> verified`.
- Never mark `verified` before both Build Gate and Test Gate pass.
- If audit disagrees, use `--override "<specific evidence>"` only with concrete reason.

Status update commands for this batch:

```bash
# Claim work
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1011,1014-1015" --set-status stub --db porting.db --execute

# Group completion (run per group with only that group's IDs)
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<group ids>" --set-status complete --db porting.db --execute

# Group verification (after test gate)
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "<group ids>" --set-status verified --db porting.db --execute
```

### Checkpoint Protocol (REQUIRED between tasks)

Between Task 2, Task 3, and Task 4:

1. Run full build:
   ```bash
   dotnet build dotnet/
   ```
2. Run full unit test project:
   ```bash
   dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
   ```
3. Confirm pass/fail/skip summary and capture it in notes.
4. Commit checkpoint before moving to next task.

---

## ANTI-STUB GUARDRAILS (NON-NEGOTIABLE)

These rules apply to both production features and tests in this plan.

### Forbidden Patterns

Do not leave any of these in touched code:

- `throw new NotImplementedException(...)`
- `// TODO` or `// PLACEHOLDER` in implemented method bodies
- Empty method bodies (including one-line `{ }` placeholders)
- Test methods with no assertions
- Tests that only assert non-null for complex query behavior

### Hard Limits

- Max `15` feature IDs per status batch update.
- Max `1` feature group per status-update cycle.
- Mandatory build + related test gate before `verified`.
- Mandatory checkpoint commit between feature groups.

### If You Get Stuck

If a feature cannot be completed without missing infrastructure:

1. Leave or move that feature to `deferred` (never ship a stub).
2. Record the blocker using `--override` with specific reason.
3. Continue with remaining features in the same group.
4. Do not force `complete/verified` to unblock the batch.

Example:

```bash
dotnet run --project tools/NatsNet.PortTracker -- \
  feature update <id> --status deferred --db porting.db \
  --override "blocked: requires <missing dependency> for non-stub implementation"
```

---

### Task 1: Batch Setup and Baseline

**Files:**
- Modify: `porting.db`
- Read: `golang/nats-server/server/filestore.go`
- Read: `docs/plans/2026-02-27-batch-13-filestore-read-query-design.md`

**Step 1: Confirm batch readiness**

Run:

```bash
dotnet run --project tools/NatsNet.PortTracker -- batch show 13 --db porting.db
```

Expected: Batch 13 displayed with dependency on Batch 12.

**Step 2: Start batch**

Run:

```bash
dotnet run --project tools/NatsNet.PortTracker -- batch start 13 --db porting.db
```

Expected: Batch status changes to in-progress (or equivalent accepted state).

**Step 3: Claim all 8 features as `stub`**

Run:

```bash
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1011,1014-1015" --set-status stub --db porting.db --execute
```

Expected: All 8 features updated to `stub`.

**Step 4: Run baseline build and unit tests**

Run:

```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
```

Expected: Baseline captured before feature changes.

**Step 5: Commit baseline checkpoint**

```bash
git add porting.db docs/plans/2026-02-27-batch-13-filestore-read-query-design.md docs/plans/2026-02-27-batch-13-filestore-read-query-plan.md
git commit -m "plan(batch13): establish filestore read/query baseline and protocol"
```

---

### Task 2: Group A Features (1006, 1007, 1008) Skip-First-Block Helpers

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs`
- Create: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs`
- Optional Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs` (only if helper accessors are required)

**Step 1: Read Go source slice for Group A**

Read:

```bash
sed -n '3241,3307p' golang/nats-server/server/filestore.go
```

Expected: Exact behavior for `checkSkipFirstBlock`, `checkSkipFirstBlockMulti`, `selectSkipFirstBlock`.

**Step 2: Write failing tests for block-selection behavior**

Add tests covering:

- `filter=""` and `filter=">"` returns `bi + 1`.
- Literal filter with no `psim` match returns `ErrStoreEOF`.
- `stop <= currentBlockIndex` returns `ErrStoreEOF`.
- `start > currentBlockIndex` jumps to selected block index.
- Multi-filter path uses `SimpleSublist` intersection semantics.

**Step 3: Run targeted tests to verify failure**

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.CheckSkipFirstBlock|FullyQualifiedName~JetStreamFileStoreReadQueryTests.SelectSkipFirstBlock"
```

Expected: Failing tests indicating missing or incorrect behavior.

**Step 4: Implement Group A methods in `FileStore.cs`**

Implement logic equivalent to Go:

```csharp
private (int Next, Exception? Error) SelectSkipFirstBlock(int bi, uint start, uint stop)
{
    var mbi = _blks[bi].Index;
    if (stop <= mbi)
        return (-1, StoreErrors.ErrStoreEOF);
    if (start > mbi && _bim.TryGetValue(start, out var mb) && mb != null)
    {
        var (ni, _) = SelectMsgBlockWithIndex(mb.Last.Seq);
        return (ni, null);
    }
    return (bi + 1, null);
}
```

Use the same boundary rules for `CheckSkipFirstBlock` and multi-filter variant.

**Step 5: Run targeted tests to green**

Run:

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.CheckSkipFirstBlock|FullyQualifiedName~JetStreamFileStoreReadQueryTests.SelectSkipFirstBlock"
```

Expected: Pass.

**Step 6: Run Stub Detection Check, Build Gate, Test Gate**

Run all commands from `MANDATORY VERIFICATION PROTOCOL`.

**Step 7: Update statuses for Group A**

```bash
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1008" --set-status complete --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1006-1008" --set-status verified --db porting.db --execute
```

Expected: IDs `1006-1008` set to verified with recorded evidence.

**Step 8: Run Checkpoint Protocol and commit**

```bash
git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs \
  porting.db
git commit -m "feat(batch13): port filestore skip-first-block query helpers"
```

---

### Task 3: Group B Features (1009, 1010, 1011) Filtered Pending Helpers

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs`

**Step 1: Read Go source slice for Group B**

```bash
sed -n '3308,3503p' golang/nats-server/server/filestore.go
```

Expected: Behavior for `numFilteredPending*`, stale `fblk` correction, optional last-seq logic.

**Step 2: Write failing tests for filtered pending**

Add tests for:

- All-subject short-circuit (`filter=""` and `filter=">"`).
- No-match returns zeroed `SimpleState`.
- Literal and wildcard totals from `psim`.
- `NumFilteredPendingNoLast` leaves `Last == 0`.
- Stale `Psi.Fblk` is corrected after first miss path.

**Step 3: Run targeted tests to verify failure**

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.NumFilteredPending"
```

Expected: Fail until implementation is in place.

**Step 4: Implement Group B methods**

Port Go behavior:

- `NumFilteredPending` delegates to with-last variant (`includeLast=true`).
- `NumFilteredPendingNoLast` delegates with `includeLast=false`.
- `NumFilteredPendingWithLast`:
  - Calculates totals from `_psim`.
  - Resolves first via start block then forward scan.
  - Schedules `Psi.Fblk` correction under write lock if stale.
  - Computes last only when requested.

**Step 5: Run targeted + related tests**

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.NumFilteredPending|FullyQualifiedName~JetStreamFileStoreTests"
```

Expected: Pass.

**Step 6: Run Stub Detection Check, Build Gate, Test Gate**

Run all commands from `MANDATORY VERIFICATION PROTOCOL`.

**Step 7: Update statuses for Group B**

```bash
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1009-1011" --set-status complete --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1009-1011" --set-status verified --db porting.db --execute
```

**Step 8: Run Checkpoint Protocol and commit**

```bash
git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs \
  porting.db
git commit -m "feat(batch13): port filestore filtered-pending query helpers"
```

---

### Task 4: Group C Features (1014, 1015) Last-Sequence and Filter-All Helpers

**Files:**
- Modify: `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs`
- Modify: `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs`

**Step 1: Read Go source slice for Group C**

```bash
sed -n '3504,3571p' golang/nats-server/server/filestore.go
```

Expected: `allLastSeqsLocked` and `filterIsAll` behavior details.

**Step 2: Write failing tests for Group C**

Add tests for:

- `AllLastSeqsLocked` returns sorted last sequence list.
- No messages or no subject tracking returns empty/nil equivalent.
- Lazy `lastNeedsUpdate` path recalculates before collecting sequence.
- `FilterIsAll` true for reordered equivalent subject set.
- `FilterIsAll` false for count mismatch or non-subset match.

**Step 3: Run targeted tests to verify failure**

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.AllLastSeqsLocked|FullyQualifiedName~JetStreamFileStoreReadQueryTests.FilterIsAll"
```

Expected: Fail until implementation is complete.

**Step 4: Implement Group C methods**

Port behavior:

- Reverse block walk collecting per-subject last sequence once.
- Keep `subs` set to avoid duplicates.
- Sort final sequence array before return.
- `FilterIsAll` compares sorted `filters` to sorted configured subjects using subset-match check.

**Step 5: Run targeted + regression tests**

```bash
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ \
  --filter "FullyQualifiedName~JetStreamFileStoreReadQueryTests.AllLastSeqsLocked|FullyQualifiedName~JetStreamFileStoreReadQueryTests.FilterIsAll|FullyQualifiedName~StorageEngineTests.FileStoreMultiLastSeqsAndLoadLastMsgWithLazySubjectState_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreAllLastSeqs_ShouldSucceed|FullyQualifiedName~JetStreamMemoryStoreTests.MemStoreMultiLastSeqs_ShouldSucceed"
```

Expected: Pass.

**Step 6: Run Stub Detection Check, Build Gate, Test Gate**

Run all commands from `MANDATORY VERIFICATION PROTOCOL`.

**Step 7: Update statuses for Group C**

```bash
dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1014-1015" --set-status complete --db porting.db --execute

dotnet run --project tools/NatsNet.PortTracker -- \
  feature batch-update --ids "1014-1015" --set-status verified --db porting.db --execute
```

**Step 8: Run Checkpoint Protocol and commit**

```bash
git add dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs \
  dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreReadQueryTests.cs \
  porting.db
git commit -m "feat(batch13): port filestore all-last-seqs and filter-is-all helpers"
```

---

### Task 5: Batch Completion and Handoff

**Files:**
- Modify: `porting.db`
- Optional Modify: `reports/current.md` (if report script updates it)

**Step 1: Full verification sweep**

```bash
dotnet build dotnet/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.Tests/
dotnet test dotnet/tests/ZB.MOM.NatsNet.Server.IntegrationTests/
```

Expected: Green build and tests (or clearly documented pre-existing failures).

**Step 2: Audit features**

```bash
dotnet run --project tools/NatsNet.PortTracker -- audit --type features --db porting.db
```

Expected: Batch 13 IDs classified consistently with implementation.

**Step 3: Complete batch**

```bash
dotnet run --project tools/NatsNet.PortTracker -- batch complete 13 --db porting.db
```

Expected: Batch 13 completes if all required statuses are valid.

**Step 4: Refresh report and inspect newly unblocked work**

```bash
./reports/generate-report.sh
dotnet run --project tools/NatsNet.PortTracker -- dependency ready --db porting.db
```

Expected: Updated report and visibility into next available batches.

---

## Evidence Log Template (Use During Execution)

For each feature ID, capture:

- Go reference: file + line range reviewed.
- Test cases added/updated.
- Targeted test command + pass summary.
- Build command + pass summary.
- Stub scan output status.
- PortTracker status command(s) executed.

This evidence is required before any `verified` transition.