natsnet/docs/plans/2026-02-27-batch-15-msgblock-consumerfilestore-design.md

# Batch 15 MsgBlock + ConsumerFileStore Design

I'm using `brainstorm` to produce this design before implementation planning.

## Context

- Batch: `15` (`MsgBlock + ConsumerFileStore`)
- Dependency: Batch `14` (`FileStore Write/Lifecycle`)
- Scope: `121` features + `89` tests
- Go source: `golang/nats-server/server/filestore.go`
- Current batch status: `pending` (all mapped features/tests currently `deferred`)

Observed with PortTracker:

- `batch show 15`: full feature/test inventory loaded from `server/filestore.go` and related tests.
- `batch list`: Batch 15 depends on Batch 14.
- `batch ready`: Batch 15 is not startable yet (dependency gate not satisfied).
- `report summary`: overall porting progress is `1924/6942 (27.7%)`.

## Problem Statement

Batch 15 is the deepest part of FileStore internals: message block recovery/scanning, cache lifecycle, compaction/tombstones, disk rewrite/compression/encryption paths, and `ConsumerFileStore` state flushing. If this batch is implemented without strict verification discipline, downstream stream/consumer lifecycle batches will inherit correctness and durability bugs that are expensive to debug.

## Constraints and Success Criteria

- Do not start Batch 15 execution until Batch 14 is complete.
- Keep implementation scoped to Batch 15 mapped IDs.
- Enforce non-stub behavior for both production code and tests.
- Promote statuses only with evidence-backed build/test proof.
- If blocked, keep IDs deferred with explicit reasons; do not fake-pass.

Success for this batch means:

1. All implementable Batch 15 features are real C# ports with parity-oriented behavior.
2. All implementable mapped tests are real behavioral tests and passing.
3. Blocked items remain deferred with concrete, auditable reasons.
4. Batch can pass final `batch complete 15` validation once dependencies are met and evidence is complete.

## Approaches Considered

### Approach 1 (Recommended): Vertical slices by feature groups and test waves

Split 121 features into seven groups (`20/20/20/20/20/20/1`) and execute each group with immediate build/test/status gates before moving on.

Pros:

- Contains risk in file-store concurrency code.
- Makes regressions local and debuggable.
- Supports strict anti-stub and evidence requirements.

Cons:

- More checkpoints and bookkeeping overhead.

### Approach 2: Feature-complete first, tests second

Implement all production features first, then port all tests.

Pros:

- Fewer context switches while coding.

Cons:

- Late detection of regressions.
- Weak traceability for status updates.
- Higher risk of last-minute stub behavior.

### Approach 3: Test-first broad wave with temporary placeholders

Attempt to satisfy test volume rapidly with minimal implementation placeholders.

Pros:

- Fast apparent progress.

Cons:

- Violates anti-stub requirement.
- Creates false confidence and rework.
- Unacceptable for durability-critical file store logic.

## Recommended Design

### 1) Implementation Topology

Primary code targets:

- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs`
- `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/StoreTypes.cs`

Scope focus:

- `MessageBlock` internals (95 mapped features)
- `ConsumerFileStore` internals (18 mapped features)
- Store enum/codec helpers and error wrappers

### 2) Verification-First Control Plane

Every feature group and test wave must pass:

- Per-feature read/port/build/test loop
- Stub scan gates (production + tests)
- Build gate after each feature group
- Related test gate before feature `verified`
- Chunked status updates (`<=15` IDs/update)
- Full checkpoint (`build + full test + commit`) between tasks

### 3) Test Strategy

Mapped tests are concentrated in:

- `JetStreamFileStoreTests.Impltests.cs` (58)
- `JwtProcessorTests.Impltests.cs` (24)
- `GatewayHandlerTests.Impltests.cs` (2)
- `ConcurrencyTests1.Impltests.cs` (1)
- `ConcurrencyTests2.Impltests.cs` (2)
- `EventsHandlerTests.Impltests.cs` (1)
- `ConfigReloaderTests.Impltests.cs` (1)

Design decision:

- Treat filestore/concurrency tests as primary verification for Batch 15 behavior.
- Treat cross-module JWT/gateway/events/reload tests as dependency-sensitive; keep deferred with reasons if blocked by non-Batch-15 surfaces.

### 4) Data and Concurrency Parity Priorities

Highest-risk behavior areas to preserve from Go:

- Block rebuild and index/tombstone accounting integrity.
- Cache ownership transitions and forced expiration races.
- Flush loops and fsync boundaries for pending writes.
- Compression/encryption conversion and checksum correctness.
- Consumer state encode/encrypt/flush sequencing.

### 5) Status Governance

- Feature lifecycle: `deferred -> stub -> complete -> verified`
- Test lifecycle: `deferred -> stub -> verified` (or `deferred` with blocker note)
- Status updates require direct evidence references (Go line reviewed, build output, targeted test output, stub scan clean).

## Non-Goals

- Pulling in work from future stream/consumer lifecycle batches beyond mapped Batch 15 IDs.
- Marking cross-module tests verified without executable evidence.
- Using placeholder methods/tests to satisfy status transitions.

## Acceptance Criteria

- Batch 15 feature groups fully implemented with non-stub logic.
- Mandatory verification protocol passed for each group/wave.
- Evidence-backed status transitions applied in chunks of max 15 IDs.
- Deferred items carry explicit blocker reasons.
- Batch closure commands are expected to pass once dependency Batch 14 is complete.