natsnet/docs/plans/2026-02-27-batch-11-filestore-init-design.md

# Batch 11 FileStore Init Design

**Date:** 2026-02-27  
**Batch:** 11 (`FileStore Init`)  
**Dependency:** Batch 8 (`Store Interfaces`)  
**Scope:** 39 features + 123 tests mapped to `server/filestore.go`

---

## 1. Context and Constraints

- Batch 11 is currently `pending` with all 39 features and all 123 tests in `deferred` status.
- Current .NET `JetStreamFileStore` implementation is mostly a delegation shell to `JetStreamMemStore`; most Batch 11 methods are not present yet.
- Existing backlog tests for this area include placeholder-style tests that must not be treated as verification evidence.
- Planning only in this session: produce design + implementation plan documents, no runtime code changes.

## 2. Success Criteria

- Implement all Batch 11 feature IDs with real C# behavior matching Go intent for `filestore.go` initialization-related scope.
- Port and verify all Batch 11 tests into real behavioral tests (no placeholder assertions).
- Enforce strict verification and anti-stub guardrails before any status updates.
- Preserve dependency order and use PortTracker updates with evidence-backed chunks.

## 3. Approach Options

### Option A (Recommended): Vertical feature groups with co-evolving tests

Implement features in 3 groups (17, 16, 6 features), each followed by targeted test waves and status updates.

- Pros: bounded risk, quick feedback, easier rollback and review, aligns with dependency and guardrail requirements.
- Cons: repeated build/test cycles increase total command count.

### Option B: Implement all 39 features first, then all 123 tests

- Pros: fewer context switches between prod/test files.
- Cons: very high integration risk; defects surface late; hard to attribute failures to specific changes.

### Option C: Utilities-first (helpers, codecs, file writers) then constructors/recovery

- Pros: helper APIs stabilize early.
- Cons: delayed validation of end-to-end init behavior; tends to accumulate partial implementations.

### Recommendation

Use **Option A**. This gives the strongest control over regressions and makes status updates defensible under audit.

## 4. Proposed Design

### 4.1 Feature Workstreams (max ~20 features/group)

- **Group 1: Store creation + encryption bootstrap + metadata + block recovery (17 features)**  
  IDs: `955, 956, 957, 958, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 972, 974`
- **Group 2: Lost-data/state rebuild + tracking/utilities (16 features)**  
  IDs: `975, 976, 978, 979, 980, 989, 990, 998, 1057, 1152, 1153, 1154, 1158, 1163, 1164, 1165`
- **Group 3: Init semaphore + consumer decode + atomic file writes (6 features)**  
  IDs: `1200, 1235, 1239, 1248, 1261, 1262`

### 4.2 File Boundaries

- Primary production files:
  - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs`
  - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs`
  - `dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs`
- Primary test files:
  - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs`
  - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs`
  - `dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs`

### 4.3 Data/Control Flow to Port

- `newFileStoreWithCreated` bootstrap flow:
  - validate config and storage mode
  - apply dynamic block sizing defaults
  - verify/create directories and writable checks
  - initialize hash/checksum state and encryption metadata
  - recover persisted stream/message state
  - enforce limits and age/timer setup
  - start background flush/state loops
- Supporting flows:
  - message block buffer pooling and recycling
  - block encryption key generation/recovery
  - lost data tracking and rebuild state accounting
  - consumer state binary header/decode path
  - atomic write helpers with optional sync and directory sync

### 4.4 Error Handling Design

- Preserve failure semantics from Go intent:
  - reject invalid storage config early
  - propagate IO/crypto errors with contextual messages
  - fail closed for corrupt headers / decode failures
  - avoid silent fallback to placeholder behavior
- Ensure lock discipline for shared mutable state and block-level operations.

### 4.5 Test Design

- Replace placeholder backlog tests with behavior-focused tests derived from Go cases.
- Keep benchmark-mapped tests as either:
  - converted deterministic assertions where appropriate, or
  - deferred with explicit infrastructure/perf rationale.
- Include focused regression tests for encryption key lifecycle, metadata integrity, state recovery, and consumer state decoding.

## 5. Risks and Mitigations

- **Risk:** large `filestore.go` surface can lead to stub creep.  
  **Mitigation:** mandatory stub scans for both src and tests, hard stop on matches.
- **Risk:** hidden behavior coupling across utility methods.  
  **Mitigation:** group-based checkpoints with full build + targeted suite each cycle.
- **Risk:** status drift (marking verified without evidence).  
  **Mitigation:** evidence logs required for every feature/test update, max 15 IDs per update.

## 6. Design Outcome

This design uses a strict vertical execution model with heavy verification gates. It is structured to be executed by a plan that enforces per-feature validation loops, anti-stub guardrails, and evidence-based PortTracker updates.