natsnet/docs/plans/2026-02-27-batch-12-filestore-recovery-design.md

Batch 12 FileStore Recovery Design

Date: 2026-02-27
Batch: 12 (FileStore Recovery)
Dependency: Batch 11 (FileStore Init)
Scope: 8 features + 0 tests mapped to golang/nats-server/server/filestore.go

---

1. Context and Constraints

• PortTracker reports Batch 12 as pending, dependent on Batch 11, with 8 deferred features and no batch-owned tests.
• Batch 12 methods are all in the FileStore recovery path:
  • warn (ID 987)
  • debug (ID 988)
  • recoverFullState (ID 991)
  • recoverTTLState (ID 992)
  • recoverMsgSchedulingState (ID 993)
  • cleanupOldMeta (ID 995)
  • recoverMsgs (ID 996)
  • expireMsgsOnRecover (ID 997)
• Current .NET JetStreamFileStore is still a mostly delegated shell; these recovery methods are not implemented yet.
• Even though Batch 12 has 0 mapped tests, reverse dependencies exist for at least:
  • test #519 (RecoverFullState corruption detection)
  • test #545 (RecoverTTLState corrupt block resilience)
• Planning-only session: produce design and implementation plan documents; do not implement feature code.

2. Success Criteria

• Implement all 8 Batch 12 features with behavior parity to Go intent for recovery flows in filestore.go.
• Introduce recovery logic without stubs/placeholders in either production or related test files.
• Verify each feature using a strict per-feature loop: Go-source read, C# implementation, build, related tests.
• Use evidence-backed PortTracker status updates in small chunks and keep dependency/order integrity.

3. Brainstormed Approach Options

Option A (Recommended): Two vertical recovery groups with strict gates

Group methods by lifecycle stage, then run full build/test/status gates between groups.

• Pros: low integration risk, easy isolation of regressions, aligns with mandatory checkpointing.
• Cons: more build/test cycles.

Option B: Implement all 8 methods in one pass, verify once at the end

• Pros: fastest coding flow initially.
• Cons: high blast radius if behavior diverges; poor auditability for status updates.

Option C: Logging + cleanup first, then all recovery methods together

• Pros: simple early wins on low-complexity methods.
• Cons: still delays hardest-state validation and can hide ordering bugs until late.

Recommendation

Use Option A. Recovery behavior is stateful and failure-prone; bounded groups plus hard gates are the safest path.

4. Proposed Design

4.1 Feature Groups (max ~20 each)

• Group 1 (5 features): 987, 988, 991, 992, 993
  • Logging wrappers and persisted state/TTL/scheduling recovery.
• Group 2 (3 features): 995, 996, 997
  • Metadata cleanup, message block recovery sweep, startup age-expiration pass.

4.2 Primary Code Boundaries

• Production:
  • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs (main implementation surface)
  • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs (state and helper interactions used by recovery logic)
  • dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs (supporting types/constants as needed)
• Related tests and verification targets:
  • dotnet/tests/ZB.MOM.NatsNet.Server.Tests/JetStream/JetStreamFileStoreTests.cs
  • dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs

4.3 Recovery Flow to Port

1. Observability wrappers (warn, debug): no-op when server/logger is absent; prefix context consistently.
2. Full state recovery (recoverFullState):
   • Read state file, validate minimum length and checksum, decrypt when configured.
   • Decode stream + subject + block state, detect corruption/outdated state, and force rebuild fallback conditions.
3. TTL and scheduling recovery (recoverTTLState, recoverMsgSchedulingState):
   • Decode persisted wheels/schedules.
   • If stale, linear scan from recovered sequence through message blocks to rebuild runtime structures.
4. Metadata cleanup (cleanupOldMeta): remove stale .idx/.fss files in message directory.
5. Message recovery (recoverMsgs): enumerate blocks in order, recover each block, reconcile stream first/last/msgs/bytes, prune orphan key files.
6. Startup age expiration (expireMsgsOnRecover): expire max-age-out messages/blocks safely, preserve tombstone behavior, and recompute top-level state.

4.4 Error Handling and Safety

• Corrupt/full-state mismatch must fail closed and trigger rebuild path instead of continuing with partial state.
• IO permissions/corruption conditions should be surfaced with actionable context (not swallowed).
• Lock ordering must remain consistent with existing FileStore concurrency patterns to avoid recovery-time deadlocks.

4.5 Verification Design (Feature + Related Tests)

• Feature verification is not allowed on build-only evidence.
• Each feature needs:
  • Go-source intent review
  • build pass
  • related test execution pass (or explicit deferred reason if test infra is unavailable)
• If related tests for a feature are still deferred/unported, feature can be moved to complete but not verified unless equivalent evidence exists and is documented.

5. Risks and Mitigations

• Risk: recovery logic complexity leads to placeholder shortcuts.
  Mitigation: mandatory stub scans after every group for both source and tests.
• Risk: subtle state-accounting regressions (first/last/msgs/bytes).
  Mitigation: group gates require targeted tests plus JetStream store regression filters.
• Risk: aggressive status updates without proof.
  Mitigation: cap status update chunks to 15 IDs and require command-output evidence per chunk.

6. Design Outcome

Batch 12 should be executed as two recovery-focused feature groups with strict evidence gates between them. This design preserves dependency discipline, prevents stub creep, and keeps feature status updates auditable against build/test outputs.