Files

Joseph Doherty f0455a1e45 Add batch plans for batches 6-7, 9-12, 16-17 (rounds 4-7)

Generated design docs and implementation plans via Codex for:
- Batch 6: Opts package-level functions
- Batch 7: Opts class methods + Reload
- Batch 9: Auth, DirStore, OCSP foundations
- Batch 10: OCSP Cache + JS Events
- Batch 11: FileStore Init
- Batch 12: FileStore Recovery
- Batch 16: Client Core (first half)
- Batch 17: Client Core (second half)

All plans include mandatory verification protocol and anti-stub guardrails.
Updated batches.md with file paths and planned status.

2026-02-27 14:56:19 -05:00

5.3 KiB

Raw Blame History

Batch 11 FileStore Init Design

Date: 2026-02-27
Batch: 11 (FileStore Init)
Dependency: Batch 8 (Store Interfaces)
Scope: 39 features + 123 tests mapped to server/filestore.go

1. Context and Constraints

Batch 11 is currently pending with all 39 features and all 123 tests in deferred status.
Current .NET JetStreamFileStore implementation is mostly a delegation shell to JetStreamMemStore; most Batch 11 methods are not present yet.
Existing backlog tests for this area include placeholder-style tests that must not be treated as verification evidence.
Planning only in this session: produce design + implementation plan documents, no runtime code changes.

2. Success Criteria

Implement all Batch 11 feature IDs with real C# behavior matching Go intent for filestore.go initialization-related scope.
Port and verify all Batch 11 tests into real behavioral tests (no placeholder assertions).
Enforce strict verification and anti-stub guardrails before any status updates.
Preserve dependency order and use PortTracker updates with evidence-backed chunks.

3. Approach Options

Option A (Recommended): Vertical feature groups with co-evolving tests

Implement features in 3 groups (17, 16, 6 features), each followed by targeted test waves and status updates.

Pros: bounded risk, quick feedback, easier rollback and review, aligns with dependency and guardrail requirements.
Cons: repeated build/test cycles increase total command count.

Option B: Implement all 39 features first, then all 123 tests

Pros: fewer context switches between prod/test files.
Cons: very high integration risk; defects surface late; hard to attribute failures to specific changes.

Option C: Utilities-first (helpers, codecs, file writers) then constructors/recovery

Pros: helper APIs stabilize early.
Cons: delayed validation of end-to-end init behavior; tends to accumulate partial implementations.

Recommendation

Use Option A. This gives the strongest control over regressions and makes status updates defensible under audit.

4. Proposed Design

4.1 Feature Workstreams (max ~20 features/group)

Group 1: Store creation + encryption bootstrap + metadata + block recovery (17 features)
IDs: 955, 956, 957, 958, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 972, 974
Group 2: Lost-data/state rebuild + tracking/utilities (16 features)
IDs: 975, 976, 978, 979, 980, 989, 990, 998, 1057, 1152, 1153, 1154, 1158, 1163, 1164, 1165
Group 3: Init semaphore + consumer decode + atomic file writes (6 features)
IDs: 1200, 1235, 1239, 1248, 1261, 1262

4.2 File Boundaries

Primary production files:
- dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStore.cs
- dotnet/src/ZB.MOM.NatsNet.Server/JetStream/MessageBlock.cs
- dotnet/src/ZB.MOM.NatsNet.Server/JetStream/FileStoreTypes.cs
Primary test files:
- dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/JetStreamFileStoreTests.Impltests.cs
- dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests1.Impltests.cs
- dotnet/tests/ZB.MOM.NatsNet.Server.Tests/ImplBacklog/ConcurrencyTests2.Impltests.cs

4.3 Data/Control Flow to Port

newFileStoreWithCreated bootstrap flow:
- validate config and storage mode
- apply dynamic block sizing defaults
- verify/create directories and writable checks
- initialize hash/checksum state and encryption metadata
- recover persisted stream/message state
- enforce limits and age/timer setup
- start background flush/state loops
Supporting flows:
- message block buffer pooling and recycling
- block encryption key generation/recovery
- lost data tracking and rebuild state accounting
- consumer state binary header/decode path
- atomic write helpers with optional sync and directory sync

4.4 Error Handling Design

Preserve failure semantics from Go intent:
- reject invalid storage config early
- propagate IO/crypto errors with contextual messages
- fail closed for corrupt headers / decode failures
- avoid silent fallback to placeholder behavior
Ensure lock discipline for shared mutable state and block-level operations.

4.5 Test Design

Replace placeholder backlog tests with behavior-focused tests derived from Go cases.
Keep benchmark-mapped tests as either:
- converted deterministic assertions where appropriate, or
- deferred with explicit infrastructure/perf rationale.
Include focused regression tests for encryption key lifecycle, metadata integrity, state recovery, and consumer state decoding.

5. Risks and Mitigations

Risk: large filestore.go surface can lead to stub creep.
Mitigation: mandatory stub scans for both src and tests, hard stop on matches.
Risk: hidden behavior coupling across utility methods.
Mitigation: group-based checkpoints with full build + targeted suite each cycle.
Risk: status drift (marking verified without evidence).
Mitigation: evidence logs required for every feature/test update, max 15 IDs per update.

6. Design Outcome

This design uses a strict vertical execution model with heavy verification gates. It is structured to be executed by a plan that enforces per-feature validation loops, anti-stub guardrails, and evidence-based PortTracker updates.

5.3 KiB Raw Blame History