CBDD/compact_plan.md

# Compression + Compaction Implementation Plan

## 1. Objectives

Implement two major capabilities in CBDD:

1. Compression for stored document payloads (including overflow support, compatibility, safety, and telemetry).
2. Compaction/Vacuum for reclaiming fragmented space and shrinking database files safely.

This plan is grounded in the current architecture:
- Storage pages and free-list in `src/CBDD.Core/Storage/PageFile.cs`
- Slot/page layout in `src/CBDD.Core/Storage/SlottedPage.cs`
- Document CRUD + overflow paths in `src/CBDD.Core/Collections/DocumentCollection.cs`
- WAL/transaction/recovery semantics in `src/CBDD.Core/Storage/StorageEngine*.cs` and `src/CBDD.Core/Transactions/WriteAheadLog.cs`
- Collection metadata in `src/CBDD.Core/Storage/StorageEngine.Collections.cs`

## 2. Key Design Decisions

### 2.1 Compression unit: **Per-document before overflow**

Chosen model: compress the full serialized BSON document first, then apply existing overflow chunking to the stored bytes.

Why this choice:
- Single compression decision per document (simple threshold logic).
- Overflow logic remains generic over byte blobs.
- Update path can compare stored compressed size vs existing slot size directly.
- Better ratio than per-chunk for many document shapes.

Consequences:
- `HasOverflow` continues to represent storage chaining only.
- `Compressed` continues to represent payload encoding.
- Read path always reconstructs full stored blob (from primary + overflow), then decompresses if flagged.

### 2.2 Codec strategy

Implement a codec abstraction with initial built-in codecs backed by .NET primitives:
- `None`
- `Brotli`
- `Deflate`

Expose via config:
- `EnableCompression`
- `Codec`
- `Level`
- `MinSizeBytes`
- `MinSavingsPercent`

Add safety knobs:
- `MaxDecompressedSizeBytes` (guardrail)
- optional `MaxCompressionInputBytes` (defensive cap for write path)

### 2.3 Metadata strategy

Use layered metadata for compatibility and decoding:

1. DB-level persistent metadata (Page 0 extension region):
- DB format version
- feature flags (compression enabled capability)
- default codec id

2. Page-level format metadata:
- page format version marker (for mixed old/new page parsing)
- optional default codec hint (for diagnostics and future tuning)

3. Slot payload metadata for compressed entries (fixed header prefix in stored payload):
- codec id
- original length
- compressed length
- checksum (CRC32 of compressed payload bytes)

This avoids breaking old uncompressed pages while still satisfying “readers know how to decode” and checksum requirements.

## 3. Workstreams

## 3.1 Workstream A: Compression Core + Config Surface

### Deliverables
- New compression options model and codec abstraction.
- Persistent file/page format metadata support.
- Telemetry primitives for compression counters.

### Changes
- Add `src/CBDD.Core/Compression/`:
  - `CompressionOptions.cs`
  - `CompressionCodec.cs`
  - `ICompressionCodec.cs`
  - `CompressionService.cs`
  - `CompressedPayloadHeader.cs`
  - `CompressionTelemetry.cs`
- Extend context/engine construction:
  - `src/CBDD.Core/DocumentDbContext.cs`
  - `src/CBDD.Core/Storage/StorageEngine.cs`
- Add DB metadata read/write helpers:
  - `src/CBDD.Core/Storage/PageFile.cs`
  - new `src/CBDD.Core/Storage/StorageEngine.Format.cs`

### Implementation tasks
1. Introduce `CompressionOptions` with defaults:
   - compression disabled by default
   - conservative thresholds (`MinSizeBytes`, `MinSavingsPercent`)
2. Add codec registry/factory.
3. Add DB format metadata block in page 0 extension with version + feature flags + default codec id.
4. Add page format marker to slotted pages on allocation path (new pages only).
5. Add telemetry counter container (thread-safe atomic counters).

### Acceptance
- Existing DB files open unchanged.
- New DB files persist format metadata.
- Compression service can roundtrip payloads with selected codec.

## 3.2 Workstream B: Insert + Read Paths (new writes first)

### Deliverables
- Compression on insert with threshold logic.
- Safe decompression on reads with checksum and size validation.
- Fallback to uncompressed write on compression failure.

### Changes
- `src/CBDD.Core/Collections/DocumentCollection.cs`:
  - `InsertDataCore`
  - `InsertIntoPage`
  - `InsertWithOverflow`
  - `FindByLocation`
- `src/CBDD.Core/Storage/SlottedPage.cs` (if slot/page metadata helpers are added)

### Implementation tasks
1. Insert path:
   - Serialize BSON (existing behavior).
   - If compression enabled and `docData.Length >= MinSizeBytes`, try codec.
   - Compute savings and only set `SlotFlags.Compressed` if threshold met.
   - Build compressed payload as: `[CompressedPayloadHeader][compressed bytes]`.
   - On any compression exception, increment failure counter and store uncompressed.
2. Overflow path:
   - Apply overflow based on stored bytes length (compressed or uncompressed).
   - No separate compression of chunks/pages.
3. Read path:
   - Existing overflow reassembly first.
   - If `Compressed` flag present:
     - parse payload header
     - validate compressed length + original length bounds
     - validate checksum before decompression
     - decompress using codec id
     - enforce `MaxDecompressedSizeBytes`
4. Corruption handling:
   - throw deterministic `InvalidDataException` for header/checksum/size violations.

### Acceptance
- Inserts/reads succeed for uncompressed docs (regression-safe).
- Mixed compressed/uncompressed documents in same collection read correctly.
- Corrupted compressed payload is detected and rejected predictably.

## 3.3 Workstream C: Update/Delete + Overflow Consistency

### Deliverables
- Compression-aware update decisions.
- Correct delete behavior for compressed+overflow combinations.

### Changes
- `src/CBDD.Core/Collections/DocumentCollection.cs`:
  - `UpdateDataCore`
  - `DeleteCore`
  - `FreeOverflowChain`

### Implementation tasks
1. Update path:
   - Recompute storage payload for new document using same compression decision logic as insert.
   - In-place update only when:
     - existing slot is non-overflow, and
     - new stored payload length <= old slot length, and
     - compression flag/metadata can be updated safely.
   - Otherwise relocate (existing delete+insert strategy).
2. Delete path:
   - Keep logical semantics unchanged.
   - Ensure overflow chain extraction still works when slot has both `Compressed` and `HasOverflow`.
3. Overflow consistency tests:
   - compressed small -> compressed overflow transitions on update
   - compressed overflow -> uncompressed small transitions

### Acceptance
- Update behavior preserves correctness for all compression/overflow combinations.
- Delete frees overflow pages for compressed and uncompressed overflow docs.

## 3.4 Workstream D: Compaction / Shrink (Offline first)

### Deliverables
- Public `Compact`/`Vacuum` maintenance API.
- Offline copy-and-swap compaction with crash-safe finalize.
- Exact pre/post space accounting.

### API surface
- Add to `DocumentDbContext`:
  - `Compact(CompactionOptions? options = null)`
  - `CompactAsync(...)`
  - alias `Vacuum(...)`
- Engine-level operation in new file:
  - `src/CBDD.Core/Storage/StorageEngine.Maintenance.cs`

### Offline mode algorithm (Phase 1)
1. Acquire exclusive maintenance gate (block writes).
2. Checkpoint WAL before start.
3. Build a temporary database file (`*.compact.tmp`) with same page config and compression config.
4. Copy logical contents collection-by-collection:
   - preserve collection metadata/index definitions
   - reinsert documents through collection APIs so locations are rewritten correctly
   - rebuild/update index roots in metadata
5. Checkpoint temp DB and fsync.
6. Atomic finalize (copy-and-swap):
   - write state marker (`*.compact.state`) for resumability
   - rename original -> backup, temp -> original
   - reset/remove WAL appropriately
   - remove marker
7. Produce `CompactionStats` with exact pre/post bytes and deltas.

### Crash safety
- Use explicit state-machine marker file with phases (`Started`, `Copied`, `Swapped`, `CleanupDone`).
- On startup, detect marker and resume/repair idempotently.

### Acceptance
- File shrinks when free tail pages exist.
- No data/index loss after compaction.
- Crash during compaction is recoverable and deterministic.

## 3.5 Workstream E: Online Compaction + Scheduling

### Deliverables
- Online mode with throttled relocation.
- Scheduling hooks (manual/startup/threshold-based trigger).

### Online mode strategy (Phase 2)
1. Background scanner identifies fragmented pages and relocation candidates.
2. Move documents in bounded batches under short write exclusion windows.
3. Update primary and secondary index locations transactionally.
4. Periodic checkpoints to bound WAL growth.
5. Tail truncation pass when contiguous free pages reach EOF.

### Scheduling hooks
- `MaintenanceOptions`:
  - `RunAtStartup`
  - `MinFragmentationPercent`
  - `MinReclaimableBytes`
  - `MaxRunDuration`
  - `OnlineThrottle` (ops/sec or pages/batch)

### Acceptance
- Writes continue during online mode except small lock windows.
- Recovery semantics remain valid with WAL and checkpoints.

## 4. Compaction Internals Required by Both Modes

### 4.1 Page defragmentation utilities
- Add slotted-page defrag helper:
  - rewrites active slots/data densely
  - recomputes `FreeSpaceStart/End`

### 4.2 Free-list consolidation + tail truncation
- Extend `PageFile` with:
  - free-page enumeration
  - free-list normalization
  - safe truncation when free pages are contiguous at end-of-file

### 4.3 Metadata/index pointer correctness
- Ensure collection metadata root IDs and index root IDs are rewritten/verified after relocation/copy.
- Add validation pass that checks all primary index locations resolve to non-deleted slots.

### 4.4 WAL coordination
- Explicit checkpoint before and after compaction.
- Ensure compaction writes follow normal WAL durability semantics.
- Keep `Recover()` behavior valid with compaction marker states.

## 5. Compatibility and Migration

### 5.1 Compatibility goals
- Read old uncompressed files unchanged.
- Support mixed pages/documents (compressed + uncompressed) in same DB.
- Preserve existing APIs unless explicitly extended.

### 5.2 Migration tool (optional one-time rewrite)
- Implement `MigrateCompression(...)` as an offline rewrite command using the same copy-and-swap machinery.
- Options:
  - target codec/level
  - per-collection include/exclude
  - dry-run estimation mode

## 6. Telemetry + Admin Tooling

### 6.1 Compression telemetry counters
- compressed document count
- bytes before/after
- compression ratio aggregate
- compression CPU time
- decompression CPU time
- compression failure count
- checksum failure count
- safety-limit rejection count

Expose via:
- `StorageEngine.GetCompressionStats()`
- context-level forwarding method in `DocumentDbContext`

### 6.2 Compaction telemetry/stats
- pre/post file size
- live bytes
- free bytes
- fragmentation percentage
- documents/pages relocated
- runtime and throughput

### 6.3 Admin inspection APIs
Add diagnostics APIs (engine/context):
- page usage by collection/page type
- compression ratio by collection
- fragmentation map and free-list summary

## 7. Tests

Add focused test suites in `tests/CBDD.Tests/`:

1. `CompressionInsertReadTests.cs`
- threshold on/off behavior
- mixed compressed/uncompressed reads
- fallback to uncompressed on forced codec error

2. `CompressionOverflowTests.cs`
- compressed docs that span overflow pages
- transitions across size thresholds

3. `CompressionCorruptionTests.cs`
- bad checksum
- bad original length
- oversized decompression guardrail
- invalid codec id

4. `CompressionCompatibilityTests.cs`
- open existing uncompressed DB files
- mixed-format pages after partial migration

5. `CompactionOfflineTests.cs`
- logical equivalence pre/post compact
- index correctness pre/post compact
- tail truncation actually reduces file size

6. `CompactionCrashRecoveryTests.cs`
- simulate crashes at each copy/swap state
- resume/finalize behavior

7. `CompactionOnlineConcurrencyTests.cs`
- concurrent writes + reads during online compact
- correctness and no deadlock

8. `CompactionWalCoordinationTests.cs`
- checkpoint before/after behavior
- recoverability with in-flight WAL entries

Also update/extend existing tests:
- `tests/CBDD.Tests/DocumentOverflowTests.cs`
- `tests/CBDD.Tests/DbContextTests.cs`
- `tests/CBDD.Tests/WalIndexTests.cs`

## 8. Benchmark Additions

Extend `tests/CBDD.Tests.Benchmark/` with:

1. `CompressionBenchmarks.cs`
- insert/update/read workloads with compression on/off
- codec and level comparison

2. `CompactionBenchmarks.cs`
- offline compact runtime
- reclaimable bytes vs elapsed

3. `MixedWorkloadBenchmarks.cs`
- insert/update/delete-heavy cache workload
- periodic compact impact

4. Update `DatabaseSizeBenchmark.cs`
- pre/post compact shrink delta reporting
- compression ratio reporting

## 9. Suggested Implementation Order (Execution Plan)

### Phase 1 (as requested): Compression config + read/write path for new writes only
- Workstream A
- Workstream B (insert/read only)
- initial tests: insert/read + compatibility

### Phase 2 (as requested): Compression-aware updates/deletes + overflow handling
- Workstream C
- overflow-focused tests + corruption guards

### Phase 3 (as requested): Offline copy-and-swap compaction/shrink
- Workstream D + shared internals from section 4
- crash-safe finalize + space accounting

### Phase 4 (as requested): Online compaction + automation hooks
- Workstream E
- concurrency and scheduling tests

## 10. Subagent Execution Safety + Completion Verification

### 10.1 Subagent ownership model

Use explicit, non-overlapping ownership to avoid unsafe parallel edits:

1. Subagent A (Compression Core)
- Owns `src/CBDD.Core/Compression/*`
- Owns format/config plumbing in `src/CBDD.Core/Storage/StorageEngine.Format.cs`, `src/CBDD.Core/Storage/StorageEngine.cs`, `src/CBDD.Core/DocumentDbContext.cs`

2. Subagent B (CRUD + Overflow Compression Semantics)
- Owns compression-aware CRUD changes in `src/CBDD.Core/Collections/DocumentCollection.cs`
- Owns slot/payload metadata helpers in `src/CBDD.Core/Storage/SlottedPage.cs` (if needed)

3. Subagent C (Compaction/Vacuum Engine)
- Owns `src/CBDD.Core/Storage/StorageEngine.Maintenance.cs`
- Owns related `PageFile` extensions in `src/CBDD.Core/Storage/PageFile.cs`

4. Subagent D (Verification Assets)
- Owns new/updated tests in `tests/CBDD.Tests/*Compression*`, `tests/CBDD.Tests/*Compaction*`
- Owns benchmark additions in `tests/CBDD.Tests.Benchmark/*`

Rule: one file has exactly one active subagent owner at a time.

### 10.2 Safe collaboration rules for subagents

1. Do not edit files outside assigned ownership scope.
2. Do not revert or reformat unrelated existing changes.
3. Do not change public contracts outside assigned workstream without explicit handoff.
4. If overlap is discovered mid-task, stop and reassign ownership before continuing.
5. Keep changes atomic and reviewable (small PR-sized batches per workstream phase).

### 10.3 Required handoff payload from each subagent

Each completion handoff MUST include:

1. Summary of implemented requirements and non-implemented items.
2. Exact touched file list.
3. Risk list (behavioral, compatibility, recovery, perf).
4. Test commands executed and pass/fail results.
5. Mapping to plan acceptance criteria sections.

### 10.4 Mandatory verification on subagent completion

Every subagent completion is verified before merge/mark-done:

1. Scope verification
- Confirm touched files are in owned scope only.
- Confirm required plan items for that phase are implemented.

2. Correctness verification
- Run targeted tests for touched behavior.
- Run related regression suites (CRUD, overflow, WAL/recovery, index consistency where applicable).

3. Safety verification
- Validate corruption/safety guard behavior (checksum, size limits, crash-state handling where applicable).
- Validate backward compatibility with old uncompressed files when relevant.

4. Performance verification
- Run benchmark smoke checks for modified hot paths.
- Verify no obvious regressions against baseline thresholds.

5. Integration verification
- Rebuild solution and run full test pass before final phase closure.

### 10.5 Verification commands (minimum gate)

Run these gates after each subagent completion (adjust filters to scope):

1. `dotnet build /Users/dohertj2/Desktop/CBDD/CBDD.slnx`
2. `dotnet test /Users/dohertj2/Desktop/CBDD/tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj`
3. Targeted suites for the changed area, for example:
- `dotnet test /Users/dohertj2/Desktop/CBDD/tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj --filter \"FullyQualifiedName~Compression\"`
- `dotnet test /Users/dohertj2/Desktop/CBDD/tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj --filter \"FullyQualifiedName~Compaction\"`
4. Benchmark smoke run when hot paths changed:
- `dotnet run -c Release --project /Users/dohertj2/Desktop/CBDD/tests/CBDD.Tests.Benchmark/ZB.MOM.WW.CBDD.Tests.Benchmark.csproj`

If any gate fails, the subagent task is not complete and must be returned for rework with failure notes.

## 11. Definition of Done (Release Gates)

1. Correctness
- all new compression/compaction test suites green
- no regressions in existing test suite

2. Compatibility
- old DB files readable with no migration required
- mixed-format operation validated

3. Safety
- decompression guardrails enforced
- corruption checks and deterministic failure behavior
- crash recovery scenarios covered

4. Performance
- documented benchmark deltas for write/read overhead and compaction throughput
- no pathological GC spikes under compression-enabled workloads

5. Operability
- telemetry counters exposed
- admin diagnostics available for page usage/compression/fragmentation