feat: add SequenceSet for sparse deletion tracking with secure erase (Gap 1.7)

Replace HashSet<ulong> _deleted in MsgBlock with SequenceSet — a sorted-range
list that compresses contiguous deletions into (Start, End) intervals. Adds
O(log n) Contains/Add via binary search on range count, matching Go's avl.SequenceSet
semantics with a simpler implementation.

- Add SequenceSet.cs: sorted-range compressed set with Add/Remove/Contains/Count/Clear
  and IEnumerable<ulong> in ascending order. Binary search for all O(log n) ops.
- Replace HashSet<ulong> _deleted and _skipSequences in MsgBlock with SequenceSet.
- Add secureErase parameter (default false) to MsgBlock.Delete(): when true, payload
  bytes are overwritten with RandomNumberGenerator.Fill() before the delete record is
  written, making original content unrecoverable on disk.
- Update FileStore.DeleteInBlock() to propagate secureErase flag.
- Update FileStore.EraseMsg() to use secureErase: true via block layer instead of
  delegating to RemoveMsg().
- Add SequenceSetTests.cs: 25 tests covering Add, Remove, Contains, Count, range
  compression, gap filling, bridge merges, enumeration, boundary values, round-trip.
- Add FileStoreTombstoneTrackingTests.cs: 12 tests covering SequenceSet tracking in
  MsgBlock, tombstone persistence through RebuildIndex recovery, secure erase
  payload overwrite verification, and FileStore.EraseMsg integration.

Go reference: filestore.go:5267 (removeMsg), filestore.go:5890 (eraseMsg),
              avl/seqset.go (SequenceSet).
This commit is contained in:
Joseph Doherty
2026-02-25 08:02:44 -05:00
parent 646a5eb2ae
commit cbe41d0efb
5 changed files with 1045 additions and 14 deletions

View File

@@ -732,14 +732,17 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
/// <summary>
/// Soft-deletes a message in the block that contains it.
/// When <paramref name="secureErase"/> is <c>true</c>, payload bytes are
/// overwritten with random data before the delete record is written.
/// Reference: golang/nats-server/server/filestore.go:5890 (eraseMsg).
/// </summary>
private void DeleteInBlock(ulong sequence)
private void DeleteInBlock(ulong sequence, bool secureErase = false)
{
foreach (var block in _blocks)
{
if (sequence >= block.FirstSequence && sequence <= block.LastSequence)
{
block.Delete(sequence);
block.Delete(sequence, secureErase);
return;
}
}
@@ -1386,15 +1389,25 @@ public sealed class FileStore : IStreamStore, IAsyncDisposable, IDisposable
}
/// <summary>
/// Overwrites a message with zeros and then soft-deletes it.
/// Secure-erases a message: overwrites its payload bytes with random data on disk,
/// then soft-deletes it (same in-memory semantics as <see cref="RemoveMsg"/>).
/// Returns <c>true</c> if the sequence existed and was erased.
/// Reference: golang/nats-server/server/filestore.go — EraseMsg.
/// Reference: golang/nats-server/server/filestore.go:5890 (eraseMsg).
/// </summary>
public bool EraseMsg(ulong seq)
{
// In .NET we don't do physical overwrite — just remove from the in-memory
// cache and soft-delete in the block layer (same semantics as RemoveMsg).
return RemoveMsg(seq);
if (!_messages.Remove(seq, out _))
return false;
if (_messages.Count == 0)
_first = _last + 1;
else
_first = _messages.Keys.Min();
// Secure erase: overwrite payload bytes with random data before marking deleted.
// Reference: golang/nats-server/server/filestore.go:5890 (eraseMsg).
DeleteInBlock(seq, secureErase: true);
return true;
}
/// <summary>