dohertj2/CBDD
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reformat / cleanup
All checks were successful
NuGet Publish / build-and-pack (push) Successful in 46s
Details
NuGet Publish / publish-to-gitea (push) Successful in 56s
Details

Browse Source
This commit is contained in:
Joseph Doherty
2026-02-21 08:10:36 -05:00
parent 4c6aaa5a3f
commit a70d8befae
176 changed files with 50555 additions and 49587 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
CBDD.slnx
View File

@@ -1,12 +1,12 @@
<Solution>
<Folder Name="/src/">
<Project Path="src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj" />
<Project Path="src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj" />
<Project Path="src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj" />
<Project Path="src/CBDD/ZB.MOM.WW.CBDD.csproj" />
<Project Path="src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj"/>
<Project Path="src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj"/>
<Project Path="src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj"/>
<Project Path="src/CBDD/ZB.MOM.WW.CBDD.csproj"/>
</Folder>
<Folder Name="/tests/">
<Project Path="tests/CBDD.Tests.Benchmark/ZB.MOM.WW.CBDD.Tests.Benchmark.csproj" />
<Project Path="tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj" />
<Project Path="tests/CBDD.Tests.Benchmark/ZB.MOM.WW.CBDD.Tests.Benchmark.csproj"/>
<Project Path="tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj"/>
</Folder>
</Solution>

34
README.md
View File

@@ -1,16 +1,21 @@
# CBDD
CBDD is an embedded, document-oriented database engine for .NET 10. It targets internal platform teams that need predictable ACID behavior, low-latency local persistence, and typed access patterns without running an external database server.
CBDD is an embedded, document-oriented database engine for .NET 10. It targets internal platform teams that need
predictable ACID behavior, low-latency local persistence, and typed access patterns without running an external database
server.
## Purpose And Business Context
CBDD provides a local data layer for services and tools that need transactional durability, deterministic startup, and high-throughput reads/writes. The primary business outcome is reducing operational overhead for workloads that do not require a networked database cluster.
CBDD provides a local data layer for services and tools that need transactional durability, deterministic startup, and
high-throughput reads/writes. The primary business outcome is reducing operational overhead for workloads that do not
require a networked database cluster.
## Ownership And Support
- Owning team: CBDD maintainers (repository owner: `@dohertj2`)
- Primary support path: open a Gitea issue in this repository with labels `incident` or `bug`
- Escalation path: follow [`docs/runbook.md`](docs/runbook.md) and page the release maintainer listed in the active release PR
- Escalation path: follow [`docs/runbook.md`](docs/runbook.md) and page the release maintainer listed in the active
release PR
## Architecture Overview
@@ -22,6 +27,7 @@ CBDD has four primary layers:
4. Source-generated mapping (`src/CBDD.SourceGenerators`)
Detailed architecture material:
- [`docs/architecture.md`](docs/architecture.md)
- [`RFC.md`](RFC.md)
- [`C-BSON.md`](C-BSON.md)
@@ -36,34 +42,44 @@ Detailed architecture material:
## Setup And Local Run
1. Clone the repository.
```bash
git clone https://gitea.dohertylan.com/dohertj2/CBDD.git
cd CBDD
```
Expected outcome: local repository checkout with `CBDD.slnx` present.
2. Restore dependencies.
```bash
dotnet restore
```
Expected outcome: restore completes without package errors.
3. Build the solution.
```bash
dotnet build CBDD.slnx -c Release
```
Expected outcome: solution builds without compiler errors.
4. Run tests.
```bash
dotnet test CBDD.slnx -c Release
```
Expected outcome: all tests pass.
5. Run the full repository fitness check.
```bash
bash scripts/fitness-check.sh
```
Expected outcome: format, build, tests, coverage threshold, and package checks complete.
## Configuration And Secrets
@@ -135,9 +151,12 @@ if (!result.Executed)
Common issues and remediation:
- Build/test environment failures: [`docs/troubleshooting.md#build-and-test-failures`](docs/troubleshooting.md#build-and-test-failures)
- Data-file recovery procedures: [`docs/troubleshooting.md#data-file-and-recovery-issues`](docs/troubleshooting.md#data-file-and-recovery-issues)
- Query/index behavior verification: [`docs/troubleshooting.md#query-and-index-issues`](docs/troubleshooting.md#query-and-index-issues)
- Build/test environment failures: [
`docs/troubleshooting.md#build-and-test-failures`](docs/troubleshooting.md#build-and-test-failures)
- Data-file recovery procedures: [
`docs/troubleshooting.md#data-file-and-recovery-issues`](docs/troubleshooting.md#data-file-and-recovery-issues)
- Query/index behavior verification: [
`docs/troubleshooting.md#query-and-index-issues`](docs/troubleshooting.md#query-and-index-issues)
## Change Governance
@@ -150,4 +169,5 @@ Common issues and remediation:
- Documentation home: [`docs/README.md`](docs/README.md)
- Major feature inventory: [`docs/features/README.md`](docs/features/README.md)
- Architecture decisions: [`docs/adr/0001-storage-engine-and-source-generation.md`](docs/adr/0001-storage-engine-and-source-generation.md)
- Architecture decisions: [
`docs/adr/0001-storage-engine-and-source-generation.md`](docs/adr/0001-storage-engine-and-source-generation.md)

43
src/CBDD.Bson/Document/BsonDocument.cs
View File

@@ -1,4 +1,4 @@
using System;
using System.Collections.Concurrent;
namespace ZB.MOM.WW.CBDD.Bson;
@@ -8,26 +8,26 @@ namespace ZB.MOM.WW.CBDD.Bson;
/// </summary>
public sealed class BsonDocument
{
private readonly ConcurrentDictionary<ushort, string>? _keys;
private readonly Memory<byte> _rawData;
private readonly System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? _keys;
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocument"/> class from raw BSON memory.
/// Initializes a new instance of the <see cref="BsonDocument" /> class from raw BSON memory.
/// </summary>
/// <param name="rawBsonData">The raw BSON data.</param>
/// <param name="keys">The optional key dictionary.</param>
public BsonDocument(Memory<byte> rawBsonData, System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? keys = null)
public BsonDocument(Memory<byte> rawBsonData, ConcurrentDictionary<ushort, string>? keys = null)
{
_rawData = rawBsonData;
_keys = keys;
}
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocument"/> class from raw BSON bytes.
/// Initializes a new instance of the <see cref="BsonDocument" /> class from raw BSON bytes.
/// </summary>
/// <param name="rawBsonData">The raw BSON data.</param>
/// <param name="keys">The optional key dictionary.</param>
public BsonDocument(byte[] rawBsonData, System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? keys = null)
public BsonDocument(byte[] rawBsonData, ConcurrentDictionary<ushort, string>? keys = null)
{
_rawData = rawBsonData;
_keys = keys;
@@ -46,15 +46,19 @@ public sealed class BsonDocument
/// <summary>
/// Creates a reader for this document
/// </summary>
public BsonSpanReader GetReader() => new BsonSpanReader(_rawData.Span, _keys ?? new System.Collections.Concurrent.ConcurrentDictionary<ushort, string>());
public BsonSpanReader GetReader()
{
return new BsonSpanReader(_rawData.Span,
_keys ?? new ConcurrentDictionary<ushort, string>());
}
/// <summary>
/// Tries to get a field value by name.
/// Returns false if field not found.
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise <see langword="null"/>.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <param name="value">When this method returns, contains the field value if found; otherwise <see langword="null" />.</param>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetString(string fieldName, out string? value)
{
value = null;
@@ -70,7 +74,7 @@ public sealed class BsonDocument
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.String)
{
@@ -89,7 +93,7 @@ public sealed class BsonDocument
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise zero.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetInt32(string fieldName, out int value)
{
value = 0;
@@ -105,7 +109,7 @@ public sealed class BsonDocument
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.Int32)
{
@@ -124,7 +128,7 @@ public sealed class BsonDocument
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise default.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetObjectId(string fieldName, out ObjectId value)
{
value = default;
@@ -140,7 +144,7 @@ public sealed class BsonDocument
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.ObjectId)
{
@@ -160,7 +164,8 @@ public sealed class BsonDocument
/// <param name="keyMap">The key map used for field name encoding.</param>
/// <param name="buildAction">The action that populates the builder.</param>
/// <returns>The created BSON document.</returns>
public static BsonDocument Create(System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap, Action<BsonDocumentBuilder> buildAction)
public static BsonDocument Create(ConcurrentDictionary<string, ushort> keyMap,
Action<BsonDocumentBuilder> buildAction)
{
var builder = new BsonDocumentBuilder(keyMap);
buildAction(builder);
@@ -173,15 +178,15 @@ public sealed class BsonDocument
/// </summary>
public sealed class BsonDocumentBuilder
{
private readonly ConcurrentDictionary<string, ushort> _keyMap;
private byte[] _buffer = new byte[1024]; // Start with 1KB
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, ushort> _keyMap;
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocumentBuilder"/> class.
/// Initializes a new instance of the <see cref="BsonDocumentBuilder" /> class.
/// </summary>
/// <param name="keyMap">The key map used for field name encoding.</param>
public BsonDocumentBuilder(System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap)
public BsonDocumentBuilder(ConcurrentDictionary<string, ushort> keyMap)
{
_keyMap = keyMap;
var writer = new BsonSpanWriter(_buffer, _keyMap);
@@ -270,7 +275,7 @@ public sealed class BsonDocumentBuilder
public BsonDocument Build()
{
// Layout: [int32 size][field bytes...][0x00 terminator]
var totalSize = _position + 5;
int totalSize = _position + 5;
var finalBuffer = new byte[totalSize];
BitConverter.TryWriteBytes(finalBuffer.AsSpan(0, 4), totalSize);

33
src/CBDD.Bson/IO/BsonBufferWriter.cs
View File

@@ -1,4 +1,3 @@
using System;
using System.Buffers;
using System.Buffers.Binary;
using System.Text;
@@ -11,30 +10,29 @@ namespace ZB.MOM.WW.CBDD.Bson;
/// </summary>
public ref struct BsonBufferWriter
{
private IBufferWriter<byte> _writer;
private int _totalBytesWritten;
private readonly IBufferWriter<byte> _writer;
/// <summary>
/// Initializes a new instance of the <see cref="BsonBufferWriter"/> struct.
/// Initializes a new instance of the <see cref="BsonBufferWriter" /> struct.
/// </summary>
/// <param name="writer">The buffer writer to write BSON bytes to.</param>
public BsonBufferWriter(IBufferWriter<byte> writer)
{
_writer = writer;
_totalBytesWritten = 0;
Position = 0;
}
/// <summary>
/// Gets the current write position in bytes.
/// </summary>
public int Position => _totalBytesWritten;
public int Position { get; private set; }
private void WriteBytes(ReadOnlySpan<byte> data)
{
var destination = _writer.GetSpan(data.Length);
data.CopyTo(destination);
_writer.Advance(data.Length);
_totalBytesWritten += data.Length;
Position += data.Length;
}
private void WriteByte(byte value)
@@ -42,7 +40,7 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(1);
span[0] = value;
_writer.Advance(1);
_totalBytesWritten++;
Position++;
}
/// <summary>
@@ -67,12 +65,15 @@ public ref struct BsonBufferWriter
public int BeginDocument()
{
// Write placeholder for size (4 bytes)
var sizePosition = _totalBytesWritten;
int sizePosition = Position;
var span = _writer.GetSpan(4);
// Initialize with default value (will be patched later)
span[0] = 0; span[1] = 0; span[2] = 0; span[3] = 0;
span[0] = 0;
span[1] = 0;
span[2] = 0;
span[3] = 0;
_writer.Advance(4);
_totalBytesWritten += 4;
Position += 4;
return sizePosition;
}
@@ -129,7 +130,7 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(4);
BinaryPrimitives.WriteInt32LittleEndian(span, value);
_writer.Advance(4);
_totalBytesWritten += 4;
Position += 4;
}
private void WriteInt64Internal(long value)
@@ -137,7 +138,7 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteInt64LittleEndian(span, value);
_writer.Advance(8);
_totalBytesWritten += 8;
Position += 8;
}
/// <summary>
@@ -189,7 +190,7 @@ public ref struct BsonBufferWriter
private void WriteStringValue(string value)
{
// String: length (int32) + UTF8 bytes + null terminator
var bytes = Encoding.UTF8.GetBytes(value);
byte[] bytes = Encoding.UTF8.GetBytes(value);
WriteInt32Internal(bytes.Length + 1); // +1 for null terminator
WriteBytes(bytes);
WriteByte(0);
@@ -200,7 +201,7 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteDoubleLittleEndian(span, value);
_writer.Advance(8);
_totalBytesWritten += 8;
Position += 8;
}
/// <summary>
@@ -243,7 +244,7 @@ public ref struct BsonBufferWriter
private void WriteCString(string value)
{
var bytes = Encoding.UTF8.GetBytes(value);
byte[] bytes = Encoding.UTF8.GetBytes(value);
WriteBytes(bytes);
WriteByte(0); // Null terminator
}

164
src/CBDD.Bson/IO/BsonSpanReader.cs
View File

@@ -1,5 +1,5 @@
using System;
using System.Buffers.Binary;
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
@@ -11,30 +11,29 @@ namespace ZB.MOM.WW.CBDD.Bson;
public ref struct BsonSpanReader
{
private ReadOnlySpan<byte> _buffer;
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<ushort, string> _keys;
private readonly ConcurrentDictionary<ushort, string> _keys;
/// <summary>
/// Initializes a new instance of the <see cref="BsonSpanReader"/> struct.
/// Initializes a new instance of the <see cref="BsonSpanReader" /> struct.
/// </summary>
/// <param name="buffer">The BSON buffer to read.</param>
/// <param name="keys">The reverse key dictionary used for compressed element headers.</param>
public BsonSpanReader(ReadOnlySpan<byte> buffer, System.Collections.Concurrent.ConcurrentDictionary<ushort, string> keys)
public BsonSpanReader(ReadOnlySpan<byte> buffer, ConcurrentDictionary<ushort, string> keys)
{
_buffer = buffer;
_position = 0;
Position = 0;
_keys = keys;
}
/// <summary>
/// Gets the current read position in the buffer.
/// </summary>
public int Position => _position;
public int Position { get; private set; }
/// <summary>
/// Gets the number of unread bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - _position;
public int Remaining => _buffer.Length - Position;
/// <summary>
/// Reads the document size (first 4 bytes of a BSON document)
@@ -44,8 +43,8 @@ public ref struct BsonSpanReader
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to read document size");
var size = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int size = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
return size;
}
@@ -57,8 +56,8 @@ public ref struct BsonSpanReader
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read BSON type");
var type = (BsonType)_buffer[_position];
_position++;
var type = (BsonType)_buffer[Position];
Position++;
return type;
}
@@ -67,15 +66,15 @@ public ref struct BsonSpanReader
/// </summary>
public string ReadCString()
{
var start = _position;
while (_position < _buffer.Length && _buffer[_position] != 0)
_position++;
int start = Position;
while (Position < _buffer.Length && _buffer[Position] != 0)
Position++;
if (_position >= _buffer.Length)
if (Position >= _buffer.Length)
throw new InvalidOperationException("Unterminated C-string");
var nameBytes = _buffer.Slice(start, _position - start);
_position++; // Skip null terminator
var nameBytes = _buffer.Slice(start, Position - start);
Position++; // Skip null terminator
return Encoding.UTF8.GetString(nameBytes);
}
@@ -86,15 +85,15 @@ public ref struct BsonSpanReader
/// <param name="destination">The destination character span.</param>
public int ReadCString(Span<char> destination)
{
var start = _position;
while (_position < _buffer.Length && _buffer[_position] != 0)
_position++;
int start = Position;
while (Position < _buffer.Length && _buffer[Position] != 0)
Position++;
if (_position >= _buffer.Length)
if (Position >= _buffer.Length)
throw new InvalidOperationException("Unterminated C-string");
var nameBytes = _buffer.Slice(start, _position - start);
_position++; // Skip null terminator
var nameBytes = _buffer.Slice(start, Position - start);
Position++; // Skip null terminator
return Encoding.UTF8.GetChars(nameBytes, destination);
}
@@ -104,14 +103,14 @@ public ref struct BsonSpanReader
/// </summary>
public string ReadString()
{
var length = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int length = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
if (length < 1)
throw new InvalidOperationException("Invalid string length");
var stringBytes = _buffer.Slice(_position, length - 1); // Exclude null terminator
_position += length;
var stringBytes = _buffer.Slice(Position, length - 1); // Exclude null terminator
Position += length;
return Encoding.UTF8.GetString(stringBytes);
}
@@ -124,8 +123,8 @@ public ref struct BsonSpanReader
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to read Int32");
var value = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int value = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
return value;
}
@@ -137,8 +136,8 @@ public ref struct BsonSpanReader
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Int64");
var value = BinaryPrimitives.ReadInt64LittleEndian(_buffer.Slice(_position, 8));
_position += 8;
long value = BinaryPrimitives.ReadInt64LittleEndian(_buffer.Slice(Position, 8));
Position += 8;
return value;
}
@@ -150,8 +149,8 @@ public ref struct BsonSpanReader
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Double");
var value = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
double value = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
return value;
}
@@ -162,20 +161,20 @@ public ref struct BsonSpanReader
public (double, double) ReadCoordinates()
{
// Skip array size (4 bytes)
_position += 4;
Position += 4;
// Skip element 0 header: Type(1) + Name("0\0") (3 bytes)
_position += 3;
var x = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
Position += 3;
double x = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
// Skip element 1 header: Type(1) + Name("1\0") (3 bytes)
_position += 3;
var y = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
Position += 3;
double y = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
// Skip end of array marker (1 byte)
_position++;
Position++;
return (x, y);
}
@@ -189,11 +188,11 @@ public ref struct BsonSpanReader
throw new InvalidOperationException("Not enough bytes to read Decimal128");
var bits = new int[4];
bits[0] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
bits[1] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 4, 4));
bits[2] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 8, 4));
bits[3] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 12, 4));
_position += 16;
bits[0] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
bits[1] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 4, 4));
bits[2] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 8, 4));
bits[3] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 12, 4));
Position += 16;
return new decimal(bits);
}
@@ -206,8 +205,8 @@ public ref struct BsonSpanReader
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read Boolean");
var value = _buffer[_position] != 0;
_position++;
bool value = _buffer[Position] != 0;
Position++;
return value;
}
@@ -216,7 +215,7 @@ public ref struct BsonSpanReader
/// </summary>
public DateTime ReadDateTime()
{
var milliseconds = ReadInt64();
long milliseconds = ReadInt64();
return DateTimeOffset.FromUnixTimeMilliseconds(milliseconds).UtcDateTime;
}
@@ -225,7 +224,7 @@ public ref struct BsonSpanReader
/// </summary>
public DateTimeOffset ReadDateTimeOffset()
{
var milliseconds = ReadInt64();
long milliseconds = ReadInt64();
return DateTimeOffset.FromUnixTimeMilliseconds(milliseconds);
}
@@ -234,7 +233,7 @@ public ref struct BsonSpanReader
/// </summary>
public TimeSpan ReadTimeSpan()
{
var ticks = ReadInt64();
long ticks = ReadInt64();
return TimeSpan.FromTicks(ticks);
}
@@ -243,7 +242,7 @@ public ref struct BsonSpanReader
/// </summary>
public DateOnly ReadDateOnly()
{
var dayNumber = ReadInt32();
int dayNumber = ReadInt32();
return DateOnly.FromDayNumber(dayNumber);
}
@@ -252,7 +251,7 @@ public ref struct BsonSpanReader
/// </summary>
public TimeOnly ReadTimeOnly()
{
var ticks = ReadInt64();
long ticks = ReadInt64();
return new TimeOnly(ticks);
}
@@ -272,8 +271,8 @@ public ref struct BsonSpanReader
if (Remaining < 12)
throw new InvalidOperationException("Not enough bytes to read ObjectId");
var oidBytes = _buffer.Slice(_position, 12);
_position += 12;
var oidBytes = _buffer.Slice(Position, 12);
Position += 12;
return new ObjectId(oidBytes);
}
@@ -283,19 +282,19 @@ public ref struct BsonSpanReader
/// <param name="subtype">When this method returns, contains the BSON binary subtype.</param>
public ReadOnlySpan<byte> ReadBinary(out byte subtype)
{
var length = ReadInt32();
int length = ReadInt32();
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read binary subtype");
subtype = _buffer[_position];
_position++;
subtype = _buffer[Position];
Position++;
if (Remaining < length)
throw new InvalidOperationException("Not enough bytes to read binary data");
var data = _buffer.Slice(_position, length);
_position += length;
var data = _buffer.Slice(Position, length);
Position += length;
return data;
}
@@ -308,37 +307,37 @@ public ref struct BsonSpanReader
switch (type)
{
case BsonType.Double:
_position += 8;
Position += 8;
break;
case BsonType.String:
var stringLength = ReadInt32();
_position += stringLength;
int stringLength = ReadInt32();
Position += stringLength;
break;
case BsonType.Document:
case BsonType.Array:
var docLength = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += docLength;
int docLength = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += docLength;
break;
case BsonType.Binary:
var binaryLength = ReadInt32();
_position += 1 + binaryLength; // subtype + data
int binaryLength = ReadInt32();
Position += 1 + binaryLength; // subtype + data
break;
case BsonType.ObjectId:
_position += 12;
Position += 12;
break;
case BsonType.Boolean:
_position += 1;
Position += 1;
break;
case BsonType.DateTime:
case BsonType.Int64:
case BsonType.Timestamp:
_position += 8;
Position += 8;
break;
case BsonType.Decimal128:
_position += 16;
Position += 16;
break;
case BsonType.Int32:
_position += 4;
Position += 4;
break;
case BsonType.Null:
// No data
@@ -355,8 +354,8 @@ public ref struct BsonSpanReader
{
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read byte");
var value = _buffer[_position];
_position++;
byte value = _buffer[Position];
Position++;
return value;
}
@@ -367,7 +366,7 @@ public ref struct BsonSpanReader
{
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to peek Int32");
return BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
return BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
}
/// <summary>
@@ -378,13 +377,11 @@ public ref struct BsonSpanReader
if (Remaining < 2)
throw new InvalidOperationException("Not enough bytes to read BSON element key ID");
var id = BinaryPrimitives.ReadUInt16LittleEndian(_buffer.Slice(_position, 2));
_position += 2;
ushort id = BinaryPrimitives.ReadUInt16LittleEndian(_buffer.Slice(Position, 2));
Position += 2;
if (!_keys.TryGetValue(id, out var key))
{
if (!_keys.TryGetValue(id, out string? key))
throw new InvalidOperationException($"BSON Key ID {id} not found in reverse key dictionary.");
}
return key;
}
@@ -392,5 +389,8 @@ public ref struct BsonSpanReader
/// <summary>
/// Returns a span containing all unread bytes.
/// </summary>
public ReadOnlySpan<byte> RemainingBytes() => _buffer[_position..];
public ReadOnlySpan<byte> RemainingBytes()
{
return _buffer[Position..];
}
}

176
src/CBDD.Bson/IO/BsonSpanWriter.cs
View File

@@ -1,5 +1,5 @@
using System;
using System.Buffers.Binary;
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
@@ -11,39 +11,38 @@ namespace ZB.MOM.WW.CBDD.Bson;
public ref struct BsonSpanWriter
{
private Span<byte> _buffer;
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, ushort> _keyMap;
private readonly ConcurrentDictionary<string, ushort> _keyMap;
/// <summary>
/// Initializes a new instance of the <see cref="BsonSpanWriter"/> struct.
/// Initializes a new instance of the <see cref="BsonSpanWriter" /> struct.
/// </summary>
/// <param name="buffer">The destination buffer to write BSON bytes into.</param>
/// <param name="keyMap">The cached key-name to key-id mapping.</param>
public BsonSpanWriter(Span<byte> buffer, System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap)
public BsonSpanWriter(Span<byte> buffer, ConcurrentDictionary<string, ushort> keyMap)
{
_buffer = buffer;
_keyMap = keyMap;
_position = 0;
Position = 0;
}
/// <summary>
/// Gets the current write position in the buffer.
/// </summary>
public int Position => _position;
public int Position { get; private set; }
/// <summary>
/// Gets the number of bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - _position;
public int Remaining => _buffer.Length - Position;
/// <summary>
/// Writes document size placeholder and returns the position to patch later
/// </summary>
public int WriteDocumentSizePlaceholder()
{
var sizePosition = _position;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), 0);
_position += 4;
int sizePosition = Position;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), 0);
Position += 4;
return sizePosition;
}
@@ -53,7 +52,7 @@ public ref struct BsonSpanWriter
/// <param name="sizePosition">The position where the size placeholder was written.</param>
public void PatchDocumentSize(int sizePosition)
{
var size = _position - sizePosition;
int size = Position - sizePosition;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(sizePosition, 4), size);
}
@@ -64,16 +63,15 @@ public ref struct BsonSpanWriter
/// <param name="name">The field name.</param>
public void WriteElementHeader(BsonType type, string name)
{
_buffer[_position] = (byte)type;
_position++;
_buffer[Position] = (byte)type;
Position++;
if (!_keyMap.TryGetValue(name, out var id))
{
throw new InvalidOperationException($"BSON Key '{name}' not found in dictionary cache. Ensure all keys are registered before serialization.");
}
if (!_keyMap.TryGetValue(name, out ushort id))
throw new InvalidOperationException(
$"BSON Key '{name}' not found in dictionary cache. Ensure all keys are registered before serialization.");
BinaryPrimitives.WriteUInt16LittleEndian(_buffer.Slice(_position, 2), id);
_position += 2;
BinaryPrimitives.WriteUInt16LittleEndian(_buffer.Slice(Position, 2), id);
Position += 2;
}
/// <summary>
@@ -81,10 +79,10 @@ public ref struct BsonSpanWriter
/// </summary>
private void WriteCString(string value)
{
var bytesWritten = Encoding.UTF8.GetBytes(value, _buffer[_position..]);
_position += bytesWritten;
_buffer[_position] = 0; // Null terminator
_position++;
int bytesWritten = Encoding.UTF8.GetBytes(value, _buffer[Position..]);
Position += bytesWritten;
_buffer[Position] = 0; // Null terminator
Position++;
}
/// <summary>
@@ -92,8 +90,8 @@ public ref struct BsonSpanWriter
/// </summary>
public void WriteEndOfDocument()
{
_buffer[_position] = 0;
_position++;
_buffer[Position] = 0;
Position++;
}
/// <summary>
@@ -105,17 +103,17 @@ public ref struct BsonSpanWriter
{
WriteElementHeader(BsonType.String, name);
var valueBytes = Encoding.UTF8.GetByteCount(value);
var stringLength = valueBytes + 1; // Include null terminator
int valueBytes = Encoding.UTF8.GetByteCount(value);
int stringLength = valueBytes + 1; // Include null terminator
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), stringLength);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), stringLength);
Position += 4;
Encoding.UTF8.GetBytes(value, _buffer[_position..]);
_position += valueBytes;
Encoding.UTF8.GetBytes(value, _buffer[Position..]);
Position += valueBytes;
_buffer[_position] = 0; // Null terminator
_position++;
_buffer[Position] = 0; // Null terminator
Position++;
}
/// <summary>
@@ -126,8 +124,8 @@ public ref struct BsonSpanWriter
public void WriteInt32(string name, int value)
{
WriteElementHeader(BsonType.Int32, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), value);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), value);
Position += 4;
}
/// <summary>
@@ -138,8 +136,8 @@ public ref struct BsonSpanWriter
public void WriteInt64(string name, long value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value);
Position += 8;
}
/// <summary>
@@ -150,8 +148,8 @@ public ref struct BsonSpanWriter
public void WriteDouble(string name, double value)
{
WriteElementHeader(BsonType.Double, name);
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), value);
_position += 8;
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), value);
Position += 8;
}
/// <summary>
@@ -164,32 +162,32 @@ public ref struct BsonSpanWriter
{
WriteElementHeader(BsonType.Array, name);
var startPos = _position;
_position += 4; // Placeholder for array size
int startPos = Position;
Position += 4; // Placeholder for array size
// Element 0: X
_buffer[_position++] = (byte)BsonType.Double;
_buffer[_position++] = 0x30; // '0'
_buffer[_position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), coordinates.Item1);
_position += 8;
_buffer[Position++] = (byte)BsonType.Double;
_buffer[Position++] = 0x30; // '0'
_buffer[Position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), coordinates.Item1);
Position += 8;
// Element 1: Y
_buffer[_position++] = (byte)BsonType.Double;
_buffer[_position++] = 0x31; // '1'
_buffer[_position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), coordinates.Item2);
_position += 8;
_buffer[Position++] = (byte)BsonType.Double;
_buffer[Position++] = 0x31; // '1'
_buffer[Position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), coordinates.Item2);
Position += 8;
_buffer[_position++] = 0x00; // End of array marker
_buffer[Position++] = 0x00; // End of array marker
// Patch array size
var size = _position - startPos;
int size = Position - startPos;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(startPos, 4), size);
}
/// <summary>
/// Writes a BSON Decimal128 element from a <see cref="decimal"/> value.
/// Writes a BSON Decimal128 element from a <see cref="decimal" /> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The decimal value.</param>
@@ -198,12 +196,12 @@ public ref struct BsonSpanWriter
WriteElementHeader(BsonType.Decimal128, name);
// Note: usage of C# decimal bits for round-trip fidelity within ZB.MOM.WW.CBDD.
// This makes it compatible with CBDD Reader but strictly speaking not standard IEEE 754-2008 Decimal128.
var bits = decimal.GetBits(value);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), bits[0]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 4, 4), bits[1]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 8, 4), bits[2]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 12, 4), bits[3]);
_position += 16;
int[] bits = decimal.GetBits(value);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), bits[0]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 4, 4), bits[1]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 8, 4), bits[2]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 12, 4), bits[3]);
Position += 16;
}
/// <summary>
@@ -214,8 +212,8 @@ public ref struct BsonSpanWriter
public void WriteBoolean(string name, bool value)
{
WriteElementHeader(BsonType.Boolean, name);
_buffer[_position] = (byte)(value ? 1 : 0);
_position++;
_buffer[Position] = (byte)(value ? 1 : 0);
Position++;
}
/// <summary>
@@ -226,58 +224,58 @@ public ref struct BsonSpanWriter
public void WriteDateTime(string name, DateTime value)
{
WriteElementHeader(BsonType.DateTime, name);
var milliseconds = new DateTimeOffset(value.ToUniversalTime()).ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), milliseconds);
_position += 8;
long milliseconds = new DateTimeOffset(value.ToUniversalTime()).ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), milliseconds);
Position += 8;
}
/// <summary>
/// Writes a BSON UTC datetime element from a <see cref="DateTimeOffset"/> value.
/// Writes a BSON UTC datetime element from a <see cref="DateTimeOffset" /> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date and time offset value.</param>
public void WriteDateTimeOffset(string name, DateTimeOffset value)
{
WriteElementHeader(BsonType.DateTime, name);
var milliseconds = value.ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), milliseconds);
_position += 8;
long milliseconds = value.ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), milliseconds);
Position += 8;
}
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeSpan"/>.
/// Writes a BSON int64 element containing ticks from a <see cref="TimeSpan" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time span value.</param>
public void WriteTimeSpan(string name, TimeSpan value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value.Ticks);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value.Ticks);
Position += 8;
}
/// <summary>
/// Writes a BSON int32 element containing the <see cref="DateOnly.DayNumber"/>.
/// Writes a BSON int32 element containing the <see cref="DateOnly.DayNumber" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date-only value.</param>
public void WriteDateOnly(string name, DateOnly value)
{
WriteElementHeader(BsonType.Int32, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), value.DayNumber);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), value.DayNumber);
Position += 4;
}
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeOnly"/>.
/// Writes a BSON int64 element containing ticks from a <see cref="TimeOnly" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time-only value.</param>
public void WriteTimeOnly(string name, TimeOnly value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value.Ticks);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value.Ticks);
Position += 8;
}
/// <summary>
@@ -298,8 +296,8 @@ public ref struct BsonSpanWriter
public void WriteObjectId(string name, ObjectId value)
{
WriteElementHeader(BsonType.ObjectId, name);
value.WriteTo(_buffer.Slice(_position, 12));
_position += 12;
value.WriteTo(_buffer.Slice(Position, 12));
Position += 12;
}
/// <summary>
@@ -322,14 +320,14 @@ public ref struct BsonSpanWriter
{
WriteElementHeader(BsonType.Binary, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), data.Length);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), data.Length);
Position += 4;
_buffer[_position] = subtype;
_position++;
_buffer[Position] = subtype;
Position++;
data.CopyTo(_buffer[_position..]);
_position += data.Length;
data.CopyTo(_buffer[Position..]);
Position += data.Length;
}
/// <summary>
@@ -353,7 +351,7 @@ public ref struct BsonSpanWriter
/// <summary>
/// Ends the current document
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder"/>.</param>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder" />.</param>
public void EndDocument(int sizePosition)
{
WriteEndOfDocument();
@@ -373,7 +371,7 @@ public ref struct BsonSpanWriter
/// <summary>
/// Ends the current BSON array
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder"/>.</param>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder" />.</param>
public void EndArray(int sizePosition)
{
WriteEndOfDocument();

19
src/CBDD.Bson/Metadata/Attributes.cs
View File

@@ -1,14 +1,11 @@
using System;
namespace ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Bson
[AttributeUsage(AttributeTargets.Property)]
public class BsonIdAttribute : Attribute
{
}
[AttributeUsage(AttributeTargets.Property)]
public class BsonIgnoreAttribute : Attribute
{
[AttributeUsage(AttributeTargets.Property)]
public class BsonIdAttribute : Attribute
{
}
[AttributeUsage(AttributeTargets.Property)]
public class BsonIgnoreAttribute : Attribute
{
}
}

29
src/CBDD.Bson/Schema/BsonField.cs
View File

@@ -1,6 +1,6 @@
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public partial class BsonField
public class BsonField
{
/// <summary>
/// Gets the field name.
@@ -33,7 +33,7 @@ public partial class BsonField
/// <param name="writer">The BSON writer.</param>
public void ToBson(ref BsonSpanWriter writer)
{
var size = writer.BeginDocument();
int size = writer.BeginDocument();
writer.WriteString("n", Name);
writer.WriteInt32("t", (int)Type);
writer.WriteBoolean("b", IsNullable);
@@ -44,10 +44,7 @@ public partial class BsonField
NestedSchema.ToBson(ref writer);
}
if (ArrayItemType != null)
{
writer.WriteInt32("a", (int)ArrayItemType.Value);
}
if (ArrayItemType != null) writer.WriteInt32("a", (int)ArrayItemType.Value);
writer.EndDocument(size);
}
@@ -61,9 +58,9 @@ public partial class BsonField
{
reader.ReadInt32(); // Read doc size
string name = "";
BsonType type = BsonType.Null;
bool isNullable = false;
var name = "";
var type = BsonType.Null;
var isNullable = false;
BsonSchema? nestedSchema = null;
BsonType? arrayItemType = null;
@@ -72,7 +69,7 @@ public partial class BsonField
var btype = reader.ReadBsonType();
if (btype == BsonType.EndOfDocument) break;
var key = reader.ReadElementHeader();
string key = reader.ReadElementHeader();
switch (key)
{
case "n": name = reader.ReadString(); break;
@@ -113,7 +110,7 @@ public partial class BsonField
/// Determines whether this field is equal to another field.
/// </summary>
/// <param name="other">The other field.</param>
/// <returns><see langword="true"/> if the fields are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the fields are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(BsonField? other)
{
if (other == null) return false;
@@ -121,8 +118,14 @@ public partial class BsonField
}
/// <inheritdoc />
public override bool Equals(object? obj) => Equals(obj as BsonField);
public override bool Equals(object? obj)
{
return Equals(obj as BsonField);
}
/// <inheritdoc />
public override int GetHashCode() => (int)GetHash();
public override int GetHashCode()
{
return (int)GetHash();
}
}

35
src/CBDD.Bson/Schema/BsonSchema.cs
View File

@@ -1,6 +1,6 @@
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public partial class BsonSchema
public class BsonSchema
{
/// <summary>
/// Gets or sets the schema title.
@@ -23,16 +23,17 @@ public partial class BsonSchema
/// <param name="writer">The BSON writer to write into.</param>
public void ToBson(ref BsonSpanWriter writer)
{
var size = writer.BeginDocument();
int size = writer.BeginDocument();
if (Title != null) writer.WriteString("t", Title);
if (Version != null) writer.WriteInt32("_v", Version.Value);
var fieldsSize = writer.BeginArray("f");
for (int i = 0; i < Fields.Count; i++)
int fieldsSize = writer.BeginArray("f");
for (var i = 0; i < Fields.Count; i++)
{
writer.WriteElementHeader(BsonType.Document, i.ToString());
Fields[i].ToBson(ref writer);
}
writer.EndArray(fieldsSize);
writer.EndDocument(size);
@@ -54,7 +55,7 @@ public partial class BsonSchema
var btype = reader.ReadBsonType();
if (btype == BsonType.EndOfDocument) break;
var key = reader.ReadElementHeader();
string key = reader.ReadElementHeader();
switch (key)
{
case "t": schema.Title = reader.ReadString(); break;
@@ -68,6 +69,7 @@ public partial class BsonSchema
reader.ReadElementHeader(); // index
schema.Fields.Add(BsonField.FromBson(ref reader));
}
break;
default: reader.SkipValue(btype); break;
}
@@ -84,10 +86,7 @@ public partial class BsonSchema
{
var hash = new HashCode();
hash.Add(Title);
foreach (var field in Fields)
{
hash.Add(field.GetHash());
}
foreach (var field in Fields) hash.Add(field.GetHash());
return hash.ToHashCode();
}
@@ -95,7 +94,7 @@ public partial class BsonSchema
/// Determines whether this schema is equal to another schema.
/// </summary>
/// <param name="other">The schema to compare with.</param>
/// <returns><see langword="true"/> when schemas are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when schemas are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(BsonSchema? other)
{
if (other == null) return false;
@@ -103,10 +102,16 @@ public partial class BsonSchema
}
/// <inheritdoc />
public override bool Equals(object? obj) => Equals(obj as BsonSchema);
public override bool Equals(object? obj)
{
return Equals(obj as BsonSchema);
}
/// <inheritdoc />
public override int GetHashCode() => (int)GetHash();
public override int GetHashCode()
{
return (int)GetHash();
}
/// <summary>
/// Enumerates all field keys in this schema, including nested schema keys.
@@ -118,12 +123,8 @@ public partial class BsonSchema
{
yield return field.Name;
if (field.NestedSchema != null)
{
foreach (var nestedKey in field.NestedSchema.GetAllKeys())
{
foreach (string nestedKey in field.NestedSchema.GetAllKeys())
yield return nestedKey;
}
}
}
}
}

42
src/CBDD.Bson/Types/ObjectId.cs
View File

@@ -1,4 +1,3 @@
using System;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Bson;
@@ -16,15 +15,15 @@ public readonly struct ObjectId : IEquatable<ObjectId>
/// <summary>
/// Empty ObjectId (all zeros)
/// </summary>
public static readonly ObjectId Empty = new ObjectId(0, 0);
public static readonly ObjectId Empty = new(0, 0);
/// <summary>
/// Maximum ObjectId (all 0xFF bytes) - useful for range queries
/// </summary>
public static readonly ObjectId MaxValue = new ObjectId(int.MaxValue, long.MaxValue);
public static readonly ObjectId MaxValue = new(int.MaxValue, long.MaxValue);
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId"/> struct from raw bytes.
/// Initializes a new instance of the <see cref="ObjectId" /> struct from raw bytes.
/// </summary>
/// <param name="bytes">The 12-byte ObjectId value.</param>
public ObjectId(ReadOnlySpan<byte> bytes)
@@ -37,7 +36,7 @@ public readonly struct ObjectId : IEquatable<ObjectId>
}
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId"/> struct from its components.
/// Initializes a new instance of the <see cref="ObjectId" /> struct from its components.
/// </summary>
/// <param name="timestamp">The Unix timestamp portion.</param>
/// <param name="randomAndCounter">The random and counter portion.</param>
@@ -53,7 +52,7 @@ public readonly struct ObjectId : IEquatable<ObjectId>
public static ObjectId NewObjectId()
{
var timestamp = (int)DateTimeOffset.UtcNow.ToUnixTimeSeconds();
var random = Random.Shared.NextInt64();
long random = Random.Shared.NextInt64();
return new ObjectId(timestamp, random);
}
@@ -86,21 +85,36 @@ public readonly struct ObjectId : IEquatable<ObjectId>
public DateTime Timestamp => DateTimeOffset.FromUnixTimeSeconds(_timestamp).UtcDateTime;
/// <summary>
/// Determines whether this instance and another <see cref="ObjectId"/> have the same value.
/// Determines whether this instance and another <see cref="ObjectId" /> have the same value.
/// </summary>
/// <param name="other">The object to compare with this instance.</param>
/// <returns><see langword="true"/> if the values are equal; otherwise, <see langword="false"/>.</returns>
public bool Equals(ObjectId other) =>
_timestamp == other._timestamp && _randomAndCounter == other._randomAndCounter;
/// <returns><see langword="true" /> if the values are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(ObjectId other)
{
return _timestamp == other._timestamp && _randomAndCounter == other._randomAndCounter;
}
/// <inheritdoc />
public override bool Equals(object? obj) => obj is ObjectId other && Equals(other);
public override bool Equals(object? obj)
{
return obj is ObjectId other && Equals(other);
}
/// <inheritdoc />
public override int GetHashCode() => HashCode.Combine(_timestamp, _randomAndCounter);
public override int GetHashCode()
{
return HashCode.Combine(_timestamp, _randomAndCounter);
}
public static bool operator ==(ObjectId left, ObjectId right) => left.Equals(right);
public static bool operator !=(ObjectId left, ObjectId right) => !left.Equals(right);
public static bool operator ==(ObjectId left, ObjectId right)
{
return left.Equals(right);
}
public static bool operator !=(ObjectId left, ObjectId right)
{
return !left.Equals(right);
}
/// <inheritdoc />
public override string ToString()

2
src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj
View File

@@ -22,7 +22,7 @@
</PropertyGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

64
src/CBDD.Core/CDC/ChangeStreamDispatcher.cs
View File

@@ -1,18 +1,16 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Channels;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.CDC;
internal sealed class ChangeStreamDispatcher : IDisposable
{
private readonly Channel<InternalChangeEvent> _channel;
private readonly ConcurrentDictionary<string, ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>> _subscriptions = new();
private readonly ConcurrentDictionary<string, int> _payloadWatcherCounts = new();
private readonly CancellationTokenSource _cts = new();
private readonly ConcurrentDictionary<string, int> _payloadWatcherCounts = new();
private readonly ConcurrentDictionary<string, ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>>
_subscriptions = new();
/// <summary>
/// Initializes a new change stream dispatcher.
@@ -28,6 +26,15 @@ internal sealed class ChangeStreamDispatcher : IDisposable
Task.Run(ProcessEventsAsync);
}
/// <summary>
/// Releases dispatcher resources.
/// </summary>
public void Dispose()
{
_cts.Cancel();
_cts.Dispose();
}
/// <summary>
/// Publishes a change event to subscribers.
/// </summary>
@@ -41,17 +48,17 @@ internal sealed class ChangeStreamDispatcher : IDisposable
/// Determines whether a collection has subscribers that require payloads.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true"/> if payload watchers exist; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if payload watchers exist; otherwise, <see langword="false" />.</returns>
public bool HasPayloadWatchers(string collectionName)
{
return _payloadWatcherCounts.TryGetValue(collectionName, out var count) && count > 0;
return _payloadWatcherCounts.TryGetValue(collectionName, out int count) && count > 0;
}
/// <summary>
/// Determines whether a collection has any subscribers.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true"/> if subscribers exist; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if subscribers exist; otherwise, <see langword="false" />.</returns>
public bool HasAnyWatchers(string collectionName)
{
return _subscriptions.TryGetValue(collectionName, out var subs) && !subs.IsEmpty;
@@ -63,15 +70,13 @@ internal sealed class ChangeStreamDispatcher : IDisposable
/// <param name="collectionName">The collection name to subscribe to.</param>
/// <param name="capturePayload">Whether this subscriber requires event payloads.</param>
/// <param name="writer">The destination channel writer.</param>
/// <returns>An <see cref="IDisposable"/> that removes the subscription when disposed.</returns>
/// <returns>An <see cref="IDisposable" /> that removes the subscription when disposed.</returns>
public IDisposable Subscribe(string collectionName, bool capturePayload, ChannelWriter<InternalChangeEvent> writer)
{
if (capturePayload)
{
_payloadWatcherCounts.AddOrUpdate(collectionName, 1, (_, count) => count + 1);
}
if (capturePayload) _payloadWatcherCounts.AddOrUpdate(collectionName, 1, (_, count) => count + 1);
var collectionSubs = _subscriptions.GetOrAdd(collectionName, _ => new ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>());
var collectionSubs = _subscriptions.GetOrAdd(collectionName,
_ => new ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>());
collectionSubs.TryAdd(writer, 0);
return new Subscription(() => Unsubscribe(collectionName, capturePayload, writer));
@@ -79,15 +84,9 @@ internal sealed class ChangeStreamDispatcher : IDisposable
private void Unsubscribe(string collectionName, bool capturePayload, ChannelWriter<InternalChangeEvent> writer)
{
if (_subscriptions.TryGetValue(collectionName, out var collectionSubs))
{
collectionSubs.TryRemove(writer, out _);
}
if (_subscriptions.TryGetValue(collectionName, out var collectionSubs)) collectionSubs.TryRemove(writer, out _);
if (capturePayload)
{
_payloadWatcherCounts.AddOrUpdate(collectionName, 0, (_, count) => Math.Max(0, count - 1));
}
if (capturePayload) _payloadWatcherCounts.AddOrUpdate(collectionName, 0, (_, count) => Math.Max(0, count - 1));
}
private async Task ProcessEventsAsync()
@@ -96,38 +95,23 @@ internal sealed class ChangeStreamDispatcher : IDisposable
{
var reader = _channel.Reader;
while (await reader.WaitToReadAsync(_cts.Token))
{
while (reader.TryRead(out var @event))
{
if (_subscriptions.TryGetValue(@event.CollectionName, out var collectionSubs))
{
foreach (var writer in collectionSubs.Keys)
{
// Optimized fan-out: non-blocking TryWrite.
// If a subscriber channel is full (unlikely with Unbounded),
// we skip or drop. Usually, subscribers will also use Unbounded.
writer.TryWrite(@event);
}
catch (OperationCanceledException)
{
}
}
}
}
catch (OperationCanceledException) { }
catch (Exception)
{
// Internal error logging could go here
}
}
/// <summary>
/// Releases dispatcher resources.
/// </summary>
public void Dispose()
{
_cts.Cancel();
_cts.Dispose();
}
private sealed class Subscription : IDisposable
{
private readonly Action _onDispose;

1
src/CBDD.Core/CDC/ChangeStreamEvent.cs
View File

@@ -1,4 +1,3 @@
using System;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.CDC;

32
src/CBDD.Core/CDC/ChangeStreamObservable.cs
View File

@@ -1,9 +1,5 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading.Channels;
using System.Threading.Tasks;
using System.Threading;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
@@ -12,11 +8,11 @@ namespace ZB.MOM.WW.CBDD.Core.CDC;
internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamEvent<TId, T>> where T : class
{
private readonly ChangeStreamDispatcher _dispatcher;
private readonly string _collectionName;
private readonly bool _capturePayload;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly string _collectionName;
private readonly ChangeStreamDispatcher _dispatcher;
private readonly ConcurrentDictionary<ushort, string> _keyReverseMap;
private readonly IDocumentMapper<TId, T> _mapper;
/// <summary>
/// Initializes a new observable wrapper for collection change events.
@@ -60,14 +56,13 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
return new CompositeDisposable(dispatcherSubscription, cts, channel.Writer, bridgeTask);
}
private async Task BridgeChannelToObserverAsync(ChannelReader<InternalChangeEvent> reader, IObserver<ChangeStreamEvent<TId, T>> observer, CancellationToken ct)
private async Task BridgeChannelToObserverAsync(ChannelReader<InternalChangeEvent> reader,
IObserver<ChangeStreamEvent<TId, T>> observer, CancellationToken ct)
{
try
{
while (await reader.WaitToReadAsync(ct))
{
while (reader.TryRead(out var internalEvent))
{
try
{
// Deserializza ID
@@ -76,9 +71,8 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
// Deserializza Payload (se presente)
T? entity = default;
if (internalEvent.PayloadBytes.HasValue)
{
entity = _mapper.Deserialize(new BsonSpanReader(internalEvent.PayloadBytes.Value.Span, _keyReverseMap));
}
entity = _mapper.Deserialize(new BsonSpanReader(internalEvent.PayloadBytes.Value.Span,
_keyReverseMap));
var externalEvent = new ChangeStreamEvent<TId, T>
{
@@ -98,8 +92,7 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
// Or we can stop the observer.
observer.OnError(ex);
}
}
}
observer.OnCompleted();
}
catch (OperationCanceledException)
@@ -114,10 +107,10 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
private sealed class CompositeDisposable : IDisposable
{
private readonly IDisposable _dispatcherSubscription;
private readonly CancellationTokenSource _cts;
private readonly ChannelWriter<InternalChangeEvent> _writer;
private readonly Task _bridgeTask;
private readonly CancellationTokenSource _cts;
private readonly IDisposable _dispatcherSubscription;
private readonly ChannelWriter<InternalChangeEvent> _writer;
private bool _disposed;
/// <summary>
@@ -127,7 +120,8 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
/// <param name="cts">The cancellation source controlling the bridge task.</param>
/// <param name="writer">The channel writer for internal change events.</param>
/// <param name="bridgeTask">The running bridge task.</param>
public CompositeDisposable(IDisposable dispatcherSubscription, CancellationTokenSource cts, ChannelWriter<InternalChangeEvent> writer, Task bridgeTask)
public CompositeDisposable(IDisposable dispatcherSubscription, CancellationTokenSource cts,
ChannelWriter<InternalChangeEvent> writer, Task bridgeTask)
{
_dispatcherSubscription = dispatcherSubscription;
_cts = cts;

16
src/CBDD.Core/CDC/CollectionCdcPublisher.cs
View File

@@ -1,6 +1,5 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.CDC;
@@ -13,14 +12,14 @@ namespace ZB.MOM.WW.CBDD.Core.CDC;
/// <typeparam name="T">Document type.</typeparam>
internal sealed class CollectionCdcPublisher<TId, T> where T : class
{
private readonly ITransactionHolder _transactionHolder;
private readonly string _collectionName;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly ChangeStreamDispatcher? _dispatcher;
private readonly ConcurrentDictionary<ushort, string> _keyReverseMap;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly ITransactionHolder _transactionHolder;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionCdcPublisher"/> class.
/// Initializes a new instance of the <see cref="CollectionCdcPublisher" /> class.
/// </summary>
/// <param name="transactionHolder">The transaction holder.</param>
/// <param name="collectionName">The collection name.</param>
@@ -74,15 +73,11 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
return;
ReadOnlyMemory<byte>? payload = null;
if (!docData.IsEmpty && _dispatcher.HasPayloadWatchers(_collectionName))
{
payload = docData.ToArray();
}
if (!docData.IsEmpty && _dispatcher.HasPayloadWatchers(_collectionName)) payload = docData.ToArray();
var idBytes = _mapper.ToIndexKey(id).Data.ToArray();
byte[] idBytes = _mapper.ToIndexKey(id).Data.ToArray();
if (transaction is Transaction t)
{
t.AddChange(new InternalChangeEvent
{
Timestamp = DateTime.UtcNow.Ticks,
@@ -93,5 +88,4 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
PayloadBytes = payload
});
}
}
}

61
src/CBDD.Core/Collections/BaseMappers.cs
View File

@@ -1,10 +1,6 @@
using System;
using System.Buffers;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using System.Linq;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson.Schema;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Collections;
@@ -52,14 +48,20 @@ public abstract class DocumentMapperBase<TId, T> : IDocumentMapper<TId, T> where
/// </summary>
/// <param name="id">The identifier value.</param>
/// <returns>The index key representation of the identifier.</returns>
public virtual IndexKey ToIndexKey(TId id) => IndexKey.Create(id);
public virtual IndexKey ToIndexKey(TId id)
{
return IndexKey.Create(id);
}
/// <summary>
/// Converts an index key back into a typed identifier value.
/// </summary>
/// <param name="key">The index key to convert.</param>
/// <returns>The typed identifier value.</returns>
public virtual TId FromIndexKey(IndexKey key) => key.As<TId>();
public virtual TId FromIndexKey(IndexKey key)
{
return key.As<TId>();
}
/// <summary>
/// Gets all mapped field keys used by this mapper.
@@ -70,7 +72,10 @@ public abstract class DocumentMapperBase<TId, T> : IDocumentMapper<TId, T> where
/// Builds the BSON schema for the mapped entity type.
/// </summary>
/// <returns>The generated BSON schema.</returns>
public virtual BsonSchema GetSchema() => BsonSchemaGenerator.FromType<T>();
public virtual BsonSchema GetSchema()
{
return BsonSchemaGenerator.FromType<T>();
}
}
/// <summary>
@@ -79,10 +84,16 @@ public abstract class DocumentMapperBase<TId, T> : IDocumentMapper<TId, T> where
public abstract class ObjectIdMapperBase<T> : DocumentMapperBase<ObjectId, T>, IDocumentMapper<T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(ObjectId id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(ObjectId id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override ObjectId FromIndexKey(IndexKey key) => key.As<ObjectId>();
public override ObjectId FromIndexKey(IndexKey key)
{
return key.As<ObjectId>();
}
}
/// <summary>
@@ -91,10 +102,16 @@ public abstract class ObjectIdMapperBase<T> : DocumentMapperBase<ObjectId, T>, I
public abstract class Int32MapperBase<T> : DocumentMapperBase<int, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(int id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(int id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override int FromIndexKey(IndexKey key) => key.As<int>();
public override int FromIndexKey(IndexKey key)
{
return key.As<int>();
}
}
/// <summary>
@@ -103,10 +120,16 @@ public abstract class Int32MapperBase<T> : DocumentMapperBase<int, T> where T :
public abstract class StringMapperBase<T> : DocumentMapperBase<string, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(string id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(string id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override string FromIndexKey(IndexKey key) => key.As<string>();
public override string FromIndexKey(IndexKey key)
{
return key.As<string>();
}
}
/// <summary>
@@ -115,8 +138,14 @@ public abstract class StringMapperBase<T> : DocumentMapperBase<string, T> where
public abstract class GuidMapperBase<T> : DocumentMapperBase<Guid, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(Guid id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(Guid id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override Guid FromIndexKey(IndexKey key) => key.As<Guid>();
public override Guid FromIndexKey(IndexKey key)
{
return key.As<Guid>();
}
}

23
src/CBDD.Core/Collections/BsonSchemaGenerator.cs
View File

@@ -1,16 +1,15 @@
using System.Reflection;
using System.Linq;
using System.Collections;
using System.Collections.Generic;
using System.Collections.Concurrent;
using System.Reflection;
using ZB.MOM.WW.CBDD.Bson;
using System;
using ZB.MOM.WW.CBDD.Bson.Schema;
namespace ZB.MOM.WW.CBDD.Core.Collections;
public static class BsonSchemaGenerator
{
private static readonly ConcurrentDictionary<Type, BsonSchema> _cache = new();
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// </summary>
@@ -21,8 +20,6 @@ public static class BsonSchemaGenerator
return FromType(typeof(T));
}
private static readonly ConcurrentDictionary<Type, BsonSchema> _cache = new();
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// </summary>
@@ -47,10 +44,7 @@ public static class BsonSchemaGenerator
AddField(schema, prop.Name, prop.PropertyType);
}
foreach (var field in fields)
{
AddField(schema, field.Name, field.FieldType);
}
foreach (var field in fields) AddField(schema, field.Name, field.FieldType);
return schema;
}
@@ -60,10 +54,7 @@ public static class BsonSchemaGenerator
name = name.ToLowerInvariant();
// Convention: id -> _id for root document
if (name.Equals("id", StringComparison.OrdinalIgnoreCase))
{
name = "_id";
}
if (name.Equals("id", StringComparison.OrdinalIgnoreCase)) name = "_id";
var (bsonType, nestedSchema, itemType) = GetBsonType(type);
@@ -106,11 +97,9 @@ public static class BsonSchemaGenerator
// Nested Objects / Structs
// If it's not a string, not a primitive, and not an array/list, treat as Document
if (type != typeof(string) && !type.IsPrimitive && !type.IsEnum)
{
// Avoid infinite recursion?
// Simple approach: generating nested schema
return (BsonType.Document, FromType(type), null);
}
return (BsonType.Undefined, null, null);
}
@@ -126,9 +115,7 @@ public static class BsonSchemaGenerator
// If type itself is IEnumerable<T>
if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof(IEnumerable<>))
{
return type.GetGenericArguments()[0];
}
var enumerableType = type.GetInterfaces()
.FirstOrDefault(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IEnumerable<>));

18
src/CBDD.Core/Collections/DocumentCollection.Scan.cs
View File

@@ -18,8 +18,8 @@ public partial class DocumentCollection<TId, T> where T : class
if (predicate == null) throw new ArgumentNullException(nameof(predicate));
var transaction = _transactionHolder.GetCurrentTransactionOrStart();
var txnId = transaction.TransactionId;
var pageCount = _storage.PageCount;
ulong txnId = transaction.TransactionId;
uint pageCount = _storage.PageCount;
var buffer = new byte[_storage.PageSize];
var pageResults = new List<T>();
@@ -28,10 +28,7 @@ public partial class DocumentCollection<TId, T> where T : class
pageResults.Clear();
ScanPage(pageId, txnId, buffer, predicate, pageResults);
foreach (var doc in pageResults)
{
yield return doc;
}
foreach (var doc in pageResults) yield return doc;
}
}
@@ -46,7 +43,7 @@ public partial class DocumentCollection<TId, T> where T : class
if (predicate == null) throw new ArgumentNullException(nameof(predicate));
var transaction = _transactionHolder.GetCurrentTransactionOrStart();
var txnId = transaction.TransactionId;
ulong txnId = transaction.TransactionId;
var pageCount = (int)_storage.PageCount;
if (degreeOfParallelism <= 0)
@@ -61,15 +58,14 @@ public partial class DocumentCollection<TId, T> where T : class
var localResults = new List<T>();
for (int i = range.Item1; i < range.Item2; i++)
{
ScanPage((uint)i, txnId, localBuffer, predicate, localResults);
}
return localResults;
});
}
private void ScanPage(uint pageId, ulong txnId, byte[] buffer, Func<BsonSpanReader, bool> predicate, List<T> results)
private void ScanPage(uint pageId, ulong txnId, byte[] buffer, Func<BsonSpanReader, bool> predicate,
List<T> results)
{
_storage.ReadPage(pageId, txnId, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
@@ -80,7 +76,7 @@ public partial class DocumentCollection<TId, T> where T : class
var slots = MemoryMarshal.Cast<byte, SlotEntry>(
buffer.AsSpan(SlottedPageHeader.Size, header.SlotCount * SlotEntry.Size));
for (int i = 0; i < header.SlotCount; i++)
for (var i = 0; i < header.SlotCount; i++)
{
var slot = slots[i];

1230
src/CBDD.Core/Collections/DocumentCollection.cs
View File

File diff suppressed because it is too large Load Diff

5
src/CBDD.Core/Collections/IDocumentMapper.cs
View File

@@ -1,9 +1,6 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using System;
using System.Buffers;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson.Schema;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Collections;

9
src/CBDD.Core/Collections/SchemaVersion.cs
View File

@@ -1,5 +1,3 @@
using System;
namespace ZB.MOM.WW.CBDD.Core.Collections;
public readonly struct SchemaVersion
@@ -15,7 +13,7 @@ public readonly struct SchemaVersion
public long Hash { get; }
/// <summary>
/// Initializes a new instance of the <see cref="SchemaVersion"/> struct.
/// Initializes a new instance of the <see cref="SchemaVersion" /> struct.
/// </summary>
/// <param name="version">The schema version number.</param>
/// <param name="hash">The schema hash.</param>
@@ -26,5 +24,8 @@ public readonly struct SchemaVersion
}
/// <inheritdoc />
public override string ToString() => $"v{Version} (0x{Hash:X16})";
public override string ToString()
{
return $"v{Version} (0x{Hash:X16})";
}
}

29
src/CBDD.Core/Compression/CompressedPayloadHeader.cs
View File

@@ -30,7 +30,7 @@ public readonly struct CompressedPayloadHeader
public uint Checksum { get; }
/// <summary>
/// Initializes a new instance of the <see cref="CompressedPayloadHeader"/> class.
/// Initializes a new instance of the <see cref="CompressedPayloadHeader" /> class.
/// </summary>
/// <param name="codec">Compression codec used for payload bytes.</param>
/// <param name="originalLength">Original uncompressed payload length.</param>
@@ -55,9 +55,10 @@ public readonly struct CompressedPayloadHeader
/// <param name="codec">Compression codec used for payload bytes.</param>
/// <param name="originalLength">Original uncompressed payload length.</param>
/// <param name="compressedPayload">Compressed payload bytes.</param>
public static CompressedPayloadHeader Create(CompressionCodec codec, int originalLength, ReadOnlySpan<byte> compressedPayload)
public static CompressedPayloadHeader Create(CompressionCodec codec, int originalLength,
ReadOnlySpan<byte> compressedPayload)
{
var checksum = ComputeChecksum(compressedPayload);
uint checksum = ComputeChecksum(compressedPayload);
return new CompressedPayloadHeader(codec, originalLength, compressedPayload.Length, checksum);
}
@@ -89,9 +90,9 @@ public readonly struct CompressedPayloadHeader
throw new ArgumentException($"Source must be at least {Size} bytes.", nameof(source));
var codec = (CompressionCodec)source[0];
var originalLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(4, 4));
var compressedLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(8, 4));
var checksum = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(12, 4));
int originalLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(4, 4));
int compressedLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(8, 4));
uint checksum = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(12, 4));
return new CompressedPayloadHeader(codec, originalLength, compressedLength, checksum);
}
@@ -108,7 +109,10 @@ public readonly struct CompressedPayloadHeader
/// Compute Checksum.
/// </summary>
/// <param name="payload">Payload bytes.</param>
public static uint ComputeChecksum(ReadOnlySpan<byte> payload) => Crc32Calculator.Compute(payload);
public static uint ComputeChecksum(ReadOnlySpan<byte> payload)
{
return Crc32Calculator.Compute(payload);
}
private static class Crc32Calculator
{
@@ -121,10 +125,10 @@ public readonly struct CompressedPayloadHeader
/// <param name="payload">Payload bytes.</param>
public static uint Compute(ReadOnlySpan<byte> payload)
{
uint crc = 0xFFFFFFFFu;
for (int i = 0; i < payload.Length; i++)
var crc = 0xFFFFFFFFu;
for (var i = 0; i < payload.Length; i++)
{
var index = (crc ^ payload[i]) & 0xFF;
uint index = (crc ^ payload[i]) & 0xFF;
crc = (crc >> 8) ^ Table[index];
}
@@ -137,10 +141,7 @@ public readonly struct CompressedPayloadHeader
for (uint i = 0; i < table.Length; i++)
{
uint value = i;
for (int bit = 0; bit < 8; bit++)
{
value = (value & 1) != 0 ? (value >> 1) ^ Polynomial : value >> 1;
}
for (var bit = 0; bit < 8; bit++) value = (value & 1) != 0 ? (value >> 1) ^ Polynomial : value >> 1;
table[i] = value;
}

9
src/CBDD.Core/Compression/CompressionOptions.cs
View File

@@ -59,16 +59,19 @@ public sealed class CompressionOptions
throw new ArgumentOutOfRangeException(nameof(MinSizeBytes), "MinSizeBytes must be non-negative.");
if (candidate.MinSavingsPercent is < 0 or > 100)
throw new ArgumentOutOfRangeException(nameof(MinSavingsPercent), "MinSavingsPercent must be between 0 and 100.");
throw new ArgumentOutOfRangeException(nameof(MinSavingsPercent),
"MinSavingsPercent must be between 0 and 100.");
if (!Enum.IsDefined(candidate.Codec))
throw new ArgumentOutOfRangeException(nameof(Codec), $"Unsupported codec: {candidate.Codec}.");
if (candidate.MaxDecompressedSizeBytes <= 0)
throw new ArgumentOutOfRangeException(nameof(MaxDecompressedSizeBytes), "MaxDecompressedSizeBytes must be greater than 0.");
throw new ArgumentOutOfRangeException(nameof(MaxDecompressedSizeBytes),
"MaxDecompressedSizeBytes must be greater than 0.");
if (candidate.MaxCompressionInputBytes is <= 0)
throw new ArgumentOutOfRangeException(nameof(MaxCompressionInputBytes), "MaxCompressionInputBytes must be greater than 0 when provided.");
throw new ArgumentOutOfRangeException(nameof(MaxCompressionInputBytes),
"MaxCompressionInputBytes must be greater than 0 when provided.");
return candidate;
}

170
src/CBDD.Core/Compression/CompressionService.cs
View File

@@ -12,7 +12,7 @@ public sealed class CompressionService
private readonly ConcurrentDictionary<CompressionCodec, ICompressionCodec> _codecs = new();
/// <summary>
/// Initializes a new instance of the <see cref="CompressionService"/> class.
/// Initializes a new instance of the <see cref="CompressionService" /> class.
/// </summary>
/// <param name="additionalCodecs">Optional additional codecs to register.</param>
public CompressionService(IEnumerable<ICompressionCodec>? additionalCodecs = null)
@@ -24,10 +24,7 @@ public sealed class CompressionService
if (additionalCodecs == null)
return;
foreach (var codec in additionalCodecs)
{
RegisterCodec(codec);
}
foreach (var codec in additionalCodecs) RegisterCodec(codec);
}
/// <summary>
@@ -45,7 +42,10 @@ public sealed class CompressionService
/// </summary>
/// <param name="codec">The codec identifier to resolve.</param>
/// <param name="compressionCodec">When this method returns, contains the resolved codec when found.</param>
/// <returns><see langword="true"/> when a codec is registered for <paramref name="codec"/>; otherwise, <see langword="false"/>.</returns>
/// <returns>
/// <see langword="true" /> when a codec is registered for <paramref name="codec" />; otherwise,
/// <see langword="false" />.
/// </returns>
public bool TryGetCodec(CompressionCodec codec, out ICompressionCodec compressionCodec)
{
return _codecs.TryGetValue(codec, out compressionCodec!);
@@ -81,10 +81,14 @@ public sealed class CompressionService
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="codec">The codec to use.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, CompressionCodec codec, int expectedLength, int maxDecompressedSizeBytes)
public byte[] Decompress(ReadOnlySpan<byte> input, CompressionCodec codec, int expectedLength,
int maxDecompressedSizeBytes)
{
return GetCodec(codec).Decompress(input, expectedLength, maxDecompressedSizeBytes);
}
@@ -97,12 +101,70 @@ public sealed class CompressionService
/// <param name="level">The compression level.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes after roundtrip.</returns>
public byte[] Roundtrip(ReadOnlySpan<byte> input, CompressionCodec codec, CompressionLevel level, int maxDecompressedSizeBytes)
public byte[] Roundtrip(ReadOnlySpan<byte> input, CompressionCodec codec, CompressionLevel level,
int maxDecompressedSizeBytes)
{
var compressed = Compress(input, codec, level);
byte[] compressed = Compress(input, codec, level);
return Decompress(compressed, codec, input.Length, maxDecompressedSizeBytes);
}
private static byte[] CompressWithCodecStream(ReadOnlySpan<byte> input, Func<Stream, Stream> streamFactory)
{
using var output = new MemoryStream(input.Length);
using (var codecStream = streamFactory(output))
{
codecStream.Write(input);
codecStream.Flush();
}
return output.ToArray();
}
private static byte[] DecompressWithCodecStream(
ReadOnlySpan<byte> input,
Func<Stream, Stream> streamFactory,
int expectedLength,
int maxDecompressedSizeBytes)
{
if (maxDecompressedSizeBytes <= 0)
throw new ArgumentOutOfRangeException(nameof(maxDecompressedSizeBytes));
using var compressed = new MemoryStream(input.ToArray(), false);
using var codecStream = streamFactory(compressed);
using var output = expectedLength > 0
? new MemoryStream(expectedLength)
: new MemoryStream();
byte[] buffer = ArrayPool<byte>.Shared.Rent(8192);
try
{
var totalWritten = 0;
while (true)
{
int bytesRead = codecStream.Read(buffer, 0, buffer.Length);
if (bytesRead <= 0)
break;
totalWritten += bytesRead;
if (totalWritten > maxDecompressedSizeBytes)
throw new InvalidDataException(
$"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
output.Write(buffer, 0, bytesRead);
}
if (expectedLength >= 0 && totalWritten != expectedLength)
throw new InvalidDataException(
$"Expected decompressed length {expectedLength}, actual {totalWritten}.");
return output.ToArray();
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
}
}
private sealed class NoneCompressionCodec : ICompressionCodec
{
/// <summary>
@@ -116,7 +178,10 @@ public sealed class CompressionService
/// <param name="input">The payload bytes to copy.</param>
/// <param name="level">The requested compression level.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level) => input.ToArray();
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return input.ToArray();
}
/// <summary>
/// Validates and returns an uncompressed payload copy.
@@ -128,10 +193,12 @@ public sealed class CompressionService
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
if (input.Length > maxDecompressedSizeBytes)
throw new InvalidDataException($"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
throw new InvalidDataException(
$"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
if (expectedLength >= 0 && expectedLength != input.Length)
throw new InvalidDataException($"Expected decompressed length {expectedLength}, actual {input.Length}.");
throw new InvalidDataException(
$"Expected decompressed length {expectedLength}, actual {input.Length}.");
return input.ToArray();
}
@@ -152,19 +219,24 @@ public sealed class CompressionService
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new BrotliStream(stream, level, leaveOpen: true));
return CompressWithCodecStream(input, stream => new BrotliStream(stream, level, true));
}
/// <summary>
/// Decompresses Brotli-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input, stream => new BrotliStream(stream, CompressionMode.Decompress, leaveOpen: true), expectedLength, maxDecompressedSizeBytes);
return DecompressWithCodecStream(input,
stream => new BrotliStream(stream, CompressionMode.Decompress, true), expectedLength,
maxDecompressedSizeBytes);
}
}
@@ -183,74 +255,24 @@ public sealed class CompressionService
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new DeflateStream(stream, level, leaveOpen: true));
return CompressWithCodecStream(input, stream => new DeflateStream(stream, level, true));
}
/// <summary>
/// Decompresses Deflate-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input, stream => new DeflateStream(stream, CompressionMode.Decompress, leaveOpen: true), expectedLength, maxDecompressedSizeBytes);
}
}
private static byte[] CompressWithCodecStream(ReadOnlySpan<byte> input, Func<Stream, Stream> streamFactory)
{
using var output = new MemoryStream(capacity: input.Length);
using (var codecStream = streamFactory(output))
{
codecStream.Write(input);
codecStream.Flush();
}
return output.ToArray();
}
private static byte[] DecompressWithCodecStream(
ReadOnlySpan<byte> input,
Func<Stream, Stream> streamFactory,
int expectedLength,
int maxDecompressedSizeBytes)
{
if (maxDecompressedSizeBytes <= 0)
throw new ArgumentOutOfRangeException(nameof(maxDecompressedSizeBytes));
using var compressed = new MemoryStream(input.ToArray(), writable: false);
using var codecStream = streamFactory(compressed);
using var output = expectedLength > 0
? new MemoryStream(capacity: expectedLength)
: new MemoryStream();
var buffer = ArrayPool<byte>.Shared.Rent(8192);
try
{
int totalWritten = 0;
while (true)
{
var bytesRead = codecStream.Read(buffer, 0, buffer.Length);
if (bytesRead <= 0)
break;
totalWritten += bytesRead;
if (totalWritten > maxDecompressedSizeBytes)
throw new InvalidDataException($"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
output.Write(buffer, 0, bytesRead);
}
if (expectedLength >= 0 && totalWritten != expectedLength)
throw new InvalidDataException($"Expected decompressed length {expectedLength}, actual {totalWritten}.");
return output.ToArray();
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
return DecompressWithCodecStream(input,
stream => new DeflateStream(stream, CompressionMode.Decompress, true), expectedLength,
maxDecompressedSizeBytes);
}
}
}

7
src/CBDD.Core/Compression/CompressionStats.cs
View File

@@ -9,30 +9,37 @@ public readonly struct CompressionStats
/// Gets or sets the CompressedDocumentCount.
/// </summary>
public long CompressedDocumentCount { get; init; }
/// <summary>
/// Gets or sets the BytesBeforeCompression.
/// </summary>
public long BytesBeforeCompression { get; init; }
/// <summary>
/// Gets or sets the BytesAfterCompression.
/// </summary>
public long BytesAfterCompression { get; init; }
/// <summary>
/// Gets or sets the CompressionCpuTicks.
/// </summary>
public long CompressionCpuTicks { get; init; }
/// <summary>
/// Gets or sets the DecompressionCpuTicks.
/// </summary>
public long DecompressionCpuTicks { get; init; }
/// <summary>
/// Gets or sets the CompressionFailureCount.
/// </summary>
public long CompressionFailureCount { get; init; }
/// <summary>
/// Gets or sets the ChecksumFailureCount.
/// </summary>
public long ChecksumFailureCount { get; init; }
/// <summary>
/// Gets or sets the SafetyLimitRejectionCount.
/// </summary>

67
src/CBDD.Core/Compression/CompressionTelemetry.cs
View File

@@ -1,5 +1,3 @@
using System.Threading;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
@@ -7,21 +5,21 @@ namespace ZB.MOM.WW.CBDD.Core.Compression;
/// </summary>
public sealed class CompressionTelemetry
{
private long _checksumFailureCount;
private long _compressedDocumentCount;
private long _compressionAttempts;
private long _compressionSuccesses;
private long _compressionCpuTicks;
private long _compressionFailures;
private long _compressionSkippedTooSmall;
private long _compressionSkippedInsufficientSavings;
private long _decompressionAttempts;
private long _decompressionSuccesses;
private long _decompressionFailures;
private long _compressionInputBytes;
private long _compressionOutputBytes;
private long _decompressionOutputBytes;
private long _compressedDocumentCount;
private long _compressionCpuTicks;
private long _compressionSkippedInsufficientSavings;
private long _compressionSkippedTooSmall;
private long _compressionSuccesses;
private long _decompressionAttempts;
private long _decompressionCpuTicks;
private long _checksumFailureCount;
private long _decompressionFailures;
private long _decompressionOutputBytes;
private long _decompressionSuccesses;
private long _safetyLimitRejectionCount;
/// <summary>
@@ -128,44 +126,68 @@ public sealed class CompressionTelemetry
/// <summary>
/// Records a failed compression operation.
/// </summary>
public void RecordCompressionFailure() => Interlocked.Increment(ref _compressionFailures);
public void RecordCompressionFailure()
{
Interlocked.Increment(ref _compressionFailures);
}
/// <summary>
/// Records that compression was skipped because the payload was too small.
/// </summary>
public void RecordCompressionSkippedTooSmall() => Interlocked.Increment(ref _compressionSkippedTooSmall);
public void RecordCompressionSkippedTooSmall()
{
Interlocked.Increment(ref _compressionSkippedTooSmall);
}
/// <summary>
/// Records that compression was skipped due to insufficient expected savings.
/// </summary>
public void RecordCompressionSkippedInsufficientSavings() => Interlocked.Increment(ref _compressionSkippedInsufficientSavings);
public void RecordCompressionSkippedInsufficientSavings()
{
Interlocked.Increment(ref _compressionSkippedInsufficientSavings);
}
/// <summary>
/// Records a decompression attempt.
/// </summary>
public void RecordDecompressionAttempt() => Interlocked.Increment(ref _decompressionAttempts);
public void RecordDecompressionAttempt()
{
Interlocked.Increment(ref _decompressionAttempts);
}
/// <summary>
/// Adds CPU ticks spent performing compression.
/// </summary>
/// <param name="ticks">The CPU ticks to add.</param>
public void RecordCompressionCpuTicks(long ticks) => Interlocked.Add(ref _compressionCpuTicks, ticks);
public void RecordCompressionCpuTicks(long ticks)
{
Interlocked.Add(ref _compressionCpuTicks, ticks);
}
/// <summary>
/// Adds CPU ticks spent performing decompression.
/// </summary>
/// <param name="ticks">The CPU ticks to add.</param>
public void RecordDecompressionCpuTicks(long ticks) => Interlocked.Add(ref _decompressionCpuTicks, ticks);
public void RecordDecompressionCpuTicks(long ticks)
{
Interlocked.Add(ref _decompressionCpuTicks, ticks);
}
/// <summary>
/// Records a checksum validation failure.
/// </summary>
public void RecordChecksumFailure() => Interlocked.Increment(ref _checksumFailureCount);
public void RecordChecksumFailure()
{
Interlocked.Increment(ref _checksumFailureCount);
}
/// <summary>
/// Records a decompression rejection due to safety limits.
/// </summary>
public void RecordSafetyLimitRejection() => Interlocked.Increment(ref _safetyLimitRejectionCount);
public void RecordSafetyLimitRejection()
{
Interlocked.Increment(ref _safetyLimitRejectionCount);
}
/// <summary>
/// Records a successful decompression operation.
@@ -180,7 +202,10 @@ public sealed class CompressionTelemetry
/// <summary>
/// Records a failed decompression operation.
/// </summary>
public void RecordDecompressionFailure() => Interlocked.Increment(ref _decompressionFailures);
public void RecordDecompressionFailure()
{
Interlocked.Increment(ref _decompressionFailures);
}
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry.

168
src/CBDD.Core/Context/DocumentDbContext.cs
View File

@@ -1,13 +1,10 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Compression;
using ZB.MOM.WW.CBDD.Core.Metadata;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Metadata;
using ZB.MOM.WW.CBDD.Core.Compression;
using System.Threading;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Core;
@@ -24,26 +21,16 @@ internal interface ICompactionAwareCollection
/// Inherit and add DocumentCollection{T} properties for your entities.
/// Use partial class for Source Generator integration.
/// </summary>
public abstract partial class DocumentDbContext : IDisposable, ITransactionHolder
public abstract class DocumentDbContext : IDisposable, ITransactionHolder
{
private readonly IStorageEngine _storage;
internal readonly CDC.ChangeStreamDispatcher _cdc;
protected bool _disposed;
private readonly SemaphoreSlim _transactionLock = new SemaphoreSlim(1, 1);
internal readonly ChangeStreamDispatcher _cdc;
private readonly List<ICompactionAwareCollection> _compactionAwareCollections = new();
/// <summary>
/// Gets the current active transaction, if any.
/// </summary>
public ITransaction? CurrentTransaction
{
get
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return field != null && (field.State == TransactionState.Active) ? field : null;
}
private set;
}
private readonly IReadOnlyDictionary<Type, object> _model;
private readonly List<IDocumentMapper> _registeredMappers = new();
private readonly IStorageEngine _storage;
private readonly SemaphoreSlim _transactionLock = new(1, 1);
protected bool _disposed;
/// <summary>
/// Creates a new database context with default configuration
@@ -91,7 +78,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
throw new ArgumentNullException(nameof(databasePath));
_storage = new StorageEngine(databasePath, config, compressionOptions, maintenanceOptions);
_cdc = new CDC.ChangeStreamDispatcher();
_cdc = new ChangeStreamDispatcher();
_storage.RegisterCdc(_cdc);
// Initialize model before collections
@@ -102,16 +89,18 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Initializes document collections for the context.
/// Gets the current active transaction, if any.
/// </summary>
protected virtual void InitializeCollections()
public ITransaction? CurrentTransaction
{
// Derived classes can override to initialize collections
get
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return field != null && field.State == TransactionState.Active ? field : null;
}
private set;
}
private readonly IReadOnlyDictionary<Type, object> _model;
private readonly List<IDocumentMapper> _registeredMappers = new();
private readonly List<ICompactionAwareCollection> _compactionAwareCollections = new();
/// <summary>
/// Gets the concrete storage engine for advanced scenarios in derived contexts.
@@ -133,6 +122,49 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
/// </summary>
protected CompressionTelemetry CompressionTelemetry => _storage.CompressionTelemetry;
/// <summary>
/// Releases resources used by the context.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
_disposed = true;
_storage?.Dispose();
_cdc?.Dispose();
_transactionLock?.Dispose();
GC.SuppressFinalize(this);
}
/// <summary>
/// Gets the current active transaction or starts a new one.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction GetCurrentTransactionOrStart()
{
return BeginTransaction();
}
/// <summary>
/// Gets the current active transaction or starts a new one asynchronously.
/// </summary>
/// <returns>The active transaction.</returns>
public async Task<ITransaction> GetCurrentTransactionOrStartAsync()
{
return await BeginTransactionAsync();
}
/// <summary>
/// Initializes document collections for the context.
/// </summary>
protected virtual void InitializeCollections()
{
// Derived classes can override to initialize collections
}
/// <summary>
/// Override to configure the model using Fluent API.
/// </summary>
@@ -158,7 +190,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
string? customName = null;
EntityTypeBuilder<T>? builder = null;
if (_model.TryGetValue(typeof(T), out var builderObj))
if (_model.TryGetValue(typeof(T), out object? builderObj))
{
builder = builderObj as EntityTypeBuilder<T>;
customName = builder?.CollectionName;
@@ -167,18 +199,12 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
_registeredMappers.Add(mapper);
var collection = new DocumentCollection<TId, T>(_storage, this, mapper, customName);
if (collection is ICompactionAwareCollection compactionAwareCollection)
{
_compactionAwareCollections.Add(compactionAwareCollection);
}
// Apply configurations from ModelBuilder
if (builder != null)
{
foreach (var indexBuilder in builder.Indexes)
{
collection.ApplyIndexBuilder(indexBuilder);
}
}
_storage.RegisterMappers(_registeredMappers);
@@ -190,7 +216,10 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
/// </summary>
/// <typeparam name="T">The type of entity to retrieve the document collection for. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;ObjectId, T&gt; instance for the specified entity type.</returns>
public DocumentCollection<ObjectId, T> Set<T>() where T : class => Set<ObjectId, T>();
public DocumentCollection<ObjectId, T> Set<T>() where T : class
{
return Set<ObjectId, T>();
}
/// <summary>
/// Gets a collection for managing documents of type T, identified by keys of type TId.
@@ -200,23 +229,9 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
/// <typeparam name="T">The type of the document to be managed. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;TId, T&gt; instance for performing operations on documents of type T.</returns>
public virtual DocumentCollection<TId, T> Set<TId, T>() where T : class
=> throw new InvalidOperationException($"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.");
/// <summary>
/// Releases resources used by the context.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
_disposed = true;
_storage?.Dispose();
_cdc?.Dispose();
_transactionLock?.Dispose();
GC.SuppressFinalize(this);
throw new InvalidOperationException(
$"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.");
}
/// <summary>
@@ -252,7 +267,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
bool lockAcquired = false;
var lockAcquired = false;
try
{
await _transactionLock.WaitAsync(ct);
@@ -270,24 +285,6 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
}
/// <summary>
/// Gets the current active transaction or starts a new one.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction GetCurrentTransactionOrStart()
{
return BeginTransaction();
}
/// <summary>
/// Gets the current active transaction or starts a new one asynchronously.
/// </summary>
/// <returns>The active transaction.</returns>
public async Task<ITransaction> GetCurrentTransactionOrStartAsync()
{
return await BeginTransactionAsync();
}
/// <summary>
/// Commits the current transaction if one is active.
/// </summary>
@@ -296,7 +293,6 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
if (CurrentTransaction != null)
{
try
{
CurrentTransaction.Commit();
@@ -306,7 +302,6 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
CurrentTransaction = null;
}
}
}
/// <summary>
/// Commits the current transaction asynchronously if one is active.
@@ -317,7 +312,6 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
if (CurrentTransaction != null)
{
try
{
await CurrentTransaction.CommitAsync(ct);
@@ -327,7 +321,6 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
CurrentTransaction = null;
}
}
}
/// <summary>
/// Executes a checkpoint using the requested mode.
@@ -348,7 +341,8 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
/// <param name="mode">Checkpoint mode to execute.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The checkpoint execution result.</returns>
public Task<CheckpointResult> CheckpointAsync(CheckpointMode mode = CheckpointMode.Truncate, CancellationToken ct = default)
public Task<CheckpointResult> CheckpointAsync(CheckpointMode mode = CheckpointMode.Truncate,
CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
@@ -395,7 +389,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Alias for <see cref="Compact(CompactionOptions?)"/>.
/// Alias for <see cref="Compact(CompactionOptions?)" />.
/// </summary>
/// <param name="options">Compaction execution options.</param>
public CompactionStats Vacuum(CompactionOptions? options = null)
@@ -409,7 +403,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Async alias for <see cref="CompactAsync(CompactionOptions?, CancellationToken)"/>.
/// Async alias for <see cref="CompactAsync(CompactionOptions?, CancellationToken)" />.
/// </summary>
/// <param name="options">Compaction execution options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
@@ -437,10 +431,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
private void RefreshCollectionBindingsAfterCompaction()
{
foreach (var collection in _compactionAwareCollections)
{
collection.RefreshIndexBindingsAfterCompaction();
}
foreach (var collection in _compactionAwareCollections) collection.RefreshIndexBindingsAfterCompaction();
}
/// <summary>
@@ -515,7 +506,8 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
/// </summary>
/// <param name="options">Compression migration options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
public Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null, CancellationToken ct = default)
public Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null,
CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));

98
src/CBDD.Core/Indexing/BTreeCursor.cs
View File

@@ -1,27 +1,24 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Bson;
using System.Collections.Generic;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
internal sealed class BTreeCursor : IBTreeCursor
{
private readonly List<IndexEntry> _currentEntries;
private readonly BTreeIndex _index;
private readonly ulong _transactionId;
private readonly IIndexStorage _storage;
private readonly ulong _transactionId;
private int _currentEntryIndex;
private BTreeNodeHeader _currentHeader;
private uint _currentPageId;
private bool _isValid;
// State
private byte[] _pageBuffer;
private uint _currentPageId;
private int _currentEntryIndex;
private BTreeNodeHeader _currentHeader;
private List<IndexEntry> _currentEntries;
private bool _isValid;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeCursor"/> class.
/// Initializes a new instance of the <see cref="BTreeCursor" /> class.
/// </summary>
/// <param name="index">The index to traverse.</param>
/// <param name="storage">The storage engine for page access.</param>
@@ -51,11 +48,11 @@ internal sealed class BTreeCursor : IBTreeCursor
/// <summary>
/// Moves the cursor to the first entry in the index.
/// </summary>
/// <returns><see langword="true"/> if an entry is available; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if an entry is available; otherwise, <see langword="false" />.</returns>
public bool MoveToFirst()
{
// Find left-most leaf
var pageId = _index.RootPageId;
uint pageId = _index.RootPageId;
while (true)
{
LoadPage(pageId);
@@ -63,7 +60,7 @@ internal sealed class BTreeCursor : IBTreeCursor
// Go to first child (P0)
// Internal node format: [Header] [P0] [Entry1] ...
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
pageId = BitConverter.ToUInt32(_pageBuffer.AsSpan(dataOffset, 4));
}
@@ -73,11 +70,11 @@ internal sealed class BTreeCursor : IBTreeCursor
/// <summary>
/// Moves the cursor to the last entry in the index.
/// </summary>
/// <returns><see langword="true"/> if an entry is available; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if an entry is available; otherwise, <see langword="false" />.</returns>
public bool MoveToLast()
{
// Find right-most leaf
var pageId = _index.RootPageId;
uint pageId = _index.RootPageId;
while (true)
{
LoadPage(pageId);
@@ -93,16 +90,17 @@ internal sealed class BTreeCursor : IBTreeCursor
// We want the last pointer.
// Re-read P0 just in case
uint lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(32 + 20, 4));
var lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(32 + 20, 4));
var offset = 32 + 20 + 4;
for (int i = 0; i < _currentHeader.EntryCount; i++)
int offset = 32 + 20 + 4;
for (var i = 0; i < _currentHeader.EntryCount; i++)
{
var keyLen = BitConverter.ToInt32(_pageBuffer.AsSpan(offset, 4));
offset += 4 + keyLen;
lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(offset, 4));
offset += 4;
}
pageId = lastPointer;
}
@@ -114,17 +112,17 @@ internal sealed class BTreeCursor : IBTreeCursor
/// </summary>
/// <param name="key">The key to seek.</param>
/// <returns>
/// <see langword="true"/> if an exact key match is found; otherwise, <see langword="false"/>.
/// <see langword="true" /> if an exact key match is found; otherwise, <see langword="false" />.
/// </returns>
public bool Seek(IndexKey key)
{
// Use Index to find leaf
var leafPageId = _index.FindLeafNode(key, _transactionId);
uint leafPageId = _index.FindLeafNode(key, _transactionId);
LoadPage(leafPageId);
ParseEntries();
// Binary search in entries
var idx = _currentEntries.BinarySearch(new IndexEntry(key, default(DocumentLocation)));
int idx = _currentEntries.BinarySearch(new IndexEntry(key, default(DocumentLocation)));
if (idx >= 0)
{
@@ -133,8 +131,7 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
// Not found, ~idx is the next larger value
_currentEntryIndex = ~idx;
@@ -143,8 +140,7 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return false; // Positioned at next greater
}
else
{
// Key is larger than max in this page, move to next page
if (_currentHeader.NextLeafPageId != 0)
{
@@ -162,22 +158,17 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = false;
return false;
}
}
}
/// <summary>
/// Moves the cursor to the next entry.
/// </summary>
/// <returns><see langword="true"/> if the cursor moved to a valid entry; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the cursor moved to a valid entry; otherwise, <see langword="false" />.</returns>
public bool MoveNext()
{
if (!_isValid) return false;
_currentEntryIndex++;
if (_currentEntryIndex < _currentEntries.Count)
{
return true;
}
if (_currentEntryIndex < _currentEntries.Count) return true;
// Move to next page
if (_currentHeader.NextLeafPageId != 0)
@@ -193,16 +184,13 @@ internal sealed class BTreeCursor : IBTreeCursor
/// <summary>
/// Moves the cursor to the previous entry.
/// </summary>
/// <returns><see langword="true"/> if the cursor moved to a valid entry; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the cursor moved to a valid entry; otherwise, <see langword="false" />.</returns>
public bool MovePrev()
{
if (!_isValid) return false;
_currentEntryIndex--;
if (_currentEntryIndex >= 0)
{
return true;
}
if (_currentEntryIndex >= 0) return true;
// Move to prev page
if (_currentHeader.PrevLeafPageId != 0)
@@ -215,6 +203,18 @@ internal sealed class BTreeCursor : IBTreeCursor
return false;
}
/// <summary>
/// Releases cursor resources.
/// </summary>
public void Dispose()
{
if (_pageBuffer != null)
{
ArrayPool<byte>.Shared.Return(_pageBuffer);
_pageBuffer = null!;
}
}
private void LoadPage(uint pageId)
{
if (_currentPageId == pageId && _pageBuffer != null) return;
@@ -229,9 +229,9 @@ internal sealed class BTreeCursor : IBTreeCursor
// Helper to parse entries from current page buffer
// (Similar to BTreeIndex.ReadLeafEntries)
_currentEntries.Clear();
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
for (int i = 0; i < _currentHeader.EntryCount; i++)
for (var i = 0; i < _currentHeader.EntryCount; i++)
{
// Read Key
var keyLen = BitConverter.ToInt32(_pageBuffer.AsSpan(dataOffset, 4));
@@ -257,13 +257,11 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
// Empty page? Should not happen in helper logic unless root leaf is empty
_isValid = false;
return false;
}
}
private bool PositionAtEnd()
{
@@ -274,22 +272,8 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
_isValid = false;
return false;
}
}
/// <summary>
/// Releases cursor resources.
/// </summary>
public void Dispose()
{
if (_pageBuffer != null)
{
ArrayPool<byte>.Shared.Return(_pageBuffer);
_pageBuffer = null!;
}
}
}

369
src/CBDD.Core/Indexing/BTreeIndex.cs
View File

@@ -1,8 +1,6 @@
using ZB.MOM.WW.CBDD.Bson;
using System.Buffers;
using System.Text.RegularExpressions;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using System;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -11,13 +9,12 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// </summary>
public sealed class BTreeIndex
{
private readonly IIndexStorage _storage;
private readonly IndexOptions _options;
private uint _rootPageId;
internal const int MaxEntriesPerNode = 100; // Low value to test splitting
private readonly IndexOptions _options;
private readonly IIndexStorage _storage;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeIndex"/> class.
/// Initializes a new instance of the <see cref="BTreeIndex" /> class.
/// </summary>
/// <param name="storage">The storage engine used to read and write index pages.</param>
/// <param name="options">The index options.</param>
@@ -30,7 +27,7 @@ public sealed class BTreeIndex
}
/// <summary>
/// Initializes a new instance of the <see cref="BTreeIndex"/> class.
/// Initializes a new instance of the <see cref="BTreeIndex" /> class.
/// </summary>
/// <param name="storage">The index storage used to read and write index pages.</param>
/// <param name="options">The index options.</param>
@@ -41,15 +38,15 @@ public sealed class BTreeIndex
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
_rootPageId = rootPageId;
RootPageId = rootPageId;
if (_rootPageId == 0)
if (RootPageId == 0)
{
// Allocate new root page (cannot use page 0 which is file header)
_rootPageId = _storage.AllocatePage();
RootPageId = _storage.AllocatePage();
// Initialize as empty leaf
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
// Clear buffer
@@ -58,7 +55,7 @@ public sealed class BTreeIndex
// Write headers
var pageHeader = new PageHeader
{
PageId = _rootPageId,
PageId = RootPageId,
PageType = PageType.Index,
FreeBytes = (ushort)(_storage.PageSize - 32),
NextPageId = 0,
@@ -75,11 +72,11 @@ public sealed class BTreeIndex
};
nodeHeader.WriteTo(pageBuffer.AsSpan(32));
_storage.WritePageImmediate(_rootPageId, pageBuffer);
_storage.WritePageImmediate(RootPageId, pageBuffer);
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
}
@@ -87,7 +84,7 @@ public sealed class BTreeIndex
/// <summary>
/// Gets the current root page identifier for the B+Tree.
/// </summary>
public uint RootPageId => _rootPageId;
public uint RootPageId { get; private set; }
/// <summary>
/// Updates the in-memory root page identifier.
@@ -98,7 +95,7 @@ public sealed class BTreeIndex
if (rootPageId == 0)
throw new ArgumentOutOfRangeException(nameof(rootPageId));
_rootPageId = rootPageId;
RootPageId = rootPageId;
}
/// <summary>
@@ -129,14 +126,14 @@ public sealed class BTreeIndex
/// <param name="transactionId">The optional transaction identifier.</param>
public void Insert(IndexKey key, DocumentLocation location, ulong? transactionId = null)
{
var txnId = transactionId ?? 0;
ulong txnId = transactionId ?? 0;
var entry = new IndexEntry(key, location);
var path = new List<uint>();
// Find the leaf node for insertion
var leafPageId = FindLeafNodeWithPath(key, path, txnId);
uint leafPageId = FindLeafNodeWithPath(key, path, txnId);
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
ReadPage(leafPageId, txnId, pageBuffer);
@@ -158,7 +155,7 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
@@ -171,25 +168,25 @@ public sealed class BTreeIndex
public bool TryFind(IndexKey key, out DocumentLocation location, ulong? transactionId = null)
{
location = default;
var txnId = transactionId ?? 0;
ulong txnId = transactionId ?? 0;
var leafPageId = FindLeafNode(key, txnId);
uint leafPageId = FindLeafNode(key, txnId);
Span<byte> pageBuffer = stackalloc byte[_storage.PageSize];
ReadPage(leafPageId, txnId, pageBuffer);
var header = BTreeNodeHeader.ReadFrom(pageBuffer[32..]);
var dataOffset = 32 + 20; // Page header + BTree node header
int dataOffset = 32 + 20; // Page header + BTree node header
// Linear search in leaf (could be optimized with binary search)
for (int i = 0; i < header.EntryCount; i++)
for (var i = 0; i < header.EntryCount; i++)
{
var entryKey = ReadIndexKey(pageBuffer, dataOffset);
if (entryKey.Equals(key))
{
// Found - read DocumentLocation (6 bytes: 4 for PageId + 2 for SlotIndex)
var locationOffset = dataOffset + entryKey.Data.Length + 4; // +4 for key length prefix
int locationOffset = dataOffset + entryKey.Data.Length + 4; // +4 for key length prefix
location = DocumentLocation.ReadFrom(pageBuffer.Slice(locationOffset, DocumentLocation.SerializedSize));
return true;
}
@@ -208,32 +205,35 @@ public sealed class BTreeIndex
/// <param name="maxKey">The upper bound key.</param>
/// <param name="direction">The scan direction.</param>
/// <param name="transactionId">The optional transaction identifier.</param>
public IEnumerable<IndexEntry> Range(IndexKey minKey, IndexKey maxKey, IndexDirection direction = IndexDirection.Forward, ulong? transactionId = null)
public IEnumerable<IndexEntry> Range(IndexKey minKey, IndexKey maxKey,
IndexDirection direction = IndexDirection.Forward, ulong? transactionId = null)
{
var txnId = transactionId ?? 0;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
ulong txnId = transactionId ?? 0;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
if (direction == IndexDirection.Forward)
{
var leafPageId = FindLeafNode(minKey, txnId);
uint leafPageId = FindLeafNode(minKey, txnId);
while (leafPageId != 0)
{
ReadPage(leafPageId, txnId, pageBuffer);
var header = BTreeNodeHeader.ReadFrom(pageBuffer.AsSpan(32));
var dataOffset = 32 + 20; // Adjusted for 20-byte header
int dataOffset = 32 + 20; // Adjusted for 20-byte header
for (int i = 0; i < header.EntryCount; i++)
for (var i = 0; i < header.EntryCount; i++)
{
var entryKey = ReadIndexKey(pageBuffer, dataOffset);
if (entryKey >= minKey && entryKey <= maxKey)
{
var locationOffset = dataOffset + 4 + entryKey.Data.Length;
var location = DocumentLocation.ReadFrom(pageBuffer.AsSpan(locationOffset, DocumentLocation.SerializedSize));
int locationOffset = dataOffset + 4 + entryKey.Data.Length;
var location =
DocumentLocation.ReadFrom(pageBuffer.AsSpan(locationOffset,
DocumentLocation.SerializedSize));
yield return new IndexEntry(entryKey, location);
}
else if (entryKey > maxKey)
@@ -250,7 +250,7 @@ public sealed class BTreeIndex
else // Backward
{
// Start from the end of the range (maxKey)
var leafPageId = FindLeafNode(maxKey, txnId);
uint leafPageId = FindLeafNode(maxKey, txnId);
while (leafPageId != 0)
{
@@ -267,13 +267,8 @@ public sealed class BTreeIndex
{
var entry = entries[i];
if (entry.Key <= maxKey && entry.Key >= minKey)
{
yield return entry;
}
else if (entry.Key < minKey)
{
yield break; // Exceeded range (below min)
}
else if (entry.Key < minKey) yield break; // Exceeded range (below min)
}
// Check if we need to continue to previous leaf
@@ -296,7 +291,7 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
@@ -314,8 +309,8 @@ public sealed class BTreeIndex
private uint FindLeafNodeWithPath(IndexKey key, List<uint> path, ulong transactionId)
{
var currentPageId = _rootPageId;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
uint currentPageId = RootPageId;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
@@ -324,10 +319,7 @@ public sealed class BTreeIndex
ReadPage(currentPageId, transactionId, pageBuffer);
var header = BTreeNodeHeader.ReadFrom(pageBuffer.AsSpan(32));
if (header.IsLeaf)
{
return currentPageId;
}
if (header.IsLeaf) return currentPageId;
path.Add(currentPageId);
currentPageId = FindChildNode(pageBuffer, header, key);
@@ -335,7 +327,7 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
@@ -348,24 +340,21 @@ public sealed class BTreeIndex
// [Entry 2: Key2, P2]
// ...
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
var p0 = BitConverter.ToUInt32(nodeBuffer.Slice(dataOffset, 4));
dataOffset += 4;
uint childPageId = p0;
// Linear search for now (optimize to binary search later)
for (int i = 0; i < header.EntryCount; i++)
for (var i = 0; i < header.EntryCount; i++)
{
var entryKey = ReadIndexKey(nodeBuffer, dataOffset);
var keyLen = 4 + entryKey.Data.Length;
var pointerOffset = dataOffset + keyLen;
int keyLen = 4 + entryKey.Data.Length;
int pointerOffset = dataOffset + keyLen;
var nextPointer = BitConverter.ToUInt32(nodeBuffer.Slice(pointerOffset, 4));
if (key < entryKey)
{
return childPageId;
}
if (key < entryKey) return childPageId;
childPageId = nextPointer;
dataOffset += keyLen + 4; // Key + Pointer
@@ -395,21 +384,18 @@ public sealed class BTreeIndex
public IEnumerable<IndexEntry> Equal(IndexKey key, ulong transactionId)
{
using var cursor = CreateCursor(transactionId);
if (cursor.Seek(key))
{
yield return cursor.Current;
if (cursor.Seek(key)) yield return cursor.Current;
// Handle duplicates if we support them? Current impl looks unique-ish per key unless multi-value index.
// BTreeIndex doesn't strictly prevent duplicates in structure, but usually unique keys.
// If unique, yield one. If not, loop.
// Assuming unique for now based on TryFind.
}
}
/// <summary>
/// Returns entries greater than the specified key.
/// </summary>
/// <param name="key">The comparison key.</param>
/// <param name="orEqual">If true, includes entries equal to <paramref name="key"/>.</param>
/// <param name="orEqual">If true, includes entries equal to <paramref name="key" />.</param>
/// <param name="transactionId">The transaction identifier used for isolation.</param>
/// <returns>An enumerable sequence of matching entries.</returns>
public IEnumerable<IndexEntry> GreaterThan(IndexKey key, bool orEqual, ulong transactionId)
@@ -418,9 +404,8 @@ public sealed class BTreeIndex
bool found = cursor.Seek(key);
if (found && !orEqual)
{
if (!cursor.MoveNext()) yield break;
}
if (!cursor.MoveNext())
yield break;
// Loop forward
do
@@ -433,7 +418,7 @@ public sealed class BTreeIndex
/// Returns entries less than the specified key.
/// </summary>
/// <param name="key">The comparison key.</param>
/// <param name="orEqual">If true, includes entries equal to <paramref name="key"/>.</param>
/// <param name="orEqual">If true, includes entries equal to <paramref name="key" />.</param>
/// <param name="transactionId">The transaction identifier used for isolation.</param>
/// <returns>An enumerable sequence of matching entries.</returns>
public IEnumerable<IndexEntry> LessThan(IndexKey key, bool orEqual, ulong transactionId)
@@ -467,19 +452,19 @@ public sealed class BTreeIndex
/// </summary>
/// <param name="start">The start key.</param>
/// <param name="end">The end key.</param>
/// <param name="startInclusive">If true, includes entries equal to <paramref name="start"/>.</param>
/// <param name="endInclusive">If true, includes entries equal to <paramref name="end"/>.</param>
/// <param name="startInclusive">If true, includes entries equal to <paramref name="start" />.</param>
/// <param name="endInclusive">If true, includes entries equal to <paramref name="end" />.</param>
/// <param name="transactionId">The transaction identifier used for isolation.</param>
/// <returns>An enumerable sequence of matching entries.</returns>
public IEnumerable<IndexEntry> Between(IndexKey start, IndexKey end, bool startInclusive, bool endInclusive, ulong transactionId)
public IEnumerable<IndexEntry> Between(IndexKey start, IndexKey end, bool startInclusive, bool endInclusive,
ulong transactionId)
{
using var cursor = CreateCursor(transactionId);
bool found = cursor.Seek(start);
if (found && !startInclusive)
{
if (!cursor.MoveNext()) yield break;
}
if (!cursor.MoveNext())
yield break;
// Iterate while <= end
do
@@ -489,7 +474,6 @@ public sealed class BTreeIndex
if (current.Key == end && !endInclusive) yield break;
yield return current;
} while (cursor.MoveNext());
}
@@ -509,13 +493,18 @@ public sealed class BTreeIndex
{
var current = cursor.Current;
string val;
try { val = current.Key.As<string>(); }
catch { break; }
try
{
val = current.Key.As<string>();
}
catch
{
break;
}
if (!val.StartsWith(prefix)) break;
yield return current;
} while (cursor.MoveNext());
}
@@ -531,13 +520,9 @@ public sealed class BTreeIndex
using var cursor = CreateCursor(transactionId);
foreach (var key in sortedKeys)
{
if (cursor.Seek(key))
{
yield return cursor.Current;
}
}
}
/// <summary>
/// Returns string-key entries that match a SQL-like pattern.
@@ -547,14 +532,14 @@ public sealed class BTreeIndex
/// <returns>An enumerable sequence of matching entries.</returns>
public IEnumerable<IndexEntry> Like(string pattern, ulong transactionId)
{
string regexPattern = "^" + System.Text.RegularExpressions.Regex.Escape(pattern)
string regexPattern = "^" + Regex.Escape(pattern)
.Replace("%", ".*")
.Replace("_", ".") + "$";
var regex = new System.Text.RegularExpressions.Regex(regexPattern, System.Text.RegularExpressions.RegexOptions.Compiled);
var regex = new Regex(regexPattern, RegexOptions.Compiled);
string prefix = "";
for (int i = 0; i < pattern.Length; i++)
var prefix = "";
for (var i = 0; i < pattern.Length; i++)
{
if (pattern[i] == '%' || pattern[i] == '_') break;
prefix += pattern[i];
@@ -563,30 +548,34 @@ public sealed class BTreeIndex
using var cursor = CreateCursor(transactionId);
if (!string.IsNullOrEmpty(prefix))
{
cursor.Seek(IndexKey.Create(prefix));
}
else
{
cursor.MoveToFirst();
}
do
{
IndexEntry current;
try { current = cursor.Current; } catch { break; } // Safe break if cursor invalid
if (!string.IsNullOrEmpty(prefix))
{
try
{
string val = current.Key.As<string>();
current = cursor.Current;
}
catch
{
break;
} // Safe break if cursor invalid
if (!string.IsNullOrEmpty(prefix))
try
{
var val = current.Key.As<string>();
if (!val.StartsWith(prefix)) break;
}
catch { break; }
catch
{
break;
}
bool match = false;
var match = false;
try
{
match = regex.IsMatch(current.Key.As<string>());
@@ -597,7 +586,6 @@ public sealed class BTreeIndex
}
if (match) yield return current;
} while (cursor.MoveNext());
}
@@ -605,10 +593,10 @@ public sealed class BTreeIndex
{
// Read current entries to determine offset
var header = BTreeNodeHeader.ReadFrom(pageBuffer[32..]);
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
// Skip existing entries to find free space
for (int i = 0; i < header.EntryCount; i++)
for (var i = 0; i < header.EntryCount; i++)
{
var keyLen = BitConverter.ToInt32(pageBuffer.Slice(dataOffset, 4));
dataOffset += 4 + keyLen + DocumentLocation.SerializedSize; // Length + Key + DocumentLocation
@@ -635,37 +623,34 @@ public sealed class BTreeIndex
private void SplitNode(uint nodePageId, List<uint> path, ulong transactionId)
{
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
ReadPage(nodePageId, transactionId, pageBuffer);
var header = BTreeNodeHeader.ReadFrom(pageBuffer.AsSpan(32));
if (header.IsLeaf)
{
SplitLeafNode(nodePageId, header, pageBuffer, path, transactionId);
}
else
{
SplitInternalNode(nodePageId, header, pageBuffer, path, transactionId);
}
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
private void SplitLeafNode(uint nodePageId, BTreeNodeHeader header, Span<byte> pageBuffer, List<uint> path, ulong transactionId)
private void SplitLeafNode(uint nodePageId, BTreeNodeHeader header, Span<byte> pageBuffer, List<uint> path,
ulong transactionId)
{
var entries = ReadLeafEntries(pageBuffer, header.EntryCount);
var splitPoint = entries.Count / 2;
int splitPoint = entries.Count / 2;
var leftEntries = entries.Take(splitPoint).ToList();
var rightEntries = entries.Skip(splitPoint).ToList();
// Create new node for right half
var newNodeId = CreateNode(isLeaf: true, transactionId);
uint newNodeId = CreateNode(true, transactionId);
// Update original node (left)
// Next -> RightNode
@@ -678,10 +663,7 @@ public sealed class BTreeIndex
WriteLeafNode(newNodeId, rightEntries, header.NextLeafPageId, nodePageId, transactionId);
// Update Original Next Node's Prev pointer to point to New Node
if (header.NextLeafPageId != 0)
{
UpdatePrevPointer(header.NextLeafPageId, newNodeId, transactionId);
}
if (header.NextLeafPageId != 0) UpdatePrevPointer(header.NextLeafPageId, newNodeId, transactionId);
// Promote key to parent (first key of right node)
var promoteKey = rightEntries[0].Key;
@@ -690,7 +672,7 @@ public sealed class BTreeIndex
private void UpdatePrevPointer(uint pageId, uint newPrevId, ulong transactionId)
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] buffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
ReadPage(pageId, transactionId, buffer);
@@ -701,24 +683,27 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
private void SplitInternalNode(uint nodePageId, BTreeNodeHeader header, Span<byte> pageBuffer, List<uint> path, ulong transactionId)
private void SplitInternalNode(uint nodePageId, BTreeNodeHeader header, Span<byte> pageBuffer, List<uint> path,
ulong transactionId)
{
var (p0, entries) = ReadInternalEntries(pageBuffer, header.EntryCount);
var splitPoint = entries.Count / 2;
(uint p0, var entries) = ReadInternalEntries(pageBuffer, header.EntryCount);
int splitPoint = entries.Count / 2;
// For internal nodes, the median key moves UP to parent and is excluded from children
var promoteKey = entries[splitPoint].Key;
var leftEntries = entries.Take(splitPoint).ToList();
var rightEntries = entries.Skip(splitPoint + 1).ToList();
var rightP0 = entries[splitPoint].PageId; // Attempting to use the pointer associated with promoted key as P0 for right node
uint rightP0 =
entries[splitPoint]
.PageId; // Attempting to use the pointer associated with promoted key as P0 for right node
// Create new internal node
var newNodeId = CreateNode(isLeaf: false, transactionId);
uint newNodeId = CreateNode(false, transactionId);
// Update left node
WriteInternalNode(nodePageId, p0, leftEntries, transactionId);
@@ -730,7 +715,8 @@ public sealed class BTreeIndex
InsertIntoParent(nodePageId, promoteKey, newNodeId, path, transactionId);
}
private void InsertIntoParent(uint leftChildPageId, IndexKey key, uint rightChildPageId, List<uint> path, ulong transactionId)
private void InsertIntoParent(uint leftChildPageId, IndexKey key, uint rightChildPageId, List<uint> path,
ulong transactionId)
{
if (path.Count == 0 || path.Last() == leftChildPageId)
{
@@ -747,10 +733,10 @@ public sealed class BTreeIndex
}
}
var parentPageId = path.Last();
uint parentPageId = path.Last();
path.RemoveAt(path.Count - 1); // Pop parent for recursive calls
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
ReadPage(parentPageId, transactionId, pageBuffer);
@@ -764,7 +750,8 @@ public sealed class BTreeIndex
// But wait, to Split we need the median.
// Better approach: Read all, add new entry, then split the collection and write back.
var (p0, entries) = ReadInternalEntries(pageBuffer.AsSpan(0, _storage.PageSize), header.EntryCount);
(uint p0, var entries) =
ReadInternalEntries(pageBuffer.AsSpan(0, _storage.PageSize), header.EntryCount);
// Insert new key/pointer in sorted order
var newEntry = new InternalEntry(key, rightChildPageId);
@@ -773,14 +760,14 @@ public sealed class BTreeIndex
else entries.Insert(insertIndex, newEntry);
// Now split these extended entries
var splitPoint = entries.Count / 2;
int splitPoint = entries.Count / 2;
var promoteKey = entries[splitPoint].Key;
var rightP0 = entries[splitPoint].PageId;
uint rightP0 = entries[splitPoint].PageId;
var leftEntries = entries.Take(splitPoint).ToList();
var rightEntries = entries.Skip(splitPoint + 1).ToList();
var newParentId = CreateNode(isLeaf: false, transactionId);
uint newParentId = CreateNode(false, transactionId);
WriteInternalNode(parentPageId, p0, leftEntries, transactionId);
WriteInternalNode(newParentId, rightP0, rightEntries, transactionId);
@@ -790,21 +777,22 @@ public sealed class BTreeIndex
else
{
// Insert directly
InsertIntoInternal(parentPageId, header, pageBuffer.AsSpan(0, _storage.PageSize), key, rightChildPageId, transactionId);
InsertIntoInternal(parentPageId, header, pageBuffer.AsSpan(0, _storage.PageSize), key, rightChildPageId,
transactionId);
}
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
private void CreateNewRoot(uint leftChildId, IndexKey key, uint rightChildId, ulong transactionId)
{
var newRootId = CreateNode(isLeaf: false, transactionId);
var entries = new List<InternalEntry> { new InternalEntry(key, rightChildId) };
uint newRootId = CreateNode(false, transactionId);
var entries = new List<InternalEntry> { new(key, rightChildId) };
WriteInternalNode(newRootId, leftChildId, entries, transactionId);
_rootPageId = newRootId; // Update in-memory root
RootPageId = newRootId; // Update in-memory root
// TODO: Update root in file header/metadata block so it persists?
// For now user passes rootPageId to ctor. BTreeIndex doesn't manage master root pointer persistence yet.
@@ -812,8 +800,8 @@ public sealed class BTreeIndex
private uint CreateNode(bool isLeaf, ulong transactionId)
{
var pageId = _storage.AllocatePage();
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
uint pageId = _storage.AllocatePage();
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
Array.Clear(pageBuffer, 0, _storage.PageSize);
@@ -846,7 +834,7 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
return pageId;
@@ -855,42 +843,45 @@ public sealed class BTreeIndex
private List<IndexEntry> ReadLeafEntries(Span<byte> pageBuffer, int count)
{
var entries = new List<IndexEntry>(count);
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
var key = ReadIndexKey(pageBuffer, dataOffset);
var locationOffset = dataOffset + 4 + key.Data.Length;
int locationOffset = dataOffset + 4 + key.Data.Length;
var location = DocumentLocation.ReadFrom(pageBuffer.Slice(locationOffset, DocumentLocation.SerializedSize));
entries.Add(new IndexEntry(key, location));
dataOffset = locationOffset + DocumentLocation.SerializedSize;
}
return entries;
}
private (uint P0, List<InternalEntry> Entries) ReadInternalEntries(Span<byte> pageBuffer, int count)
{
var entries = new List<InternalEntry>(count);
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
var p0 = BitConverter.ToUInt32(pageBuffer.Slice(dataOffset, 4));
dataOffset += 4;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
var key = ReadIndexKey(pageBuffer, dataOffset);
var ptrOffset = dataOffset + 4 + key.Data.Length;
int ptrOffset = dataOffset + 4 + key.Data.Length;
var pageId = BitConverter.ToUInt32(pageBuffer.Slice(ptrOffset, 4));
entries.Add(new InternalEntry(key, pageId));
dataOffset = ptrOffset + 4;
}
return (p0, entries);
}
private void WriteLeafNode(uint pageId, List<IndexEntry> entries, uint nextLeafId, uint prevLeafId, ulong? transactionId = null)
private void WriteLeafNode(uint pageId, List<IndexEntry> entries, uint nextLeafId, uint prevLeafId,
ulong? transactionId = null)
{
var txnId = transactionId ?? 0;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
ulong txnId = transactionId ?? 0;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
Array.Clear(pageBuffer, 0, _storage.PageSize);
@@ -919,7 +910,7 @@ public sealed class BTreeIndex
nodeHeader.WriteTo(pageBuffer.AsSpan(32, 20));
// Write entries with DocumentLocation (6 bytes instead of ObjectId 12 bytes)
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
foreach (var entry in entries)
{
BitConverter.TryWriteBytes(pageBuffer.AsSpan(dataOffset, 4), entry.Key.Data.Length);
@@ -933,13 +924,13 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
private void WriteInternalNode(uint pageId, uint p0, List<InternalEntry> entries, ulong transactionId)
{
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
Array.Clear(pageBuffer, 0, _storage.PageSize);
@@ -967,7 +958,7 @@ public sealed class BTreeIndex
nodeHeader.WriteTo(pageBuffer.AsSpan(32, 20));
// Write P0
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
BitConverter.TryWriteBytes(pageBuffer.AsSpan(dataOffset, 4), p0);
dataOffset += 4;
@@ -985,14 +976,15 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
private void InsertIntoInternal(uint pageId, BTreeNodeHeader header, Span<byte> pageBuffer, IndexKey key, uint rightChildId, ulong transactionId)
private void InsertIntoInternal(uint pageId, BTreeNodeHeader header, Span<byte> pageBuffer, IndexKey key,
uint rightChildId, ulong transactionId)
{
// Read, insert, write back. In production do in-place shift.
var (p0, entries) = ReadInternalEntries(pageBuffer, header.EntryCount);
(uint p0, var entries) = ReadInternalEntries(pageBuffer, header.EntryCount);
var newEntry = new InternalEntry(key, rightChildId);
int insertIndex = entries.FindIndex(e => e.Key > key);
@@ -1025,11 +1017,11 @@ public sealed class BTreeIndex
/// <param name="transactionId">The optional transaction identifier.</param>
public bool Delete(IndexKey key, DocumentLocation location, ulong? transactionId = null)
{
var txnId = transactionId ?? 0;
ulong txnId = transactionId ?? 0;
var path = new List<uint>();
var leafPageId = FindLeafNodeWithPath(key, path, txnId);
uint leafPageId = FindLeafNodeWithPath(key, path, txnId);
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
ReadPage(leafPageId, txnId, pageBuffer);
@@ -1037,14 +1029,11 @@ public sealed class BTreeIndex
// Check if key exists in leaf
var entries = ReadLeafEntries(pageBuffer, header.EntryCount);
var entryIndex = entries.FindIndex(e => e.Key.Equals(key) &&
int entryIndex = entries.FindIndex(e => e.Key.Equals(key) &&
e.Location.PageId == location.PageId &&
e.Location.SlotIndex == location.SlotIndex);
if (entryIndex == -1)
{
return false; // Not found
}
if (entryIndex == -1) return false; // Not found
// Remove entry
entries.RemoveAt(entryIndex);
@@ -1055,34 +1044,29 @@ public sealed class BTreeIndex
// Check for underflow (min 50% fill)
// Simplified: min 1 entry for now, or MaxEntries/2
int minEntries = MaxEntriesPerNode / 2;
if (entries.Count < minEntries && _rootPageId != leafPageId)
{
HandleUnderflow(leafPageId, path, txnId);
}
if (entries.Count < minEntries && RootPageId != leafPageId) HandleUnderflow(leafPageId, path, txnId);
return true;
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
private void HandleUnderflow(uint nodeId, List<uint> path, ulong transactionId)
{
if (path.Count == 0)
{
// Node is root
if (nodeId == _rootPageId)
{
if (nodeId == RootPageId)
// Special case: Collapse root if it has only 1 child (and is not a leaf)
// For now, simpliest implementation: do nothing for root underflow unless it's empty
// If it's a leaf root, it can be empty.
return;
}
}
var parentPageId = path[^1]; // Parent is last in path (before current node removed? No, path contains ancestors)
uint
parentPageId =
path[^1]; // Parent is last in path (before current node removed? No, path contains ancestors)
// Wait, FindLeafNodeWithPath adds ancestors. So path.Last() is not current node, it's parent.
// Let's verify FindLeafNodeWithPath:
// path.Add(currentPageId); currentPageId = FindChildNode(...);
@@ -1091,37 +1075,34 @@ public sealed class BTreeIndex
// Correct.
// So path.Last() is the parent.
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
ReadPage(parentPageId, transactionId, pageBuffer);
var parentHeader = BTreeNodeHeader.ReadFrom(pageBuffer.AsSpan(32));
var (p0, parentEntries) = ReadInternalEntries(pageBuffer, parentHeader.EntryCount);
(uint p0, var parentEntries) = ReadInternalEntries(pageBuffer, parentHeader.EntryCount);
// Find index of current node in parent
int childIndex = -1;
if (p0 == nodeId) childIndex = -1; // -1 indicates P0
else
{
childIndex = parentEntries.FindIndex(e => e.PageId == nodeId);
}
// Try to borrow from siblings
if (BorrowFromSibling(nodeId, parentPageId, childIndex, parentEntries, p0, transactionId))
{
return; // Rebalanced
}
if (BorrowFromSibling(nodeId, parentPageId, childIndex, parentEntries, p0,
transactionId)) return; // Rebalanced
// Borrow failed, valid siblings are too small -> MERGE
MergeWithSibling(nodeId, parentPageId, childIndex, parentEntries, p0, path, transactionId);
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
private bool BorrowFromSibling(uint nodeId, uint parentId, int childIndex, List<InternalEntry> parentEntries, uint p0, ulong transactionId)
private bool BorrowFromSibling(uint nodeId, uint parentId, int childIndex, List<InternalEntry> parentEntries,
uint p0, ulong transactionId)
{
// TODO: Implement rotation (borrow from left or right sibling)
// Complexity: High. Need to update Parent, Sibling, and Node.
@@ -1130,7 +1111,8 @@ public sealed class BTreeIndex
return false;
}
private void MergeWithSibling(uint nodeId, uint parentId, int childIndex, List<InternalEntry> parentEntries, uint p0, List<uint> path, ulong transactionId)
private void MergeWithSibling(uint nodeId, uint parentId, int childIndex, List<InternalEntry> parentEntries,
uint p0, List<uint> path, ulong transactionId)
{
// Identify sibling to merge with.
// If P0 (childIndex -1), merge with right sibling (Entry 0).
@@ -1167,14 +1149,10 @@ public sealed class BTreeIndex
// Remove separator key and right pointer from Parent
if (childIndex == -1)
{
parentEntries.RemoveAt(0); // Removing Entry 0 (Key 0, P1) - P1 was Right Node
// P0 remains P0 (which was Left Node)
}
else
{
parentEntries.RemoveAt(childIndex); // Remove entry pointing to Right Node
}
// Write updated Parent
WriteInternalNode(parentId, p0, parentEntries, transactionId);
@@ -1186,23 +1164,23 @@ public sealed class BTreeIndex
// Recursive Underflow Check on Parent
int minInternal = MaxEntriesPerNode / 2;
if (parentEntries.Count < minInternal && parentId != _rootPageId)
if (parentEntries.Count < minInternal && parentId != RootPageId)
{
var parentPath = new List<uint>(path.Take(path.Count - 1)); // Path to grandparent
HandleUnderflow(parentId, parentPath, transactionId);
}
else if (parentId == _rootPageId && parentEntries.Count == 0)
else if (parentId == RootPageId && parentEntries.Count == 0)
{
// Root collapse: Root has 0 entries (only P0).
// P0 becomes new root.
_rootPageId = p0; // P0 is the merged node (LeftNode)
RootPageId = p0; // P0 is the merged node (LeftNode)
// TODO: Update persistent root pointer if stored
}
}
private void MergeNodes(uint leftNodeId, uint rightNodeId, IndexKey separatorKey, ulong transactionId)
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
byte[] buffer = ArrayPool<byte>.Shared.Rent(_storage.PageSize);
try
{
// Read both nodes
@@ -1218,7 +1196,9 @@ public sealed class BTreeIndex
var leftEntries = ReadLeafEntries(buffer, leftHeader.EntryCount);
ReadPage(rightNodeId, transactionId, buffer);
var rightEntries = ReadLeafEntries(buffer.AsSpan(0, _storage.PageSize), ((BTreeNodeHeader.ReadFrom(buffer.AsSpan(32))).EntryCount)); // Dirty read reuse buffer? No, bad hygiene.
var rightEntries = ReadLeafEntries(buffer.AsSpan(0, _storage.PageSize),
BTreeNodeHeader.ReadFrom(buffer.AsSpan(32))
.EntryCount); // Dirty read reuse buffer? No, bad hygiene.
// Re-read right clean
var rightHeader = BTreeNodeHeader.ReadFrom(buffer.AsSpan(32));
rightEntries = ReadLeafEntries(buffer, rightHeader.EntryCount);
@@ -1229,24 +1209,23 @@ public sealed class BTreeIndex
// Update Left
// Next -> Right.Next
// Prev -> Left.Prev (unchanged)
WriteLeafNode(leftNodeId, leftEntries, rightHeader.NextLeafPageId, leftHeader.PrevLeafPageId, transactionId);
WriteLeafNode(leftNodeId, leftEntries, rightHeader.NextLeafPageId, leftHeader.PrevLeafPageId,
transactionId);
// Update Right.Next's Prev pointer to point to Left (since Right is gone)
if (rightHeader.NextLeafPageId != 0)
{
UpdatePrevPointer(rightHeader.NextLeafPageId, leftNodeId, transactionId);
}
}
else
{
// Internal Node Merge
ReadPage(leftNodeId, transactionId, buffer);
// leftHeader is already read and valid
var (leftP0, leftEntries) = ReadInternalEntries(buffer, leftHeader.EntryCount);
(uint leftP0, var leftEntries) = ReadInternalEntries(buffer, leftHeader.EntryCount);
ReadPage(rightNodeId, transactionId, buffer);
var rightHeader = BTreeNodeHeader.ReadFrom(buffer.AsSpan(32));
var (rightP0, rightEntries) = ReadInternalEntries(buffer, rightHeader.EntryCount);
(uint rightP0, var rightEntries) = ReadInternalEntries(buffer, rightHeader.EntryCount);
// Add Separator Key (from parent) pointing to Right's P0
leftEntries.Add(new InternalEntry(separatorKey, rightP0));
@@ -1260,7 +1239,7 @@ public sealed class BTreeIndex
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
}

20
src/CBDD.Core/Indexing/BTreeStructures.cs
View File

@@ -1,6 +1,5 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -21,7 +20,7 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
public DocumentLocation Location { get; set; }
/// <summary>
/// Initializes a new instance of the <see cref="IndexEntry"/> struct.
/// Initializes a new instance of the <see cref="IndexEntry" /> struct.
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
@@ -34,7 +33,7 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
// Backward compatibility: constructor that takes ObjectId (for migration)
// Will be removed once all code is migrated
/// <summary>
/// Initializes a legacy instance of the <see cref="IndexEntry"/> struct for migration scenarios.
/// Initializes a legacy instance of the <see cref="IndexEntry" /> struct for migration scenarios.
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="documentId">The legacy document identifier.</param>
@@ -51,7 +50,7 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
/// </summary>
/// <param name="other">The other index entry to compare.</param>
/// <returns>
/// A value less than zero if this instance is less than <paramref name="other"/>,
/// A value less than zero if this instance is less than <paramref name="other" />,
/// zero if they are equal, or greater than zero if this instance is greater.
/// </returns>
public int CompareTo(IndexEntry other)
@@ -64,10 +63,10 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
/// </summary>
/// <param name="obj">The object to compare.</param>
/// <returns>
/// A value less than zero if this instance is less than <paramref name="obj"/>,
/// A value less than zero if this instance is less than <paramref name="obj" />,
/// zero if they are equal, or greater than zero if this instance is greater.
/// </returns>
/// <exception cref="ArgumentException">Thrown when <paramref name="obj"/> is not an <see cref="IndexEntry"/>.</exception>
/// <exception cref="ArgumentException">Thrown when <paramref name="obj" /> is not an <see cref="IndexEntry" />.</exception>
public int CompareTo(object? obj)
{
if (obj is IndexEntry other) return CompareTo(other);
@@ -120,7 +119,7 @@ public struct BTreeNodeHeader
if (destination.Length < 20)
throw new ArgumentException("Destination must be at least 20 bytes");
BitConverter.TryWriteBytes(destination[0..4], PageId);
BitConverter.TryWriteBytes(destination[..4], PageId);
destination[4] = (byte)(IsLeaf ? 1 : 0);
BitConverter.TryWriteBytes(destination[5..7], EntryCount);
BitConverter.TryWriteBytes(destination[7..11], ParentPageId);
@@ -140,17 +139,14 @@ public struct BTreeNodeHeader
var header = new BTreeNodeHeader
{
PageId = BitConverter.ToUInt32(source[0..4]),
PageId = BitConverter.ToUInt32(source[..4]),
IsLeaf = source[4] != 0,
EntryCount = BitConverter.ToUInt16(source[5..7]),
ParentPageId = BitConverter.ToUInt32(source[7..11]),
NextLeafPageId = BitConverter.ToUInt32(source[11..15])
};
if (source.Length >= 20)
{
header.PrevLeafPageId = BitConverter.ToUInt32(source[15..19]);
}
if (source.Length >= 20) header.PrevLeafPageId = BitConverter.ToUInt32(source[15..19]);
return header;
}

89
src/CBDD.Core/Indexing/CollectionIndexDefinition.cs
View File

@@ -1,6 +1,4 @@
using System;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -11,6 +9,44 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionIndexDefinition<T> where T : class
{
/// <summary>
/// Creates a new index definition
/// </summary>
/// <param name="name">Index name</param>
/// <param name="propertyPaths">Property paths for the index</param>
/// <param name="keySelectorExpression">Expression to extract key from document</param>
/// <param name="isUnique">Enforce uniqueness</param>
/// <param name="type">Index structure type (BTree or Hash)</param>
/// <param name="isPrimary">Is this the primary key index</param>
/// <param name="dimensions">The vector dimensions for vector indexes.</param>
/// <param name="metric">The distance metric for vector indexes.</param>
public CollectionIndexDefinition(
string name,
string[] propertyPaths,
Expression<Func<T, object>> keySelectorExpression,
bool isUnique = false,
IndexType type = IndexType.BTree,
bool isPrimary = false,
int dimensions = 0,
VectorMetric metric = VectorMetric.Cosine)
{
if (string.IsNullOrWhiteSpace(name))
throw new ArgumentException("Index name cannot be empty", nameof(name));
if (propertyPaths == null || propertyPaths.Length == 0)
throw new ArgumentException("Property paths cannot be empty", nameof(propertyPaths));
Name = name;
PropertyPaths = propertyPaths;
KeySelectorExpression = keySelectorExpression ?? throw new ArgumentNullException(nameof(keySelectorExpression));
KeySelector = keySelectorExpression.Compile(); // Compile for performance
IsUnique = isUnique;
Type = type;
IsPrimary = isPrimary;
Dimensions = dimensions;
Metric = metric;
}
/// <summary>
/// Unique name for this index (auto-generated or user-specified)
/// </summary>
@@ -54,44 +90,6 @@ public sealed class CollectionIndexDefinition<T> where T : class
/// </summary>
public bool IsPrimary { get; }
/// <summary>
/// Creates a new index definition
/// </summary>
/// <param name="name">Index name</param>
/// <param name="propertyPaths">Property paths for the index</param>
/// <param name="keySelectorExpression">Expression to extract key from document</param>
/// <param name="isUnique">Enforce uniqueness</param>
/// <param name="type">Index structure type (BTree or Hash)</param>
/// <param name="isPrimary">Is this the primary key index</param>
/// <param name="dimensions">The vector dimensions for vector indexes.</param>
/// <param name="metric">The distance metric for vector indexes.</param>
public CollectionIndexDefinition(
string name,
string[] propertyPaths,
Expression<Func<T, object>> keySelectorExpression,
bool isUnique = false,
IndexType type = IndexType.BTree,
bool isPrimary = false,
int dimensions = 0,
VectorMetric metric = VectorMetric.Cosine)
{
if (string.IsNullOrWhiteSpace(name))
throw new ArgumentException("Index name cannot be empty", nameof(name));
if (propertyPaths == null || propertyPaths.Length == 0)
throw new ArgumentException("Property paths cannot be empty", nameof(propertyPaths));
Name = name;
PropertyPaths = propertyPaths;
KeySelectorExpression = keySelectorExpression ?? throw new ArgumentNullException(nameof(keySelectorExpression));
KeySelector = keySelectorExpression.Compile(); // Compile for performance
IsUnique = isUnique;
Type = type;
IsPrimary = isPrimary;
Dimensions = dimensions;
Metric = metric;
}
/// <summary>
/// Converts this high-level definition to low-level IndexOptions for BTreeIndex
/// </summary>
@@ -136,11 +134,9 @@ public sealed class CollectionIndexDefinition<T> where T : class
if (propertyPaths.Length > PropertyPaths.Length)
return false;
for (int i = 0; i < propertyPaths.Length; i++)
{
for (var i = 0; i < propertyPaths.Length; i++)
if (!PropertyPaths[i].Equals(propertyPaths[i], StringComparison.OrdinalIgnoreCase))
return false;
}
return true;
}
@@ -148,8 +144,8 @@ public sealed class CollectionIndexDefinition<T> where T : class
/// <inheritdoc />
public override string ToString()
{
var uniqueStr = IsUnique ? "Unique" : "Non-Unique";
var paths = string.Join(", ", PropertyPaths);
string uniqueStr = IsUnique ? "Unique" : "Non-Unique";
string paths = string.Join(", ", PropertyPaths);
return $"{Name} ({uniqueStr} {Type} on [{paths}])";
}
}
@@ -197,6 +193,7 @@ public sealed class CollectionIndexInfo
/// <inheritdoc />
public override string ToString()
{
return $"{Name}: {string.Join(", ", PropertyPaths)} ({EstimatedDocumentCount} docs, {EstimatedSizeBytes:N0} bytes)";
return
$"{Name}: {string.Join(", ", PropertyPaths)} ({EstimatedDocumentCount} docs, {EstimatedSizeBytes:N0} bytes)";
}
}

191
src/CBDD.Core/Indexing/CollectionIndexManager.cs
View File

@@ -1,7 +1,4 @@
using System;
using System.Collections.Generic;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
@@ -16,16 +13,16 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
{
private readonly Dictionary<string, CollectionSecondaryIndex<TId, T>> _indexes;
private readonly IStorageEngine _storage;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly object _lock = new();
private bool _disposed;
private readonly string _collectionName;
private readonly Dictionary<string, CollectionSecondaryIndex<TId, T>> _indexes;
private readonly object _lock = new();
private readonly IDocumentMapper<TId, T> _mapper;
private readonly IStorageEngine _storage;
private bool _disposed;
private CollectionMetadata _metadata;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}"/> class.
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}" /> class.
/// </summary>
/// <param name="storage">The storage engine used to persist index data and metadata.</param>
/// <param name="mapper">The document mapper for the collection type.</param>
@@ -36,12 +33,13 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}"/> class from the storage abstraction.
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}" /> class from the storage abstraction.
/// </summary>
/// <param name="storage">The storage abstraction used to persist index state.</param>
/// <param name="mapper">The document mapper for the collection.</param>
/// <param name="collectionName">An optional collection name override.</param>
internal CollectionIndexManager(IStorageEngine storage, IDocumentMapper<TId, T> mapper, string? collectionName = null)
internal CollectionIndexManager(IStorageEngine storage, IDocumentMapper<TId, T> mapper,
string? collectionName = null)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_mapper = mapper ?? throw new ArgumentNullException(nameof(mapper));
@@ -49,17 +47,53 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
_indexes = new Dictionary<string, CollectionSecondaryIndex<TId, T>>(StringComparer.OrdinalIgnoreCase);
// Load existing metadata via storage
_metadata = _storage.GetCollectionMetadata(_collectionName) ?? new CollectionMetadata { Name = _collectionName };
_metadata = _storage.GetCollectionMetadata(_collectionName) ??
new CollectionMetadata { Name = _collectionName };
// Initialize indexes from metadata
foreach (var idxMeta in _metadata.Indexes)
{
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type, idxMeta.Dimensions, idxMeta.Metric);
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type,
idxMeta.Dimensions, idxMeta.Metric);
var index = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper, idxMeta.RootPageId);
_indexes[idxMeta.Name] = index;
}
}
/// <summary>
/// Gets the root page identifier for the primary index.
/// </summary>
public uint PrimaryRootPageId => _metadata.PrimaryRootPageId;
/// <summary>
/// Releases resources used by the index manager.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// No auto-save on dispose to avoid unnecessary I/O if no changes
lock (_lock)
{
foreach (var index in _indexes.Values)
try
{
index.Dispose();
}
catch
{
/* Best effort */
}
_indexes.Clear();
_disposed = true;
}
GC.SuppressFinalize(this);
}
private void UpdateMetadata()
{
_metadata.Indexes.Clear();
@@ -129,7 +163,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
throw new ArgumentNullException(nameof(keySelector));
// Extract property paths from expression
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
// Generate name if not provided
name ??= GenerateIndexName(propertyPaths);
@@ -158,10 +192,11 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
/// <param name="metric">Distance metric used by the vector index.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
public CollectionSecondaryIndex<TId, T> CreateVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector,
int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
@@ -169,15 +204,13 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
// Reuse the original parameter from keySelector to avoid invalid expression trees.
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector, false, dimensions, metric);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector,
false, dimensions, metric);
return CreateIndex(definition);
}
}
@@ -194,7 +227,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
string? name = null,
bool unique = false)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
name ??= GenerateIndexName(propertyPaths);
lock (_lock)
@@ -220,10 +253,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
{
// Convert LambdaExpression to Expression<Func<T, object>> properly by sharing parameters
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
@@ -244,8 +274,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
VectorMetric metric = VectorMetric.Cosine,
string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
@@ -253,14 +283,12 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector, false, dimensions, metric);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector,
false, dimensions, metric);
return CreateIndex(definition);
}
}
@@ -275,8 +303,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
LambdaExpression keySelector,
string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
@@ -284,14 +312,12 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Spatial);
var definition =
new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Spatial);
return CreateIndex(definition);
}
}
@@ -425,10 +451,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Insert(document, location, transaction);
}
foreach (var index in _indexes.Values) index.Insert(document, location, transaction);
}
}
@@ -440,7 +463,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
/// <param name="oldLocation">Physical location of old document</param>
/// <param name="newLocation">Physical location of new document</param>
/// <param name="transaction">Transaction context</param>
public void UpdateInAll(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation, ITransaction transaction)
public void UpdateInAll(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation,
ITransaction transaction)
{
if (oldDocument == null)
throw new ArgumentNullException(nameof(oldDocument));
@@ -450,11 +474,9 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Update(oldDocument, newDocument, oldLocation, newLocation, transaction);
}
}
}
/// <summary>
/// Deletes a document from all indexes
@@ -469,10 +491,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Delete(document, location, transaction);
}
foreach (var index in _indexes.Values) index.Delete(document, location, transaction);
}
}
@@ -484,20 +503,17 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return $"idx_{string.Join("_", propertyPaths)}";
}
private CollectionIndexDefinition<T> RebuildDefinition(string name, string[] paths, bool isUnique, IndexType type, int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
private CollectionIndexDefinition<T> RebuildDefinition(string name, string[] paths, bool isUnique, IndexType type,
int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
{
var param = Expression.Parameter(typeof(T), "u");
Expression body;
if (paths.Length == 1)
{
body = Expression.PropertyOrField(param, paths[0]);
}
else
{
body = Expression.NewArrayInit(typeof(object),
paths.Select(p => Expression.Convert(Expression.PropertyOrField(param, p), typeof(object))));
}
var objectBody = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(objectBody, param);
@@ -505,11 +521,6 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return new CollectionIndexDefinition<T>(name, paths, lambda, isUnique, type, false, dimensions, metric);
}
/// <summary>
/// Gets the root page identifier for the primary index.
/// </summary>
public uint PrimaryRootPageId => _metadata.PrimaryRootPageId;
/// <summary>
/// Rebinds cached metadata and index instances from persisted metadata.
/// </summary>
@@ -525,8 +536,13 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
throw new ObjectDisposedException(nameof(CollectionIndexManager<TId, T>));
foreach (var index in _indexes.Values)
try
{
try { index.Dispose(); } catch { /* Best effort */ }
index.Dispose();
}
catch
{
/* Best effort */
}
_indexes.Clear();
@@ -534,7 +550,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
foreach (var idxMeta in _metadata.Indexes)
{
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type, idxMeta.Dimensions, idxMeta.Metric);
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type,
idxMeta.Dimensions, idxMeta.Metric);
var index = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper, idxMeta.RootPageId);
_indexes[idxMeta.Name] = index;
}
@@ -561,37 +578,16 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
/// Gets the current collection metadata.
/// </summary>
/// <returns>The collection metadata.</returns>
public CollectionMetadata GetMetadata() => _metadata;
public CollectionMetadata GetMetadata()
{
return _metadata;
}
private void SaveMetadata()
{
UpdateMetadata();
_storage.SaveCollectionMetadata(_metadata);
}
/// <summary>
/// Releases resources used by the index manager.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// No auto-save on dispose to avoid unnecessary I/O if no changes
lock (_lock)
{
foreach (var index in _indexes.Values)
{
try { index.Dispose(); } catch { /* Best effort */ }
}
_indexes.Clear();
_disposed = true;
}
GC.SuppressFinalize(this);
}
}
/// <summary>
@@ -607,35 +603,30 @@ public static class ExpressionAnalyzer
public static string[] ExtractPropertyPaths(LambdaExpression expression)
{
if (expression.Body is MemberExpression memberExpr)
{
// Simple property: p => p.Age
return new[] { memberExpr.Member.Name };
}
else if (expression.Body is NewExpression newExpr)
{
if (expression.Body is NewExpression newExpr)
// Compound key via anonymous type: p => new { p.City, p.Age }
return newExpr.Arguments
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
else if (expression.Body is UnaryExpression { NodeType: ExpressionType.Convert } unaryExpr)
if (expression.Body is UnaryExpression { NodeType: ExpressionType.Convert } unaryExpr)
{
// Handle Convert(Member) or Convert(New)
if (unaryExpr.Operand is MemberExpression innerMember)
{
// Wrapped property: p => (object)p.Age
return new[] { innerMember.Member.Name };
}
else if (unaryExpr.Operand is NewExpression innerNew)
{
if (unaryExpr.Operand is NewExpression innerNew)
// Wrapped anonymous type: p => (object)new { p.City, p.Age }
return innerNew.Arguments
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
}
throw new ArgumentException(
"Expression must be a property accessor (p => p.Property) or anonymous type (p => new { p.Prop1, p.Prop2 })",

217
src/CBDD.Core/Indexing/CollectionSecondaryIndex.cs
View File

@@ -1,11 +1,8 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using System;
using System.Linq;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -18,30 +15,13 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : class
{
private readonly CollectionIndexDefinition<T> _definition;
private readonly BTreeIndex? _btreeIndex;
private readonly VectorSearchIndex? _vectorIndex;
private readonly RTreeIndex? _spatialIndex;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly RTreeIndex? _spatialIndex;
private readonly VectorSearchIndex? _vectorIndex;
private bool _disposed;
/// <summary>
/// Gets the index definition
/// </summary>
public CollectionIndexDefinition<T> Definition => _definition;
/// <summary>
/// Gets the underlying BTree index (for advanced scenarios)
/// </summary>
public BTreeIndex? BTreeIndex => _btreeIndex;
/// <summary>
/// Gets the root page identifier for the underlying index structure.
/// </summary>
public uint RootPageId => _btreeIndex?.RootPageId ?? _vectorIndex?.RootPageId ?? _spatialIndex?.RootPageId ?? 0;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}"/> class.
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}" /> class.
/// </summary>
/// <param name="definition">The index definition.</param>
/// <param name="storage">The storage engine.</param>
@@ -57,7 +37,8 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}"/> class from index storage abstractions.
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}" /> class from index storage
/// abstractions.
/// </summary>
/// <param name="definition">The index definition.</param>
/// <param name="storage">The index storage abstraction.</param>
@@ -69,7 +50,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
IDocumentMapper<TId, T> mapper,
uint rootPageId = 0)
{
_definition = definition ?? throw new ArgumentNullException(nameof(definition));
Definition = definition ?? throw new ArgumentNullException(nameof(definition));
_mapper = mapper ?? throw new ArgumentNullException(nameof(mapper));
var indexOptions = definition.ToIndexOptions();
@@ -77,23 +58,53 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
if (indexOptions.Type == IndexType.Vector)
{
_vectorIndex = new VectorSearchIndex(storage, indexOptions, rootPageId);
_btreeIndex = null;
BTreeIndex = null;
_spatialIndex = null;
}
else if (indexOptions.Type == IndexType.Spatial)
{
_spatialIndex = new RTreeIndex(storage, indexOptions, rootPageId);
_btreeIndex = null;
BTreeIndex = null;
_vectorIndex = null;
}
else
{
_btreeIndex = new BTreeIndex(storage, indexOptions, rootPageId);
BTreeIndex = new BTreeIndex(storage, indexOptions, rootPageId);
_vectorIndex = null;
_spatialIndex = null;
}
}
/// <summary>
/// Gets the index definition
/// </summary>
public CollectionIndexDefinition<T> Definition { get; }
/// <summary>
/// Gets the underlying BTree index (for advanced scenarios)
/// </summary>
public BTreeIndex? BTreeIndex { get; }
/// <summary>
/// Gets the root page identifier for the underlying index structure.
/// </summary>
public uint RootPageId => BTreeIndex?.RootPageId ?? _vectorIndex?.RootPageId ?? _spatialIndex?.RootPageId ?? 0;
/// <summary>
/// Releases resources used by this index wrapper.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// BTreeIndex doesn't currently implement IDisposable
// Future: may need to flush buffers, close resources
_disposed = true;
GC.SuppressFinalize(this);
}
/// <summary>
/// Inserts a document into this index
/// </summary>
@@ -106,7 +117,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
throw new ArgumentNullException(nameof(document));
// Extract key using compiled selector (fast!)
var keyValue = _definition.KeySelector(document);
object? keyValue = Definition.KeySelector(document);
if (keyValue == null)
return; // Skip null keys
@@ -114,32 +125,24 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
// Vector Index Support
if (keyValue is float[] singleVector)
{
_vectorIndex.Insert(singleVector, location, transaction);
}
else if (keyValue is IEnumerable<float[]> vectors)
{
foreach (var v in vectors)
{
foreach (float[] v in vectors)
_vectorIndex.Insert(v, location, transaction);
}
}
}
else if (_spatialIndex != null)
{
// Geospatial Index Support
if (keyValue is ValueTuple<double, double> t)
{
_spatialIndex.Insert(GeoBox.FromPoint(new GeoPoint(t.Item1, t.Item2)), location, transaction);
}
}
else if (_btreeIndex != null)
else if (BTreeIndex != null)
{
// BTree Index logic
var userKey = ConvertToIndexKey(keyValue);
var documentId = _mapper.GetId(document);
var compositeKey = CreateCompositeKey(userKey, _mapper.ToIndexKey(documentId));
_btreeIndex.Insert(compositeKey, location, transaction?.TransactionId);
BTreeIndex.Insert(compositeKey, location, transaction?.TransactionId);
}
}
@@ -152,7 +155,8 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
/// <param name="oldLocation">Physical location of old document</param>
/// <param name="newLocation">Physical location of new document</param>
/// <param name="transaction">Optional transaction</param>
public void Update(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation, ITransaction transaction)
public void Update(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation,
ITransaction transaction)
{
if (oldDocument == null)
throw new ArgumentNullException(nameof(oldDocument));
@@ -160,8 +164,8 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
throw new ArgumentNullException(nameof(newDocument));
// Extract keys from both versions
var oldKey = _definition.KeySelector(oldDocument);
var newKey = _definition.KeySelector(newDocument);
object? oldKey = Definition.KeySelector(oldDocument);
object? newKey = Definition.KeySelector(newDocument);
// If keys are the same, no index update needed (optimization)
if (Equals(oldKey, newKey))
@@ -174,7 +178,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
var oldUserKey = ConvertToIndexKey(oldKey);
var oldCompositeKey = CreateCompositeKey(oldUserKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Delete(oldCompositeKey, oldLocation, transaction?.TransactionId);
BTreeIndex?.Delete(oldCompositeKey, oldLocation, transaction?.TransactionId);
}
// Insert new entry if it has a key
@@ -182,7 +186,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
var newUserKey = ConvertToIndexKey(newKey);
var newCompositeKey = CreateCompositeKey(newUserKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Insert(newCompositeKey, newLocation, transaction?.TransactionId);
BTreeIndex?.Insert(newCompositeKey, newLocation, transaction?.TransactionId);
}
}
@@ -198,7 +202,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
throw new ArgumentNullException(nameof(document));
// Extract key
var keyValue = _definition.KeySelector(document);
object? keyValue = Definition.KeySelector(document);
if (keyValue == null)
return; // Nothing to delete
@@ -207,7 +211,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
// Create composite key and delete
var compositeKey = CreateCompositeKey(userKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Delete(compositeKey, location, transaction?.TransactionId);
BTreeIndex?.Delete(compositeKey, location, transaction?.TransactionId);
}
/// <summary>
@@ -222,17 +226,16 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
return null;
if (_vectorIndex != null && key is float[] query)
{
return _vectorIndex.Search(query, 1, transaction: transaction).FirstOrDefault().Location;
}
if (_btreeIndex != null)
if (BTreeIndex != null)
{
var userKey = ConvertToIndexKey(key);
var minComposite = CreateCompositeKeyBoundary(userKey, useMinObjectId: true);
var maxComposite = CreateCompositeKeyBoundary(userKey, useMinObjectId: false);
var firstEntry = _btreeIndex.Range(minComposite, maxComposite, IndexDirection.Forward, transaction?.TransactionId).FirstOrDefault();
return firstEntry.Location.PageId == 0 ? null : (DocumentLocation?)firstEntry.Location;
var minComposite = CreateCompositeKeyBoundary(userKey, true);
var maxComposite = CreateCompositeKeyBoundary(userKey, false);
var firstEntry = BTreeIndex
.Range(minComposite, maxComposite, IndexDirection.Forward, transaction?.TransactionId).FirstOrDefault();
return firstEntry.Location.PageId == 0 ? null : firstEntry.Location;
}
return null;
@@ -246,7 +249,8 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction.</param>
/// <returns>The matching vector search results.</returns>
public IEnumerable<VectorSearchResult> VectorSearch(float[] query, int k, int efSearch = 100, ITransaction? transaction = null)
public IEnumerable<VectorSearchResult> VectorSearch(float[] query, int k, int efSearch = 100,
ITransaction? transaction = null)
{
if (_vectorIndex == null)
throw new InvalidOperationException("This index is not a vector index.");
@@ -260,16 +264,14 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
/// <param name="center">The center point.</param>
/// <param name="radiusKm">The search radius in kilometers.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Near((double Latitude, double Longitude) center, double radiusKm, ITransaction? transaction = null)
public IEnumerable<DocumentLocation> Near((double Latitude, double Longitude) center, double radiusKm,
ITransaction? transaction = null)
{
if (_spatialIndex == null)
throw new InvalidOperationException("This index is not a spatial index.");
var queryBox = SpatialMath.BoundingBox(center.Latitude, center.Longitude, radiusKm);
foreach (var loc in _spatialIndex.Search(queryBox, transaction))
{
yield return loc;
}
foreach (var loc in _spatialIndex.Search(queryBox, transaction)) yield return loc;
}
/// <summary>
@@ -278,7 +280,8 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
/// <param name="min">The minimum latitude/longitude corner.</param>
/// <param name="max">The maximum latitude/longitude corner.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Within((double Latitude, double Longitude) min, (double Latitude, double Longitude) max, ITransaction? transaction = null)
public IEnumerable<DocumentLocation> Within((double Latitude, double Longitude) min,
(double Latitude, double Longitude) max, ITransaction? transaction = null)
{
if (_spatialIndex == null)
throw new InvalidOperationException("This index is not a spatial index.");
@@ -295,9 +298,10 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
/// <param name="direction">Scan direction.</param>
/// <param name="transaction">Optional transaction to read uncommitted changes</param>
/// <returns>Enumerable of document locations in key order</returns>
public IEnumerable<DocumentLocation> Range(object? minKey, object? maxKey, IndexDirection direction = IndexDirection.Forward, ITransaction? transaction = null)
public IEnumerable<DocumentLocation> Range(object? minKey, object? maxKey,
IndexDirection direction = IndexDirection.Forward, ITransaction? transaction = null)
{
if (_btreeIndex == null) yield break;
if (BTreeIndex == null) yield break;
// Handle unbounded ranges
IndexKey actualMinKey;
@@ -313,12 +317,12 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
actualMinKey = new IndexKey(new byte[0]);
var userMaxKey = ConvertToIndexKey(maxKey!);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, useMinObjectId: false); // Max boundary
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, false); // Max boundary
}
else if (maxKey == null)
{
var userMinKey = ConvertToIndexKey(minKey);
actualMinKey = CreateCompositeKeyBoundary(userMinKey, useMinObjectId: true); // Min boundary
actualMinKey = CreateCompositeKeyBoundary(userMinKey, true); // Min boundary
actualMaxKey = new IndexKey(Enumerable.Repeat((byte)0xFF, 255).ToArray());
}
else
@@ -330,17 +334,15 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
// Create composite boundaries:
// Min: (userMinKey, ObjectId.Empty) - captures all docs with key >= userMinKey
// Max: (userMaxKey, ObjectId.MaxValue) - captures all docs with key <= userMaxKey
actualMinKey = CreateCompositeKeyBoundary(userMinKey, useMinObjectId: true);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, useMinObjectId: false);
actualMinKey = CreateCompositeKeyBoundary(userMinKey, true);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, false);
}
// Use BTreeIndex.Range with WAL-aware reads and direction
// Extract DocumentLocation from each entry
foreach (var entry in _btreeIndex.Range(actualMinKey, actualMaxKey, direction, transaction?.TransactionId))
{
foreach (var entry in BTreeIndex.Range(actualMinKey, actualMaxKey, direction, transaction?.TransactionId))
yield return entry.Location;
}
}
/// <summary>
/// Gets statistics about this index
@@ -349,16 +351,38 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
return new CollectionIndexInfo
{
Name = _definition.Name,
PropertyPaths = _definition.PropertyPaths,
IsUnique = _definition.IsUnique,
Type = _definition.Type,
IsPrimary = _definition.IsPrimary,
Name = Definition.Name,
PropertyPaths = Definition.PropertyPaths,
IsUnique = Definition.IsUnique,
Type = Definition.Type,
IsPrimary = Definition.IsPrimary,
EstimatedDocumentCount = 0, // TODO: Track or calculate document count
EstimatedSizeBytes = 0 // TODO: Calculate index size
};
}
/// <summary>
/// Converts a CLR value to an IndexKey for BTree storage.
/// Supports all common .NET types.
/// </summary>
private IndexKey ConvertToIndexKey(object value)
{
return value switch
{
ObjectId objectId => new IndexKey(objectId),
string str => new IndexKey(str),
int intVal => new IndexKey(intVal),
long longVal => new IndexKey(longVal),
DateTime dateTime => new IndexKey(dateTime.Ticks),
bool boolVal => new IndexKey(boolVal ? 1 : 0),
byte[] byteArray => new IndexKey(byteArray),
// For compound keys or complex types, use ToString and serialize
// TODO: Better compound key serialization
_ => new IndexKey(value.ToString() ?? string.Empty)
};
}
#region Composite Key Support (SQLite-style for Duplicate Keys)
/// <summary>
@@ -388,7 +412,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
// For range boundaries, we use an empty key for Min and a very large key for Max
// to wrap around all possible IDs for this user key.
IndexKey idBoundary = useMinObjectId
var idBoundary = useMinObjectId
? new IndexKey(Array.Empty<byte>())
: new IndexKey(Enumerable.Repeat((byte)0xFF, 16).ToArray()); // Using 16 as a safe max for GUID/ObjectId
@@ -402,7 +426,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
private IndexKey ExtractUserKey(IndexKey compositeKey)
{
// Composite key = UserKey + ObjectId(12 bytes)
var userKeyLength = compositeKey.Data.Length - 12;
int userKeyLength = compositeKey.Data.Length - 12;
if (userKeyLength <= 0)
return compositeKey; // Fallback for malformed keys
@@ -411,41 +435,4 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
}
#endregion
/// <summary>
/// Converts a CLR value to an IndexKey for BTree storage.
/// Supports all common .NET types.
/// </summary>
private IndexKey ConvertToIndexKey(object value)
{
return value switch
{
ObjectId objectId => new IndexKey(objectId),
string str => new IndexKey(str),
int intVal => new IndexKey(intVal),
long longVal => new IndexKey(longVal),
DateTime dateTime => new IndexKey(dateTime.Ticks),
bool boolVal => new IndexKey(boolVal ? 1 : 0),
byte[] byteArray => new IndexKey(byteArray),
// For compound keys or complex types, use ToString and serialize
// TODO: Better compound key serialization
_ => new IndexKey(value.ToString() ?? string.Empty)
};
}
/// <summary>
/// Releases resources used by this index wrapper.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// BTreeIndex doesn't currently implement IDisposable
// Future: may need to flush buffers, close resources
_disposed = true;
GC.SuppressFinalize(this);
}
}

6
src/CBDD.Core/Indexing/GeoSpatialExtensions.cs
View File

@@ -10,7 +10,8 @@ public static class GeoSpatialExtensions
/// <param name="center">The center point (Latitude, Longitude) for the proximity search.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>True if the point is within the specified radius.</returns>
public static bool Near(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) center, double radiusKm)
public static bool Near(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) center,
double radiusKm)
{
return true;
}
@@ -23,7 +24,8 @@ public static class GeoSpatialExtensions
/// <param name="min">The minimum (Latitude, Longitude) of the bounding box.</param>
/// <param name="max">The maximum (Latitude, Longitude) of the bounding box.</param>
/// <returns>True if the point is within the specified bounding box.</returns>
public static bool Within(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) min, (double Latitude, double Longitude) max)
public static bool Within(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) min,
(double Latitude, double Longitude) max)
{
return true;
}

27
src/CBDD.Core/Indexing/HashIndex.cs
View File

@@ -1,7 +1,4 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -15,7 +12,7 @@ public sealed class HashIndex
private readonly IndexOptions _options;
/// <summary>
/// Initializes a new instance of the <see cref="HashIndex"/> class.
/// Initializes a new instance of the <see cref="HashIndex" /> class.
/// </summary>
/// <param name="options">The index options.</param>
public HashIndex(IndexOptions options)
@@ -32,9 +29,9 @@ public sealed class HashIndex
public void Insert(IndexKey key, DocumentLocation location)
{
if (_options.Unique && TryFind(key, out _))
throw new InvalidOperationException($"Duplicate key violation for unique index");
throw new InvalidOperationException("Duplicate key violation for unique index");
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
{
@@ -50,23 +47,21 @@ public sealed class HashIndex
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">When this method returns, contains the matched document location if found.</param>
/// <returns><see langword="true"/> if a matching entry is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if a matching entry is found; otherwise, <see langword="false" />.</returns>
public bool TryFind(IndexKey key, out DocumentLocation location)
{
location = default;
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
return false;
foreach (var entry in bucket)
{
if (entry.Key == key)
{
location = entry.Location;
return true;
}
}
return false;
}
@@ -76,16 +71,15 @@ public sealed class HashIndex
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
/// <returns><see langword="true"/> if an entry is removed; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if an entry is removed; otherwise, <see langword="false" />.</returns>
public bool Remove(IndexKey key, DocumentLocation location)
{
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
return false;
for (int i = 0; i < bucket.Count; i++)
{
for (var i = 0; i < bucket.Count; i++)
if (bucket[i].Key == key &&
bucket[i].Location.PageId == location.PageId &&
bucket[i].Location.SlotIndex == location.SlotIndex)
@@ -97,7 +91,6 @@ public sealed class HashIndex
return true;
}
}
return false;
}
@@ -109,15 +102,13 @@ public sealed class HashIndex
/// <returns>All matching index entries.</returns>
public IEnumerable<IndexEntry> FindAll(IndexKey key)
{
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
yield break;
foreach (var entry in bucket)
{
if (entry.Key == key)
yield return entry;
}
}
}

3
src/CBDD.Core/Indexing/IBTreeCursor.cs
View File

@@ -1,6 +1,3 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>

89
src/CBDD.Core/Indexing/IndexKey.cs
View File

@@ -1,6 +1,5 @@
using System.Text;
using ZB.MOM.WW.CBDD.Bson;
using System;
using System.Linq;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -17,15 +16,15 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
/// <summary>
/// Gets the minimum possible index key.
/// </summary>
public static IndexKey MinKey => new IndexKey(Array.Empty<byte>());
public static IndexKey MinKey => new(Array.Empty<byte>());
/// <summary>
/// Gets the maximum possible index key.
/// </summary>
public static IndexKey MaxKey => new IndexKey(Enumerable.Repeat((byte)0xFF, 32).ToArray());
public static IndexKey MaxKey => new(Enumerable.Repeat((byte)0xFF, 32).ToArray());
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from raw key bytes.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from raw key bytes.
/// </summary>
/// <param name="data">The key bytes.</param>
public IndexKey(ReadOnlySpan<byte> data)
@@ -35,7 +34,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from an object identifier.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from an object identifier.
/// </summary>
/// <param name="objectId">The object identifier value.</param>
public IndexKey(ObjectId objectId)
@@ -46,7 +45,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a 32-bit integer.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a 32-bit integer.
/// </summary>
/// <param name="value">The integer value.</param>
public IndexKey(int value)
@@ -56,7 +55,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a 64-bit integer.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a 64-bit integer.
/// </summary>
/// <param name="value">The integer value.</param>
public IndexKey(long value)
@@ -66,17 +65,17 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a string.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a string.
/// </summary>
/// <param name="value">The string value.</param>
public IndexKey(string value)
{
_data = System.Text.Encoding.UTF8.GetBytes(value);
_data = Encoding.UTF8.GetBytes(value);
_hashCode = ComputeHashCode(_data);
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a GUID.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a GUID.
/// </summary>
/// <param name="value">The GUID value.</param>
public IndexKey(Guid value)
@@ -95,18 +94,19 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
/// </summary>
/// <param name="other">The key to compare with.</param>
/// <returns>
/// A value less than zero if this key is less than <paramref name="other"/>, zero if equal, or greater than zero if greater.
/// A value less than zero if this key is less than <paramref name="other" />, zero if equal, or greater than zero if
/// greater.
/// </returns>
public readonly int CompareTo(IndexKey other)
{
if (_data == null) return other._data == null ? 0 : -1;
if (other._data == null) return 1;
var minLength = Math.Min(_data.Length, other._data.Length);
int minLength = Math.Min(_data.Length, other._data.Length);
for (int i = 0; i < minLength; i++)
for (var i = 0; i < minLength; i++)
{
var cmp = _data[i].CompareTo(other._data[i]);
int cmp = _data[i].CompareTo(other._data[i]);
if (cmp != 0)
return cmp;
}
@@ -118,7 +118,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
/// Determines whether this key equals another key.
/// </summary>
/// <param name="other">The key to compare with.</param>
/// <returns><see langword="true"/> if the keys are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the keys are equal; otherwise, <see langword="false" />.</returns>
public readonly bool Equals(IndexKey other)
{
if (_hashCode != other._hashCode)
@@ -131,17 +131,46 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <inheritdoc />
public override readonly bool Equals(object? obj) => obj is IndexKey other && Equals(other);
public readonly override bool Equals(object? obj)
{
return obj is IndexKey other && Equals(other);
}
/// <inheritdoc />
public override readonly int GetHashCode() => _hashCode;
public readonly override int GetHashCode()
{
return _hashCode;
}
public static bool operator ==(IndexKey left, IndexKey right) => left.Equals(right);
public static bool operator !=(IndexKey left, IndexKey right) => !left.Equals(right);
public static bool operator <(IndexKey left, IndexKey right) => left.CompareTo(right) < 0;
public static bool operator >(IndexKey left, IndexKey right) => left.CompareTo(right) > 0;
public static bool operator <=(IndexKey left, IndexKey right) => left.CompareTo(right) <= 0;
public static bool operator >=(IndexKey left, IndexKey right) => left.CompareTo(right) >= 0;
public static bool operator ==(IndexKey left, IndexKey right)
{
return left.Equals(right);
}
public static bool operator !=(IndexKey left, IndexKey right)
{
return !left.Equals(right);
}
public static bool operator <(IndexKey left, IndexKey right)
{
return left.CompareTo(right) < 0;
}
public static bool operator >(IndexKey left, IndexKey right)
{
return left.CompareTo(right) > 0;
}
public static bool operator <=(IndexKey left, IndexKey right)
{
return left.CompareTo(right) <= 0;
}
public static bool operator >=(IndexKey left, IndexKey right)
{
return left.CompareTo(right) >= 0;
}
private static int ComputeHashCode(ReadOnlySpan<byte> data)
{
@@ -151,7 +180,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Creates an <see cref="IndexKey"/> from a supported CLR value.
/// Creates an <see cref="IndexKey" /> from a supported CLR value.
/// </summary>
/// <typeparam name="T">The CLR type of the value.</typeparam>
/// <param name="value">The value to convert.</param>
@@ -167,11 +196,12 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
if (typeof(T) == typeof(Guid)) return new IndexKey((Guid)(object)value);
if (typeof(T) == typeof(byte[])) return new IndexKey((byte[])(object)value);
throw new NotSupportedException($"Type {typeof(T).Name} is not supported as an IndexKey. Provide a custom mapping.");
throw new NotSupportedException(
$"Type {typeof(T).Name} is not supported as an IndexKey. Provide a custom mapping.");
}
/// <summary>
/// Converts this key to a CLR value of type <typeparamref name="T"/>.
/// Converts this key to a CLR value of type <typeparamref name="T" />.
/// </summary>
/// <typeparam name="T">The CLR type to read from this key.</typeparam>
/// <returns>The converted value.</returns>
@@ -182,10 +212,11 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
if (typeof(T) == typeof(ObjectId)) return (T)(object)new ObjectId(_data);
if (typeof(T) == typeof(int)) return (T)(object)BitConverter.ToInt32(_data);
if (typeof(T) == typeof(long)) return (T)(object)BitConverter.ToInt64(_data);
if (typeof(T) == typeof(string)) return (T)(object)System.Text.Encoding.UTF8.GetString(_data);
if (typeof(T) == typeof(string)) return (T)(object)Encoding.UTF8.GetString(_data);
if (typeof(T) == typeof(Guid)) return (T)(object)new Guid(_data);
if (typeof(T) == typeof(byte[])) return (T)(object)_data;
throw new NotSupportedException($"Type {typeof(T).Name} cannot be extracted from IndexKey. Provide a custom mapping.");
throw new NotSupportedException(
$"Type {typeof(T).Name} cannot be extracted from IndexKey. Provide a custom mapping.");
}
}

26
src/CBDD.Core/Indexing/IndexOptions.cs
View File

@@ -83,36 +83,45 @@ public readonly struct IndexOptions
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateBTree(params string[] fields) => new()
public static IndexOptions CreateBTree(params string[] fields)
{
return new IndexOptions
{
Type = IndexType.BTree,
Unique = false,
Fields = fields
};
}
/// <summary>
/// Creates unique B+Tree index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateUnique(params string[] fields) => new()
public static IndexOptions CreateUnique(params string[] fields)
{
return new IndexOptions
{
Type = IndexType.BTree,
Unique = true,
Fields = fields
};
}
/// <summary>
/// Creates hash index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateHash(params string[] fields) => new()
public static IndexOptions CreateHash(params string[] fields)
{
return new IndexOptions
{
Type = IndexType.Hash,
Unique = false,
Fields = fields
};
}
/// <summary>
/// Creates vector index options.
@@ -123,7 +132,10 @@ public readonly struct IndexOptions
/// <param name="ef">The candidate list size used during index construction.</param>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateVector(int dimensions, VectorMetric metric = VectorMetric.Cosine, int m = 16, int ef = 200, params string[] fields) => new()
public static IndexOptions CreateVector(int dimensions, VectorMetric metric = VectorMetric.Cosine, int m = 16,
int ef = 200, params string[] fields)
{
return new IndexOptions
{
Type = IndexType.Vector,
Unique = false,
@@ -133,16 +145,20 @@ public readonly struct IndexOptions
M = m,
EfConstruction = ef
};
}
/// <summary>
/// Creates spatial index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateSpatial(params string[] fields) => new()
public static IndexOptions CreateSpatial(params string[] fields)
{
return new IndexOptions
{
Type = IndexType.Spatial,
Unique = false,
Fields = fields
};
}
}

14
src/CBDD.Core/Indexing/Internal/GeoTypes.cs
View File

@@ -23,11 +23,16 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
/// </summary>
public static GeoBox Empty => new(double.MaxValue, double.MaxValue, double.MinValue, double.MinValue);
/// <summary>
/// Gets the area of this bounding box.
/// </summary>
public double Area => Math.Max(0, MaxLat - MinLat) * Math.Max(0, MaxLon - MinLon);
/// <summary>
/// Determines whether this box contains the specified point.
/// </summary>
/// <param name="point">The point to test.</param>
/// <returns><see langword="true"/> if the point is inside this box; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the point is inside this box; otherwise, <see langword="false" />.</returns>
public bool Contains(GeoPoint point)
{
return point.Latitude >= MinLat && point.Latitude <= MaxLat &&
@@ -38,7 +43,7 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
/// Determines whether this box intersects another box.
/// </summary>
/// <param name="other">The other box to test.</param>
/// <returns><see langword="true"/> if the boxes intersect; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the boxes intersect; otherwise, <see langword="false" />.</returns>
public bool Intersects(GeoBox other)
{
return !(other.MinLat > MaxLat || other.MaxLat < MinLat ||
@@ -82,9 +87,4 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
Math.Max(MaxLat, other.MaxLat),
Math.Max(MaxLon, other.MaxLon));
}
/// <summary>
/// Gets the area of this bounding box.
/// </summary>
public double Area => Math.Max(0, MaxLat - MinLat) * Math.Max(0, MaxLon - MinLon);
}

4
src/CBDD.Core/Indexing/InternalEntry.cs
View File

@@ -1,5 +1,3 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
public struct InternalEntry
@@ -15,7 +13,7 @@ public struct InternalEntry
public uint PageId { get; set; }
/// <summary>
/// Initializes a new instance of the <see cref="InternalEntry"/> struct.
/// Initializes a new instance of the <see cref="InternalEntry" /> struct.
/// </summary>
/// <param name="key">The separator key.</param>
/// <param name="pageId">The child page identifier.</param>

127
src/CBDD.Core/Indexing/RTreeIndex.cs
View File

@@ -1,8 +1,7 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using System.Buffers;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -12,14 +11,13 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// </summary>
internal class RTreeIndex : IDisposable
{
private readonly IIndexStorage _storage;
private readonly IndexOptions _options;
private uint _rootPageId;
private readonly object _lock = new();
private readonly IndexOptions _options;
private readonly int _pageSize;
private readonly IIndexStorage _storage;
/// <summary>
/// Initializes a new instance of the <see cref="RTreeIndex"/> class.
/// Initializes a new instance of the <see cref="RTreeIndex" /> class.
/// </summary>
/// <param name="storage">The storage engine used for page operations.</param>
/// <param name="options">The index options.</param>
@@ -28,30 +26,37 @@ internal class RTreeIndex : IDisposable
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
_rootPageId = rootPageId;
RootPageId = rootPageId;
_pageSize = _storage.PageSize;
if (_rootPageId == 0)
{
InitializeNewIndex();
}
if (RootPageId == 0) InitializeNewIndex();
}
/// <summary>
/// Gets the current root page identifier.
/// </summary>
public uint RootPageId => _rootPageId;
public uint RootPageId { get; private set; }
/// <summary>
/// Releases resources used by the index.
/// </summary>
public void Dispose()
{
}
private void InitializeNewIndex()
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_rootPageId = _storage.AllocatePage();
SpatialPage.Initialize(buffer, _rootPageId, true, 0);
_storage.WritePageImmediate(_rootPageId, buffer);
RootPageId = _storage.AllocatePage();
SpatialPage.Initialize(buffer, RootPageId, true, 0);
_storage.WritePageImmediate(RootPageId, buffer);
}
finally
{
ReturnPageBuffer(buffer);
}
finally { ReturnPageBuffer(buffer); }
}
/// <summary>
@@ -62,12 +67,12 @@ internal class RTreeIndex : IDisposable
/// <returns>A sequence of matching document locations.</returns>
public IEnumerable<DocumentLocation> Search(GeoBox area, ITransaction? transaction = null)
{
if (_rootPageId == 0) yield break;
if (RootPageId == 0) yield break;
var stack = new Stack<uint>();
stack.Push(_rootPageId);
stack.Push(RootPageId);
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
while (stack.Count > 0)
@@ -78,25 +83,24 @@ internal class RTreeIndex : IDisposable
bool isLeaf = SpatialPage.GetIsLeaf(buffer);
ushort count = SpatialPage.GetEntryCount(buffer);
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var mbr, out var pointer);
if (area.Intersects(mbr))
{
if (isLeaf)
{
yield return pointer;
}
else
{
stack.Push(pointer.PageId);
}
}
}
}
finally
{
ReturnPageBuffer(buffer);
}
finally { ReturnPageBuffer(buffer); }
}
/// <summary>
@@ -109,7 +113,7 @@ internal class RTreeIndex : IDisposable
{
lock (_lock)
{
var leafPageId = ChooseLeaf(_rootPageId, mbr, transaction);
uint leafPageId = ChooseLeaf(RootPageId, mbr, transaction);
InsertIntoNode(leafPageId, mbr, loc, transaction);
}
}
@@ -117,7 +121,7 @@ internal class RTreeIndex : IDisposable
private uint ChooseLeaf(uint rootId, GeoBox mbr, ITransaction? transaction)
{
uint currentId = rootId;
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
while (true)
@@ -127,10 +131,10 @@ internal class RTreeIndex : IDisposable
ushort count = SpatialPage.GetEntryCount(buffer);
uint bestChild = 0;
double minEnlargement = double.MaxValue;
double minArea = double.MaxValue;
var minEnlargement = double.MaxValue;
var minArea = double.MaxValue;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var childMbr, out var pointer);
@@ -156,12 +160,15 @@ internal class RTreeIndex : IDisposable
currentId = bestChild;
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void InsertIntoNode(uint pageId, GeoBox mbr, DocumentLocation pointer, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -186,17 +193,20 @@ internal class RTreeIndex : IDisposable
SplitNode(pageId, mbr, pointer, transaction);
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void UpdateMBRUpwards(uint pageId, ITransaction? transaction)
{
var buffer = RentPageBuffer();
var parentBuffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
byte[] parentBuffer = RentPageBuffer();
try
{
uint currentId = pageId;
while (currentId != _rootPageId)
while (currentId != RootPageId)
{
_storage.ReadPage(currentId, transaction?.TransactionId, buffer);
var currentMbr = SpatialPage.CalculateMBR(buffer);
@@ -206,9 +216,9 @@ internal class RTreeIndex : IDisposable
_storage.ReadPage(parentId, transaction?.TransactionId, parentBuffer);
ushort count = SpatialPage.GetEntryCount(parentBuffer);
bool changed = false;
var changed = false;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(parentBuffer, i, out var mbr, out var pointer);
if (pointer.PageId == currentId)
@@ -218,6 +228,7 @@ internal class RTreeIndex : IDisposable
SpatialPage.WriteEntry(parentBuffer, i, currentMbr, pointer);
changed = true;
}
break;
}
}
@@ -241,8 +252,8 @@ internal class RTreeIndex : IDisposable
private void SplitNode(uint pageId, GeoBox newMbr, DocumentLocation newPointer, ITransaction? transaction)
{
var buffer = RentPageBuffer();
var newBuffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
byte[] newBuffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -253,11 +264,12 @@ internal class RTreeIndex : IDisposable
// Collect all entries including the new one
var entries = new List<(GeoBox Mbr, DocumentLocation Pointer)>();
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var m, out var p);
entries.Add((m, p));
}
entries.Add((newMbr, newPointer));
// Pick Seeds
@@ -277,8 +289,8 @@ internal class RTreeIndex : IDisposable
SpatialPage.WriteEntry(newBuffer, 0, seed2.Mbr, seed2.Pointer);
SpatialPage.SetEntryCount(newBuffer, 1);
GeoBox mbr1 = seed1.Mbr;
GeoBox mbr2 = seed2.Mbr;
var mbr1 = seed1.Mbr;
var mbr2 = seed2.Mbr;
// Distribute remaining entries
while (entries.Count > 0)
@@ -320,7 +332,7 @@ internal class RTreeIndex : IDisposable
}
// Propagate split upwards
if (pageId == _rootPageId)
if (pageId == RootPageId)
{
// New Root
uint newRootId = _storage.AllocatePage();
@@ -334,7 +346,7 @@ internal class RTreeIndex : IDisposable
else
_storage.WritePageImmediate(newRootId, buffer);
_rootPageId = newRootId;
RootPageId = newRootId;
// Update parent pointers
UpdateParentPointer(pageId, newRootId, transaction);
@@ -356,7 +368,7 @@ internal class RTreeIndex : IDisposable
private void UpdateParentPointer(uint pageId, uint parentId, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -366,17 +378,20 @@ internal class RTreeIndex : IDisposable
else
_storage.WritePageImmediate(pageId, buffer);
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void PickSeeds(List<(GeoBox Mbr, DocumentLocation Pointer)> entries, out (GeoBox Mbr, DocumentLocation Pointer) s1, out (GeoBox Mbr, DocumentLocation Pointer) s2)
private void PickSeeds(List<(GeoBox Mbr, DocumentLocation Pointer)> entries,
out (GeoBox Mbr, DocumentLocation Pointer) s1, out (GeoBox Mbr, DocumentLocation Pointer) s2)
{
double maxWaste = double.MinValue;
var maxWaste = double.MinValue;
s1 = entries[0];
s2 = entries[1];
for (int i = 0; i < entries.Count; i++)
{
for (var i = 0; i < entries.Count; i++)
for (int j = i + 1; j < entries.Count; j++)
{
var combined = entries[i].Mbr.ExpandTo(entries[j].Mbr);
@@ -389,7 +404,6 @@ internal class RTreeIndex : IDisposable
}
}
}
}
private byte[] RentPageBuffer()
{
@@ -400,11 +414,4 @@ internal class RTreeIndex : IDisposable
{
ArrayPool<byte>.Shared.Return(buffer);
}
/// <summary>
/// Releases resources used by the index.
/// </summary>
public void Dispose()
{
}
}

26
src/CBDD.Core/Indexing/SpatialMath.cs
View File

@@ -13,7 +13,10 @@ public static class SpatialMath
/// <param name="p1">The first point.</param>
/// <param name="p2">The second point.</param>
/// <returns>The distance in kilometers.</returns>
internal static double DistanceKm(GeoPoint p1, GeoPoint p2) => DistanceKm(p1.Latitude, p1.Longitude, p2.Latitude, p2.Longitude);
internal static double DistanceKm(GeoPoint p1, GeoPoint p2)
{
return DistanceKm(p1.Latitude, p1.Longitude, p2.Latitude, p2.Longitude);
}
/// <summary>
/// Calculates distance between two coordinates on Earth using Haversine formula.
@@ -42,7 +45,10 @@ public static class SpatialMath
/// <param name="center">The center point.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>The bounding box.</returns>
internal static GeoBox BoundingBox(GeoPoint center, double radiusKm) => BoundingBox(center.Latitude, center.Longitude, radiusKm);
internal static GeoBox BoundingBox(GeoPoint center, double radiusKm)
{
return BoundingBox(center.Latitude, center.Longitude, radiusKm);
}
/// <summary>
/// Creates a bounding box from a coordinate and radius.
@@ -51,7 +57,10 @@ public static class SpatialMath
/// <param name="lon">The center longitude.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>The bounding box.</returns>
internal static GeoBox InternalBoundingBox(double lat, double lon, double radiusKm) => BoundingBox(lat, lon, radiusKm);
internal static GeoBox InternalBoundingBox(double lat, double lon, double radiusKm)
{
return BoundingBox(lat, lon, radiusKm);
}
/// <summary>
/// Creates a bounding box centered at a coordinate with a given radius in kilometers.
@@ -72,6 +81,13 @@ public static class SpatialMath
lon + dLon);
}
private static double ToRadians(double degrees) => degrees * Math.PI / 180.0;
private static double ToDegrees(double radians) => radians * 180.0 / Math.PI;
private static double ToRadians(double degrees)
{
return degrees * Math.PI / 180.0;
}
private static double ToDegrees(double radians)
{
return radians * 180.0 / Math.PI;
}
}

20
src/CBDD.Core/Indexing/VectorMath.cs
View File

@@ -1,7 +1,5 @@
using System.Runtime.Intrinsics;
using System.Runtime.Intrinsics.X86;
using System.Runtime.InteropServices;
using System.Numerics;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -56,7 +54,7 @@ public static class VectorMath
throw new ArgumentException("Vectors must have same length");
float dot = 0;
int i = 0;
var i = 0;
// SIMD Optimization for .NET
if (Vector.IsHardwareAccelerated && v1.Length >= Vector<float>.Count)
@@ -65,20 +63,14 @@ public static class VectorMath
var v1Span = MemoryMarshal.Cast<float, Vector<float>>(v1);
var v2Span = MemoryMarshal.Cast<float, Vector<float>>(v2);
foreach (var chunk in Enumerable.Range(0, v1Span.Length))
{
vDot += v1Span[chunk] * v2Span[chunk];
}
foreach (int chunk in Enumerable.Range(0, v1Span.Length)) vDot += v1Span[chunk] * v2Span[chunk];
dot = Vector.Dot(vDot, Vector<float>.One);
i = v1Span.Length * Vector<float>.Count;
}
// Remaining elements
for (; i < v1.Length; i++)
{
dot += v1[i] * v2[i];
}
for (; i < v1.Length; i++) dot += v1[i] * v2[i];
return dot;
}
@@ -95,7 +87,7 @@ public static class VectorMath
throw new ArgumentException("Vectors must have same length");
float dist = 0;
int i = 0;
var i = 0;
if (Vector.IsHardwareAccelerated && v1.Length >= Vector<float>.Count)
{
@@ -103,7 +95,7 @@ public static class VectorMath
var v1Span = MemoryMarshal.Cast<float, Vector<float>>(v1);
var v2Span = MemoryMarshal.Cast<float, Vector<float>>(v2);
foreach (var chunk in Enumerable.Range(0, v1Span.Length))
foreach (int chunk in Enumerable.Range(0, v1Span.Length))
{
var diff = v1Span[chunk] - v2Span[chunk];
vDist += diff * diff;

10
src/CBDD.Core/Indexing/VectorSearchExtensions.cs
View File

@@ -9,7 +9,10 @@ public static class VectorSearchExtensions
/// <param name="vector">The vector property of the entity.</param>
/// <param name="query">The query vector to compare against.</param>
/// <param name="k">Number of nearest neighbors to return.</param>
/// <returns>True if the document is part of the top-k results (always returns true when evaluated in memory for compilation purposes).</returns>
/// <returns>
/// True if the document is part of the top-k results (always returns true when evaluated in memory for
/// compilation purposes).
/// </returns>
public static bool VectorSearch(this float[] vector, float[] query, int k)
{
return true;
@@ -22,7 +25,10 @@ public static class VectorSearchExtensions
/// <param name="vectors">The vector collection of the entity.</param>
/// <param name="query">The query vector to compare against.</param>
/// <param name="k">Number of nearest neighbors to return.</param>
/// <returns>True if the document is part of the top-k results (always returns true when evaluated in memory for compilation purposes).</returns>
/// <returns>
/// True if the document is part of the top-k results (always returns true when evaluated in memory for
/// compilation purposes).
/// </returns>
public static bool VectorSearch(this IEnumerable<float[]> vectors, float[] query, int k)
{
return true;

155
src/CBDD.Core/Indexing/VectorSearchIndex.cs
View File

@@ -1,6 +1,6 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
@@ -10,17 +10,10 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// </summary>
public sealed class VectorSearchIndex
{
private struct NodeReference
{
public uint PageId;
public int NodeIndex;
public int MaxLevel;
}
private readonly IndexOptions _options;
private readonly Random _random = new(42);
private readonly IIndexStorage _storage;
private readonly IndexOptions _options;
private uint _rootPageId;
private readonly Random _random = new(42);
/// <summary>
/// Initializes a new vector search index.
@@ -43,13 +36,13 @@ public sealed class VectorSearchIndex
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
_rootPageId = rootPageId;
RootPageId = rootPageId;
}
/// <summary>
/// Gets the root page identifier of the index.
/// </summary>
public uint RootPageId => _rootPageId;
public uint RootPageId { get; private set; }
/// <summary>
/// Inserts a vector and its document location into the index.
@@ -60,28 +53,33 @@ public sealed class VectorSearchIndex
public void Insert(float[] vector, DocumentLocation docLocation, ITransaction? transaction = null)
{
if (vector.Length != _options.Dimensions)
throw new ArgumentException($"Vector dimension mismatch. Expected {_options.Dimensions}, got {vector.Length}");
throw new ArgumentException(
$"Vector dimension mismatch. Expected {_options.Dimensions}, got {vector.Length}");
// 1. Determine level for new node
int targetLevel = GetRandomLevel();
// 2. If index is empty, create first page and first node
if (_rootPageId == 0)
if (RootPageId == 0)
{
_rootPageId = CreateNewPage(transaction);
var pageBuffer = RentPageBuffer();
RootPageId = CreateNewPage(transaction);
byte[] pageBuffer = RentPageBuffer();
try
{
_storage.ReadPage(_rootPageId, transaction?.TransactionId, pageBuffer);
_storage.ReadPage(RootPageId, transaction?.TransactionId, pageBuffer);
VectorPage.WriteNode(pageBuffer, 0, docLocation, targetLevel, vector, _options.Dimensions);
VectorPage.IncrementNodeCount(pageBuffer); // Helper needs to be added or handled
if (transaction != null)
_storage.WritePage(_rootPageId, transaction.TransactionId, pageBuffer);
_storage.WritePage(RootPageId, transaction.TransactionId, pageBuffer);
else
_storage.WritePageImmediate(_rootPageId, pageBuffer);
_storage.WritePageImmediate(RootPageId, pageBuffer);
}
finally { ReturnPageBuffer(pageBuffer); }
finally
{
ReturnPageBuffer(pageBuffer);
}
return;
}
@@ -92,9 +90,7 @@ public sealed class VectorSearchIndex
// 4. Greedy search down to targetLevel+1
for (int l = entryPoint.MaxLevel; l > targetLevel; l--)
{
currentPoint = GreedySearch(currentPoint, vector, l, transaction);
}
// 5. Create the new node
var newNode = AllocateNode(vector, docLocation, targetLevel, transaction);
@@ -105,23 +101,18 @@ public sealed class VectorSearchIndex
var neighbors = SearchLayer(currentPoint, vector, _options.EfConstruction, l, transaction);
var selectedNeighbors = SelectNeighbors(neighbors, vector, _options.M, l, transaction);
foreach (var neighbor in selectedNeighbors)
{
AddBidirectionalLink(newNode, neighbor, l, transaction);
}
foreach (var neighbor in selectedNeighbors) AddBidirectionalLink(newNode, neighbor, l, transaction);
// Move currentPoint down for next level if available
currentPoint = GreedySearch(currentPoint, vector, l, transaction);
}
// 7. Update entry point if new node is higher
if (targetLevel > entryPoint.MaxLevel)
{
UpdateEntryPoint(newNode, transaction);
}
if (targetLevel > entryPoint.MaxLevel) UpdateEntryPoint(newNode, transaction);
}
private IEnumerable<NodeReference> SelectNeighbors(IEnumerable<NodeReference> candidates, float[] query, int m, int level, ITransaction? transaction)
private IEnumerable<NodeReference> SelectNeighbors(IEnumerable<NodeReference> candidates, float[] query, int m,
int level, ITransaction? transaction)
{
// Simple heuristic: just take top M nearest.
// HNSW Paper suggests more complex heuristic to maintain connectivity diversity.
@@ -136,14 +127,14 @@ public sealed class VectorSearchIndex
private void Link(NodeReference from, NodeReference to, int level, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(from.PageId, transaction?.TransactionId, buffer);
var links = VectorPage.GetLinksSpan(buffer, from.NodeIndex, level, _options.Dimensions, _options.M);
// Find first empty slot (PageId == 0)
for (int i = 0; i < links.Length; i += 6)
for (var i = 0; i < links.Length; i += 6)
{
var existing = DocumentLocation.ReadFrom(links.Slice(i, 6));
if (existing.PageId == 0)
@@ -160,7 +151,10 @@ public sealed class VectorSearchIndex
// If full, we should technically prune or redistribute links as per HNSW paper.
// For now, we assume M is large enough or we skip (limited connectivity).
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private NodeReference AllocateNode(float[] vector, DocumentLocation docLoc, int level, ITransaction? transaction)
@@ -168,10 +162,10 @@ public sealed class VectorSearchIndex
// Find a page with space or create new
// For simplicity, we search for a page with available slots or append to a new one.
// Implementation omitted for brevity but required for full persistence.
uint pageId = _rootPageId; // Placeholder: need allocation strategy
int index = 0;
uint pageId = RootPageId; // Placeholder: need allocation strategy
var index = 0;
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -184,7 +178,10 @@ public sealed class VectorSearchIndex
else
_storage.WritePageImmediate(pageId, buffer);
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
return new NodeReference { PageId = pageId, NodeIndex = index, MaxLevel = level };
}
@@ -197,7 +194,7 @@ public sealed class VectorSearchIndex
private NodeReference GreedySearch(NodeReference entryPoint, float[] query, int level, ITransaction? transaction)
{
bool changed = true;
var changed = true;
var current = entryPoint;
float currentDist = VectorMath.Distance(query, LoadVector(current, transaction), _options.Metric);
@@ -215,10 +212,12 @@ public sealed class VectorSearchIndex
}
}
}
return current;
}
private IEnumerable<NodeReference> SearchLayer(NodeReference entryPoint, float[] query, int ef, int level, ITransaction? transaction)
private IEnumerable<NodeReference> SearchLayer(NodeReference entryPoint, float[] query, int ef, int level,
ITransaction? transaction)
{
var visited = new HashSet<NodeReference>();
var candidates = new PriorityQueue<NodeReference, float>();
@@ -233,14 +232,13 @@ public sealed class VectorSearchIndex
{
float d_c = 0;
candidates.TryPeek(out var c, out d_c);
result.TryPeek(out var f, out var d_f);
result.TryPeek(out var f, out float d_f);
if (d_c > -d_f) break;
candidates.Dequeue();
foreach (var e in GetNeighbors(c, level, transaction))
{
if (!visited.Contains(e))
{
visited.Add(e);
@@ -255,7 +253,6 @@ public sealed class VectorSearchIndex
}
}
}
}
// Convert result to list (ordered by distance)
var list = new List<NodeReference>();
@@ -268,20 +265,23 @@ public sealed class VectorSearchIndex
{
// For now, assume a fixed location or track it in page 0 of index
// TODO: Real implementation
return new NodeReference { PageId = _rootPageId, NodeIndex = 0, MaxLevel = 0 };
return new NodeReference { PageId = RootPageId, NodeIndex = 0, MaxLevel = 0 };
}
private float[] LoadVector(NodeReference node, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
float[] vector = new float[_options.Dimensions];
var vector = new float[_options.Dimensions];
VectorPage.ReadNodeData(buffer, node.NodeIndex, out _, out _, vector);
return vector;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
/// <summary>
@@ -292,24 +292,22 @@ public sealed class VectorSearchIndex
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction context.</param>
/// <returns>The nearest vector search results.</returns>
public IEnumerable<VectorSearchResult> Search(float[] query, int k, int efSearch = 100, ITransaction? transaction = null)
public IEnumerable<VectorSearchResult> Search(float[] query, int k, int efSearch = 100,
ITransaction? transaction = null)
{
if (_rootPageId == 0) yield break;
if (RootPageId == 0) yield break;
var entryPoint = GetEntryPoint();
var currentPoint = entryPoint;
// 1. Greedy search through higher layers to find entry point for level 0
for (int l = entryPoint.MaxLevel; l > 0; l--)
{
currentPoint = GreedySearch(currentPoint, query, l, transaction);
}
for (int l = entryPoint.MaxLevel; l > 0; l--) currentPoint = GreedySearch(currentPoint, query, l, transaction);
// 2. Comprehensive search on level 0
var nearest = SearchLayer(currentPoint, query, Math.Max(efSearch, k), 0, transaction);
// 3. Return top-k results
int count = 0;
var count = 0;
foreach (var node in nearest)
{
if (count++ >= k) break;
@@ -322,26 +320,29 @@ public sealed class VectorSearchIndex
private DocumentLocation LoadDocumentLocation(NodeReference node, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
VectorPage.ReadNodeData(buffer, node.NodeIndex, out var loc, out _, new float[0]); // Vector not needed here
return loc;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private IEnumerable<NodeReference> GetNeighbors(NodeReference node, int level, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
var results = new List<NodeReference>();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
var links = VectorPage.GetLinksSpan(buffer, node.NodeIndex, level, _options.Dimensions, _options.M);
for (int i = 0; i < links.Length; i += 6)
for (var i = 0; i < links.Length; i += 6)
{
var loc = DocumentLocation.ReadFrom(links.Slice(i, 6));
if (loc.PageId == 0) break; // End of links
@@ -349,7 +350,11 @@ public sealed class VectorSearchIndex
results.Add(new NodeReference { PageId = loc.PageId, NodeIndex = loc.SlotIndex });
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
return results;
}
@@ -357,29 +362,43 @@ public sealed class VectorSearchIndex
{
// Probability p = 1/M for each level
double p = 1.0 / _options.M;
int level = 0;
while (_random.NextDouble() < p && level < 15)
{
level++;
}
var level = 0;
while (_random.NextDouble() < p && level < 15) level++;
return level;
}
private uint CreateNewPage(ITransaction? transaction)
{
uint pageId = _storage.AllocatePage();
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
VectorPage.Initialize(buffer, pageId, _options.Dimensions, _options.M);
_storage.WritePageImmediate(pageId, buffer);
return pageId;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private byte[] RentPageBuffer() => System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
private void ReturnPageBuffer(byte[] buffer) => System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
private byte[] RentPageBuffer()
{
return ArrayPool<byte>.Shared.Rent(_storage.PageSize);
}
private void ReturnPageBuffer(byte[] buffer)
{
ArrayPool<byte>.Shared.Return(buffer);
}
private struct NodeReference
{
public uint PageId;
public int NodeIndex;
public int MaxLevel;
}
}
public record struct VectorSearchResult(DocumentLocation Location, float Distance);

66
src/CBDD.Core/Metadata/EntityTypeBuilder.cs
View File

@@ -1,5 +1,5 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Metadata;
@@ -54,7 +54,8 @@ public class EntityTypeBuilder<T> where T : class
/// <param name="name">The optional index name.</param>
/// <param name="unique">A value indicating whether the index is unique.</param>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasIndex<TKey>(Expression<Func<T, TKey>> keySelector, string? name = null, bool unique = false)
public EntityTypeBuilder<T> HasIndex<TKey>(Expression<Func<T, TKey>> keySelector, string? name = null,
bool unique = false)
{
Indexes.Add(new IndexBuilder<T>(keySelector, name, unique));
return this;
@@ -69,7 +70,8 @@ public class EntityTypeBuilder<T> where T : class
/// <param name="metric">The vector similarity metric.</param>
/// <param name="name">The optional index name.</param>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
public EntityTypeBuilder<T> HasVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions,
VectorMetric metric = VectorMetric.Cosine, string? name = null)
{
Indexes.Add(new IndexBuilder<T>(keySelector, name, false, IndexType.Vector, dimensions, metric));
return this;
@@ -108,10 +110,7 @@ public class EntityTypeBuilder<T> where T : class
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasConversion<TConverter>()
{
if (!string.IsNullOrEmpty(PrimaryKeyName))
{
PropertyConverters[PrimaryKeyName] = typeof(TConverter);
}
if (!string.IsNullOrEmpty(PrimaryKeyName)) PropertyConverters[PrimaryKeyName] = typeof(TConverter);
return this;
}
@@ -123,7 +122,7 @@ public class EntityTypeBuilder<T> where T : class
/// <returns>A builder for the selected property.</returns>
public PropertyBuilder Property<TProperty>(Expression<Func<T, TProperty>> propertyExpression)
{
var propertyName = ExpressionAnalyzer.ExtractPropertyPaths(propertyExpression).FirstOrDefault();
string? propertyName = ExpressionAnalyzer.ExtractPropertyPaths(propertyExpression).FirstOrDefault();
return new PropertyBuilder(this, propertyName);
}
@@ -133,7 +132,7 @@ public class EntityTypeBuilder<T> where T : class
private readonly string? _propertyName;
/// <summary>
/// Initializes a new instance of the <see cref="PropertyBuilder"/> class.
/// Initializes a new instance of the <see cref="PropertyBuilder" /> class.
/// </summary>
/// <param name="parent">The parent entity type builder.</param>
/// <param name="propertyName">The property name.</param>
@@ -149,10 +148,7 @@ public class EntityTypeBuilder<T> where T : class
/// <returns>The current property builder.</returns>
public PropertyBuilder ValueGeneratedOnAdd()
{
if (_propertyName == _parent.PrimaryKeyName)
{
_parent.ValueGeneratedOnAdd = true;
}
if (_propertyName == _parent.PrimaryKeyName) _parent.ValueGeneratedOnAdd = true;
return this;
}
@@ -163,10 +159,7 @@ public class EntityTypeBuilder<T> where T : class
/// <returns>The current property builder.</returns>
public PropertyBuilder HasConversion<TConverter>()
{
if (!string.IsNullOrEmpty(_propertyName))
{
_parent.PropertyConverters[_propertyName] = typeof(TConverter);
}
if (!string.IsNullOrEmpty(_propertyName)) _parent.PropertyConverters[_propertyName] = typeof(TConverter);
return this;
}
}
@@ -174,6 +167,26 @@ public class EntityTypeBuilder<T> where T : class
public class IndexBuilder<T>
{
/// <summary>
/// Initializes a new instance of the <see cref="IndexBuilder{T}" /> class.
/// </summary>
/// <param name="keySelector">The index key selector expression.</param>
/// <param name="name">The optional index name.</param>
/// <param name="unique">A value indicating whether the index is unique.</param>
/// <param name="type">The index type.</param>
/// <param name="dimensions">The vector dimensions.</param>
/// <param name="metric">The vector metric.</param>
public IndexBuilder(LambdaExpression keySelector, string? name, bool unique, IndexType type = IndexType.BTree,
int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
{
KeySelector = keySelector;
Name = name;
IsUnique = unique;
Type = type;
Dimensions = dimensions;
Metric = metric;
}
/// <summary>
/// Gets the index key selector expression.
/// </summary>
@@ -203,23 +216,4 @@ public class IndexBuilder<T>
/// Gets the vector metric.
/// </summary>
public VectorMetric Metric { get; }
/// <summary>
/// Initializes a new instance of the <see cref="IndexBuilder{T}"/> class.
/// </summary>
/// <param name="keySelector">The index key selector expression.</param>
/// <param name="name">The optional index name.</param>
/// <param name="unique">A value indicating whether the index is unique.</param>
/// <param name="type">The index type.</param>
/// <param name="dimensions">The vector dimensions.</param>
/// <param name="metric">The vector metric.</param>
public IndexBuilder(LambdaExpression keySelector, string? name, bool unique, IndexType type = IndexType.BTree, int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
{
KeySelector = keySelector;
Name = name;
IsUnique = unique;
Type = type;
Dimensions = dimensions;
Metric = metric;
}
}

13
src/CBDD.Core/Metadata/ModelBuilder.cs
View File

@@ -1,6 +1,3 @@
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Metadata;
public class ModelBuilder
@@ -11,14 +8,15 @@ public class ModelBuilder
/// Gets or creates the entity builder for the specified entity type.
/// </summary>
/// <typeparam name="T">The entity type.</typeparam>
/// <returns>The entity builder for <typeparamref name="T"/>.</returns>
/// <returns>The entity builder for <typeparamref name="T" />.</returns>
public EntityTypeBuilder<T> Entity<T>() where T : class
{
if (!_entityBuilders.TryGetValue(typeof(T), out var builder))
if (!_entityBuilders.TryGetValue(typeof(T), out object? builder))
{
builder = new EntityTypeBuilder<T>();
_entityBuilders[typeof(T)] = builder;
}
return (EntityTypeBuilder<T>)builder;
}
@@ -26,5 +24,8 @@ public class ModelBuilder
/// Gets all registered entity builders.
/// </summary>
/// <returns>A read-only dictionary of entity builders keyed by entity type.</returns>
public IReadOnlyDictionary<Type, object> GetEntityBuilders() => _entityBuilders;
public IReadOnlyDictionary<Type, object> GetEntityBuilders()
{
return _entityBuilders;
}
}

7
src/CBDD.Core/Query/BTreeExpressionVisitor.cs
View File

@@ -9,13 +9,15 @@ internal class BTreeExpressionVisitor : ExpressionVisitor
/// <summary>
/// Gets the query model built while visiting an expression tree.
/// </summary>
public QueryModel GetModel() => _model;
public QueryModel GetModel()
{
return _model;
}
/// <inheritdoc />
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Queryable))
{
switch (node.Method.Name)
{
case "Where":
@@ -35,7 +37,6 @@ internal class BTreeExpressionVisitor : ExpressionVisitor
VisitSkip(node);
break;
}
}
return base.VisitMethodCall(node);
}

40
src/CBDD.Core/Query/BTreeQueryProvider.cs
View File

@@ -1,6 +1,6 @@
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using static ZB.MOM.WW.CBDD.Core.Query.IndexOptimizer;
@@ -11,7 +11,7 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
private readonly DocumentCollection<TId, T> _collection;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeQueryProvider{TId, T}"/> class.
/// Initializes a new instance of the <see cref="BTreeQueryProvider{TId, T}" /> class.
/// </summary>
/// <param name="collection">The backing document collection.</param>
public BTreeQueryProvider(DocumentCollection<TId, T> collection)
@@ -23,15 +23,14 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
/// Creates a query from the specified expression.
/// </summary>
/// <param name="expression">The query expression.</param>
/// <returns>An <see cref="IQueryable"/> representing the query.</returns>
/// <returns>An <see cref="IQueryable" /> representing the query.</returns>
public IQueryable CreateQuery(Expression expression)
{
var elementType = expression.Type.GetGenericArguments()[0];
try
{
return (IQueryable)Activator.CreateInstance(
typeof(BTreeQueryable<>).MakeGenericType(elementType),
new object[] { this, expression })!;
typeof(BTreeQueryable<>).MakeGenericType(elementType), this, expression)!;
}
catch (TargetInvocationException ex)
{
@@ -44,7 +43,7 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
/// </summary>
/// <typeparam name="TElement">The element type of the query.</typeparam>
/// <param name="expression">The query expression.</param>
/// <returns>An <see cref="IQueryable{T}"/> representing the query.</returns>
/// <returns>An <see cref="IQueryable{T}" /> representing the query.</returns>
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
return new BTreeQueryable<TElement>(this, expression);
@@ -80,45 +79,30 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
IEnumerable<T> sourceData = null!;
// A. Try Index Optimization (Only if Where clause exists)
var indexOpt = IndexOptimizer.TryOptimize<T>(model, _collection.GetIndexes());
var indexOpt = TryOptimize<T>(model, _collection.GetIndexes());
if (indexOpt != null)
{
if (indexOpt.IsVectorSearch)
{
sourceData = _collection.VectorSearch(indexOpt.IndexName, indexOpt.VectorQuery!, indexOpt.K);
}
else if (indexOpt.IsSpatialSearch)
{
sourceData = indexOpt.SpatialType == SpatialQueryType.Near
? _collection.Near(indexOpt.IndexName, indexOpt.SpatialPoint, indexOpt.RadiusKm)
: _collection.Within(indexOpt.IndexName, indexOpt.SpatialMin, indexOpt.SpatialMax);
}
else
{
sourceData = _collection.QueryIndex(indexOpt.IndexName, indexOpt.MinValue, indexOpt.MaxValue);
}
}
// B. Try Scan Optimization (if no index used)
if (sourceData == null)
{
Func<ZB.MOM.WW.CBDD.Bson.BsonSpanReader, bool>? bsonPredicate = null;
if (model.WhereClause != null)
{
bsonPredicate = BsonExpressionEvaluator.TryCompile<T>(model.WhereClause);
}
Func<BsonSpanReader, bool>? bsonPredicate = null;
if (model.WhereClause != null) bsonPredicate = BsonExpressionEvaluator.TryCompile<T>(model.WhereClause);
if (bsonPredicate != null)
{
sourceData = _collection.Scan(bsonPredicate);
}
if (bsonPredicate != null) sourceData = _collection.Scan(bsonPredicate);
}
// C. Fallback to Full Scan
if (sourceData == null)
{
sourceData = _collection.FindAll();
}
if (sourceData == null) sourceData = _collection.FindAll();
// 3. Rewrite Expression Tree to use Enumerable
// Replace the "Root" IQueryable with our sourceData IEnumerable
@@ -140,10 +124,8 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
// We need to turn it into a Func<TResult> and invoke it.
if (rewrittenExpression.Type != typeof(TResult))
{
// If TResult is object (non-generic Execute), we need to cast
rewrittenExpression = Expression.Convert(rewrittenExpression, typeof(TResult));
}
var lambda = Expression.Lambda<Func<TResult>>(rewrittenExpression);
var compiled = lambda.Compile();
@@ -162,11 +144,9 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
{
// If we found a Queryable, that's our root source
if (Root == null && node.Value is IQueryable q)
{
// We typically want the "base" queryable (the BTreeQueryable instance)
// In a chain like Coll.Where.Select, the root is Coll.
Root = q;
}
return base.VisitConstant(node);
}
}

1
src/CBDD.Core/Query/BTreeQueryable.cs
View File

@@ -1,5 +1,4 @@
using System.Collections;
using System.Linq;
using System.Linq.Expressions;
namespace ZB.MOM.WW.CBDD.Core.Query;

29
src/CBDD.Core/Query/BsonExpressionEvaluator.cs
View File

@@ -10,7 +10,7 @@ internal static class BsonExpressionEvaluator
/// </summary>
/// <typeparam name="T">The entity type of the original expression.</typeparam>
/// <param name="expression">The lambda expression to compile.</param>
/// <returns>A compiled predicate when supported; otherwise, <see langword="null"/>.</returns>
/// <returns>A compiled predicate when supported; otherwise, <see langword="null" />.</returns>
public static Func<BsonSpanReader, bool>? TryCompile<T>(LambdaExpression expression)
{
// Simple optimization for: x => x.Prop op Constant
@@ -29,12 +29,11 @@ internal static class BsonExpressionEvaluator
}
if (left is MemberExpression member && right is ConstantExpression constant)
{
// Check if member is property of parameter
if (member.Expression == expression.Parameters[0])
{
var propertyName = member.Member.Name.ToLowerInvariant();
var value = constant.Value;
string propertyName = member.Member.Name.ToLowerInvariant();
object? value = constant.Value;
// Handle Id mapping?
// If property is "id", Bson field is "_id"
@@ -43,12 +42,13 @@ internal static class BsonExpressionEvaluator
return CreatePredicate(propertyName, value, nodeType);
}
}
}
return null;
}
private static ExpressionType Flip(ExpressionType type) => type switch
private static ExpressionType Flip(ExpressionType type)
{
return type switch
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
@@ -56,8 +56,10 @@ internal static class BsonExpressionEvaluator
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}
private static Func<BsonSpanReader, bool>? CreatePredicate(string propertyName, object? targetValue, ExpressionType op)
private static Func<BsonSpanReader, bool>? CreatePredicate(string propertyName, object? targetValue,
ExpressionType op)
{
// We need to return a delegate that searches for propertyName in BsonSpanReader and compares
@@ -71,13 +73,11 @@ internal static class BsonExpressionEvaluator
var type = reader.ReadBsonType();
if (type == 0) break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == propertyName)
{
// Found -> read value and compare
return Compare(ref reader, type, targetValue, op);
}
reader.SkipValue(type);
}
@@ -86,6 +86,7 @@ internal static class BsonExpressionEvaluator
{
return false;
}
return false; // Not found
};
}
@@ -97,9 +98,8 @@ internal static class BsonExpressionEvaluator
if (type == BsonType.Int32)
{
var val = reader.ReadInt32();
int val = reader.ReadInt32();
if (target is int targetInt)
{
return op switch
{
ExpressionType.Equal => val == targetInt,
@@ -111,13 +111,12 @@ internal static class BsonExpressionEvaluator
_ => false
};
}
}
else if (type == BsonType.String)
{
var val = reader.ReadString();
string val = reader.ReadString();
if (target is string targetStr)
{
var cmp = string.Compare(val, targetStr, StringComparison.Ordinal);
int cmp = string.Compare(val, targetStr, StringComparison.Ordinal);
return op switch
{
ExpressionType.Equal => cmp == 0,

15
src/CBDD.Core/Query/EnumerableRewriter.cs
View File

@@ -1,6 +1,3 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
@@ -12,7 +9,7 @@ internal class EnumerableRewriter : ExpressionVisitor
private readonly object _target;
/// <summary>
/// Initializes a new instance of the <see cref="EnumerableRewriter"/> class.
/// Initializes a new instance of the <see cref="EnumerableRewriter" /> class.
/// </summary>
/// <param name="source">The original queryable source to replace.</param>
/// <param name="target">The target enumerable-backed object.</param>
@@ -26,10 +23,7 @@ internal class EnumerableRewriter : ExpressionVisitor
protected override Expression VisitConstant(ConstantExpression node)
{
// Replace the IQueryable source with the materialized IEnumerable
if (node.Value == _source)
{
return Expression.Constant(_target);
}
if (node.Value == _source) return Expression.Constant(_target);
return base.VisitConstant(node);
}
@@ -38,11 +32,11 @@ internal class EnumerableRewriter : ExpressionVisitor
{
if (node.Method.DeclaringType == typeof(Queryable))
{
var methodName = node.Method.Name;
string methodName = node.Method.Name;
var typeArgs = node.Method.GetGenericArguments();
var args = new Expression[node.Arguments.Count];
for (int i = 0; i < node.Arguments.Count; i++)
for (var i = 0; i < node.Arguments.Count; i++)
{
var arg = Visit(node.Arguments[i]);
@@ -52,6 +46,7 @@ internal class EnumerableRewriter : ExpressionVisitor
var lambda = (LambdaExpression)quote.Operand;
arg = Expression.Constant(lambda.Compile());
}
args[i] = arg;
}

442
src/CBDD.Core/Query/IndexOptimizer.cs
View File

@@ -5,6 +5,227 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal static class IndexOptimizer
{
public enum SpatialQueryType
{
Near,
Within
}
/// <summary>
/// Attempts to optimize a query model using available indexes.
/// </summary>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="model">The query model.</param>
/// <param name="indexes">The available collection indexes.</param>
/// <returns>An optimization result when optimization is possible; otherwise, <see langword="null" />.</returns>
public static OptimizationResult? TryOptimize<T>(QueryModel model, IEnumerable<CollectionIndexInfo> indexes)
{
if (model.WhereClause == null) return null;
return OptimizeExpression(model.WhereClause.Body, model.WhereClause.Parameters[0], indexes);
}
private static OptimizationResult? OptimizeExpression(Expression expression, ParameterExpression parameter,
IEnumerable<CollectionIndexInfo> indexes)
{
// ... (Existing AndAlso logic remains the same) ...
if (expression is BinaryExpression binary && binary.NodeType == ExpressionType.AndAlso)
{
var left = OptimizeExpression(binary.Left, parameter, indexes);
var right = OptimizeExpression(binary.Right, parameter, indexes);
if (left != null && right != null && left.IndexName == right.IndexName)
return new OptimizationResult
{
IndexName = left.IndexName,
MinValue = left.MinValue ?? right.MinValue,
MaxValue = left.MaxValue ?? right.MaxValue,
IsRange = true
};
return left ?? right;
}
// Handle Simple Binary Predicates
(string? propertyName, object? value, var op) = ParseSimplePredicate(expression, parameter);
if (propertyName != null)
{
var index = indexes.FirstOrDefault(i => Matches(i, propertyName));
if (index != null)
{
var result = new OptimizationResult { IndexName = index.Name };
switch (op)
{
case ExpressionType.Equal:
result.MinValue = value;
result.MaxValue = value;
result.IsRange = false;
break;
case ExpressionType.GreaterThan:
case ExpressionType.GreaterThanOrEqual:
result.MinValue = value;
result.MaxValue = null;
result.IsRange = true;
break;
case ExpressionType.LessThan:
case ExpressionType.LessThanOrEqual:
result.MinValue = null;
result.MaxValue = value;
result.IsRange = true;
break;
}
return result;
}
}
// Handle StartsWith
if (expression is MethodCallExpression call && call.Method.Name == "StartsWith" &&
call.Object is MemberExpression member)
if (member.Expression == parameter && call.Arguments[0] is ConstantExpression constant &&
constant.Value is string prefix)
{
var index = indexes.FirstOrDefault(i => Matches(i, member.Member.Name));
if (index != null && index.Type == IndexType.BTree)
{
string nextPrefix = IncrementPrefix(prefix);
return new OptimizationResult
{
IndexName = index.Name,
MinValue = prefix,
MaxValue = nextPrefix,
IsRange = true
};
}
}
// Handle Method Calls (VectorSearch, Near, Within)
if (expression is MethodCallExpression mcall)
{
// VectorSearch(this float[] vector, float[] query, int k)
if (mcall.Method.Name == "VectorSearch" && mcall.Arguments[0] is MemberExpression vMember &&
vMember.Expression == parameter)
{
float[] query = EvaluateExpression<float[]>(mcall.Arguments[1]);
var k = EvaluateExpression<int>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Vector && Matches(i, vMember.Member.Name));
if (index != null)
return new OptimizationResult
{
IndexName = index.Name,
IsVectorSearch = true,
VectorQuery = query,
K = k
};
}
// Near(this (double, double) point, (double, double) center, double radiusKm)
if (mcall.Method.Name == "Near" && mcall.Arguments[0] is MemberExpression nMember &&
nMember.Expression == parameter)
{
var center = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var radius = EvaluateExpression<double>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, nMember.Member.Name));
if (index != null)
return new OptimizationResult
{
IndexName = index.Name,
IsSpatialSearch = true,
SpatialType = SpatialQueryType.Near,
SpatialPoint = center,
RadiusKm = radius
};
}
// Within(this (double, double) point, (double, double) min, (double, double) max)
if (mcall.Method.Name == "Within" && mcall.Arguments[0] is MemberExpression wMember &&
wMember.Expression == parameter)
{
var min = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var max = EvaluateExpression<(double, double)>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, wMember.Member.Name));
if (index != null)
return new OptimizationResult
{
IndexName = index.Name,
IsSpatialSearch = true,
SpatialType = SpatialQueryType.Within,
SpatialMin = min,
SpatialMax = max
};
}
}
return null;
}
private static string IncrementPrefix(string prefix)
{
if (string.IsNullOrEmpty(prefix)) return null!;
char lastChar = prefix[prefix.Length - 1];
if (lastChar == char.MaxValue) return prefix; // Cannot increment
return prefix.Substring(0, prefix.Length - 1) + (char)(lastChar + 1);
}
private static T EvaluateExpression<T>(Expression expression)
{
if (expression is ConstantExpression constant) return (T)constant.Value!;
// Evaluate more complex expressions (closures, properties, etc.)
var lambda = Expression.Lambda(expression);
var compiled = lambda.Compile();
return (T)compiled.DynamicInvoke()!;
}
private static bool Matches(CollectionIndexInfo index, string propertyName)
{
if (index.PropertyPaths == null || index.PropertyPaths.Length == 0) return false;
return index.PropertyPaths[0].Equals(propertyName, StringComparison.OrdinalIgnoreCase);
}
private static (string? propertyName, object? value, ExpressionType op) ParseSimplePredicate(Expression expression,
ParameterExpression parameter)
{
if (expression is BinaryExpression binary)
{
var left = binary.Left;
var right = binary.Right;
var nodeType = binary.NodeType;
if (right is MemberExpression && left is ConstantExpression)
{
(left, right) = (right, left);
nodeType = Flip(nodeType);
}
if (left is MemberExpression member && right is ConstantExpression constant)
if (member.Expression == parameter)
return (member.Member.Name, constant.Value, nodeType);
// Handle Convert
if (left is UnaryExpression unary && unary.Operand is MemberExpression member2 &&
right is ConstantExpression constant2)
if (member2.Expression == parameter)
return (member2.Member.Name, constant2.Value, nodeType);
}
return (null, null, ExpressionType.Default);
}
private static ExpressionType Flip(ExpressionType type)
{
return type switch
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}
/// <summary>
/// Represents the selected index and bounds for an optimized query.
/// </summary>
@@ -75,225 +296,4 @@ internal static class IndexOptimizer
/// </summary>
public SpatialQueryType SpatialType { get; set; }
}
public enum SpatialQueryType { Near, Within }
/// <summary>
/// Attempts to optimize a query model using available indexes.
/// </summary>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="model">The query model.</param>
/// <param name="indexes">The available collection indexes.</param>
/// <returns>An optimization result when optimization is possible; otherwise, <see langword="null"/>.</returns>
public static OptimizationResult? TryOptimize<T>(QueryModel model, IEnumerable<CollectionIndexInfo> indexes)
{
if (model.WhereClause == null) return null;
return OptimizeExpression(model.WhereClause.Body, model.WhereClause.Parameters[0], indexes);
}
private static OptimizationResult? OptimizeExpression(Expression expression, ParameterExpression parameter, IEnumerable<CollectionIndexInfo> indexes)
{
// ... (Existing AndAlso logic remains the same) ...
if (expression is BinaryExpression binary && binary.NodeType == ExpressionType.AndAlso)
{
var left = OptimizeExpression(binary.Left, parameter, indexes);
var right = OptimizeExpression(binary.Right, parameter, indexes);
if (left != null && right != null && left.IndexName == right.IndexName)
{
return new OptimizationResult
{
IndexName = left.IndexName,
MinValue = left.MinValue ?? right.MinValue,
MaxValue = left.MaxValue ?? right.MaxValue,
IsRange = true
};
}
return left ?? right;
}
// Handle Simple Binary Predicates
var (propertyName, value, op) = ParseSimplePredicate(expression, parameter);
if (propertyName != null)
{
var index = indexes.FirstOrDefault(i => Matches(i, propertyName));
if (index != null)
{
var result = new OptimizationResult { IndexName = index.Name };
switch (op)
{
case ExpressionType.Equal:
result.MinValue = value;
result.MaxValue = value;
result.IsRange = false;
break;
case ExpressionType.GreaterThan:
case ExpressionType.GreaterThanOrEqual:
result.MinValue = value;
result.MaxValue = null;
result.IsRange = true;
break;
case ExpressionType.LessThan:
case ExpressionType.LessThanOrEqual:
result.MinValue = null;
result.MaxValue = value;
result.IsRange = true;
break;
}
return result;
}
}
// Handle StartsWith
if (expression is MethodCallExpression call && call.Method.Name == "StartsWith" && call.Object is MemberExpression member)
{
if (member.Expression == parameter && call.Arguments[0] is ConstantExpression constant && constant.Value is string prefix)
{
var index = indexes.FirstOrDefault(i => Matches(i, member.Member.Name));
if (index != null && index.Type == IndexType.BTree)
{
var nextPrefix = IncrementPrefix(prefix);
return new OptimizationResult
{
IndexName = index.Name,
MinValue = prefix,
MaxValue = nextPrefix,
IsRange = true
};
}
}
}
// Handle Method Calls (VectorSearch, Near, Within)
if (expression is MethodCallExpression mcall)
{
// VectorSearch(this float[] vector, float[] query, int k)
if (mcall.Method.Name == "VectorSearch" && mcall.Arguments[0] is MemberExpression vMember && vMember.Expression == parameter)
{
var query = EvaluateExpression<float[]>(mcall.Arguments[1]);
var k = EvaluateExpression<int>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Vector && Matches(i, vMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
IsVectorSearch = true,
VectorQuery = query,
K = k
};
}
}
// Near(this (double, double) point, (double, double) center, double radiusKm)
if (mcall.Method.Name == "Near" && mcall.Arguments[0] is MemberExpression nMember && nMember.Expression == parameter)
{
var center = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var radius = EvaluateExpression<double>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, nMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
IsSpatialSearch = true,
SpatialType = SpatialQueryType.Near,
SpatialPoint = center,
RadiusKm = radius
};
}
}
// Within(this (double, double) point, (double, double) min, (double, double) max)
if (mcall.Method.Name == "Within" && mcall.Arguments[0] is MemberExpression wMember && wMember.Expression == parameter)
{
var min = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var max = EvaluateExpression<(double, double)>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, wMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
IsSpatialSearch = true,
SpatialType = SpatialQueryType.Within,
SpatialMin = min,
SpatialMax = max
};
}
}
}
return null;
}
private static string IncrementPrefix(string prefix)
{
if (string.IsNullOrEmpty(prefix)) return null!;
char lastChar = prefix[prefix.Length - 1];
if (lastChar == char.MaxValue) return prefix; // Cannot increment
return prefix.Substring(0, prefix.Length - 1) + (char)(lastChar + 1);
}
private static T EvaluateExpression<T>(Expression expression)
{
if (expression is ConstantExpression constant)
{
return (T)constant.Value!;
}
// Evaluate more complex expressions (closures, properties, etc.)
var lambda = Expression.Lambda(expression);
var compiled = lambda.Compile();
return (T)compiled.DynamicInvoke()!;
}
private static bool Matches(CollectionIndexInfo index, string propertyName)
{
if (index.PropertyPaths == null || index.PropertyPaths.Length == 0) return false;
return index.PropertyPaths[0].Equals(propertyName, StringComparison.OrdinalIgnoreCase);
}
private static (string? propertyName, object? value, ExpressionType op) ParseSimplePredicate(Expression expression, ParameterExpression parameter)
{
if (expression is BinaryExpression binary)
{
var left = binary.Left;
var right = binary.Right;
var nodeType = binary.NodeType;
if (right is MemberExpression && left is ConstantExpression)
{
(left, right) = (right, left);
nodeType = Flip(nodeType);
}
if (left is MemberExpression member && right is ConstantExpression constant)
{
if (member.Expression == parameter)
return (member.Member.Name, constant.Value, nodeType);
}
// Handle Convert
if (left is UnaryExpression unary && unary.Operand is MemberExpression member2 && right is ConstantExpression constant2)
{
if (member2.Expression == parameter)
return (member2.Member.Name, constant2.Value, nodeType);
}
}
return (null, null, ExpressionType.Default);
}
private static ExpressionType Flip(ExpressionType type) => type switch
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}

110
src/CBDD.Core/Storage/DictionaryPage.cs
View File

@@ -1,6 +1,6 @@
using System.Runtime.InteropServices;
using System.Buffers;
using System.Buffers.Binary;
using System.Text;
using ZB.MOM.WW.CBDD.Core;
namespace ZB.MOM.WW.CBDD.Core.Storage;
@@ -49,8 +49,8 @@ public struct DictionaryPage
header.WriteTo(page);
// 2. Initialize Counts
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), 0);
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), (ushort)page.Length);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), 0);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), (ushort)page.Length);
}
/// <summary>
@@ -60,32 +60,29 @@ public struct DictionaryPage
/// <param name="page">The page buffer.</param>
/// <param name="key">The dictionary key.</param>
/// <param name="value">The value mapped to the key.</param>
/// <returns><see langword="true"/> if the entry was inserted; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the entry was inserted; otherwise, <see langword="false" />.</returns>
public static bool Insert(Span<byte> page, string key, ushort value)
{
var keyByteCount = Encoding.UTF8.GetByteCount(key);
int keyByteCount = Encoding.UTF8.GetByteCount(key);
if (keyByteCount > 255) throw new ArgumentException("Key length must be <= 255 bytes");
// Entry Size: KeyLen(1) + Key(N) + Value(2)
var entrySize = 1 + keyByteCount + 2;
var requiredSpace = entrySize + 2; // +2 for Offset entry
int entrySize = 1 + keyByteCount + 2;
int requiredSpace = entrySize + 2; // +2 for Offset entry
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
var freeSpaceEnd = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(FreeSpaceEndOffset));
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort freeSpaceEnd = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(FreeSpaceEndOffset));
var offsetsEnd = OffsetsStart + (count * 2);
var freeSpace = freeSpaceEnd - offsetsEnd;
int offsetsEnd = OffsetsStart + count * 2;
int freeSpace = freeSpaceEnd - offsetsEnd;
if (freeSpace < requiredSpace)
{
return false; // Page Full
}
if (freeSpace < requiredSpace) return false; // Page Full
// 1. Prepare Data
var insertionOffset = (ushort)(freeSpaceEnd - entrySize);
page[insertionOffset] = (byte)keyByteCount; // Write Key Length
Encoding.UTF8.GetBytes(key, page.Slice(insertionOffset + 1, keyByteCount)); // Write Key
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(insertionOffset + 1 + keyByteCount), value); // Write Value
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(insertionOffset + 1 + keyByteCount), value); // Write Value
// 2. Insert Offset into Sorted List
// Find insert Index using spans
@@ -95,21 +92,21 @@ public struct DictionaryPage
// Shift offsets if needed
if (insertIndex < count)
{
var src = page.Slice(OffsetsStart + (insertIndex * 2), (count - insertIndex) * 2);
var dest = page.Slice(OffsetsStart + ((insertIndex + 1) * 2));
var src = page.Slice(OffsetsStart + insertIndex * 2, (count - insertIndex) * 2);
var dest = page.Slice(OffsetsStart + (insertIndex + 1) * 2);
src.CopyTo(dest);
}
// Write new offset
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(OffsetsStart + (insertIndex * 2)), insertionOffset);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(OffsetsStart + insertIndex * 2), insertionOffset);
// 3. Update Metadata
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), (ushort)(count + 1));
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), insertionOffset);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), (ushort)(count + 1));
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), insertionOffset);
// Update FreeBytes in header (approximate)
var pageHeader = PageHeader.ReadFrom(page);
pageHeader.FreeBytes = (ushort)(insertionOffset - (OffsetsStart + ((count + 1) * 2)));
pageHeader.FreeBytes = (ushort)(insertionOffset - (OffsetsStart + (count + 1) * 2));
pageHeader.WriteTo(page);
return true;
@@ -121,31 +118,31 @@ public struct DictionaryPage
/// <param name="page">The page buffer.</param>
/// <param name="keyBytes">The UTF-8 encoded key bytes.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <returns><see langword="true"/> if the key was found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the key was found; otherwise, <see langword="false" />.</returns>
public static bool TryFind(ReadOnlySpan<byte> page, ReadOnlySpan<byte> keyBytes, out ushort value)
{
value = 0;
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
if (count == 0) return false;
// Binary Search
int low = 0;
var low = 0;
int high = count - 1;
while (low <= high)
{
int mid = low + (high - low) / 2;
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (mid * 2)));
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + mid * 2));
// Read Key at Offset
var keyLen = page[offset];
byte keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
int comparison = entryKeySpan.SequenceCompareTo(keyBytes);
if (comparison == 0)
{
value = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
value = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
return true;
}
@@ -166,15 +163,16 @@ public struct DictionaryPage
/// <param name="key">The key to search for.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns><see langword="true"/> if the key was found; otherwise, <see langword="false"/>.</returns>
public static bool TryFindGlobal(StorageEngine storage, uint startPageId, string key, out ushort value, ulong? transactionId = null)
/// <returns><see langword="true" /> if the key was found; otherwise, <see langword="false" />.</returns>
public static bool TryFindGlobal(StorageEngine storage, uint startPageId, string key, out ushort value,
ulong? transactionId = null)
{
var keyByteCount = Encoding.UTF8.GetByteCount(key);
Span<byte> keyBytes = keyByteCount <= 256 ? stackalloc byte[keyByteCount] : new byte[keyByteCount];
int keyByteCount = Encoding.UTF8.GetByteCount(key);
var keyBytes = keyByteCount <= 256 ? stackalloc byte[keyByteCount] : new byte[keyByteCount];
Encoding.UTF8.GetBytes(key, keyBytes);
var pageId = startPageId;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(storage.PageSize);
uint pageId = startPageId;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(storage.PageSize);
try
{
while (pageId != 0)
@@ -183,10 +181,7 @@ public struct DictionaryPage
storage.ReadPage(pageId, transactionId, pageBuffer);
// TryFind in this page
if (TryFind(pageBuffer, keyBytes, out value))
{
return true;
}
if (TryFind(pageBuffer, keyBytes, out value)) return true;
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
@@ -195,7 +190,7 @@ public struct DictionaryPage
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
value = 0;
@@ -204,15 +199,15 @@ public struct DictionaryPage
private static int FindInsertIndex(ReadOnlySpan<byte> page, int count, ReadOnlySpan<byte> keyBytes)
{
int low = 0;
var low = 0;
int high = count - 1;
while (low <= high)
{
int mid = low + (high - low) / 2;
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (mid * 2)));
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + mid * 2));
var keyLen = page[offset];
byte keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
int comparison = entryKeySpan.SequenceCompareTo(keyBytes);
@@ -223,6 +218,7 @@ public struct DictionaryPage
else
high = mid - 1;
}
return low;
}
@@ -233,18 +229,20 @@ public struct DictionaryPage
/// <returns>All key-value pairs in the page.</returns>
public static IEnumerable<(string Key, ushort Value)> GetAll(ReadOnlySpan<byte> page)
{
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
var list = new List<(string Key, ushort Value)>();
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (i * 2)));
var keyLen = page[offset];
var keyStr = Encoding.UTF8.GetString(page.Slice(offset + 1, keyLen));
var val = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + i * 2));
byte keyLen = page[offset];
string keyStr = Encoding.UTF8.GetString(page.Slice(offset + 1, keyLen));
ushort val = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
list.Add((keyStr, val));
}
return list;
}
/// <summary>
/// Retrieves all key-value pairs across a chain of DictionaryPages.
/// Used for rebuilding the in-memory cache.
@@ -253,10 +251,11 @@ public struct DictionaryPage
/// <param name="startPageId">The first page in the dictionary chain.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns>All key-value pairs across the page chain.</returns>
public static IEnumerable<(string Key, ushort Value)> FindAllGlobal(StorageEngine storage, uint startPageId, ulong? transactionId = null)
public static IEnumerable<(string Key, ushort Value)> FindAllGlobal(StorageEngine storage, uint startPageId,
ulong? transactionId = null)
{
var pageId = startPageId;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(storage.PageSize);
uint pageId = startPageId;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(storage.PageSize);
try
{
while (pageId != 0)
@@ -265,10 +264,7 @@ public struct DictionaryPage
storage.ReadPage(pageId, transactionId, pageBuffer);
// Get all entries in this page
foreach (var entry in GetAll(pageBuffer))
{
yield return entry;
}
foreach (var entry in GetAll(pageBuffer)) yield return entry;
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
@@ -277,7 +273,7 @@ public struct DictionaryPage
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
}

5
src/CBDD.Core/Storage/IIndexStorage.cs
View File

@@ -9,15 +9,18 @@ internal interface IIndexStorage
/// Gets or sets the PageSize.
/// </summary>
int PageSize { get; }
/// <summary>
/// Executes AllocatePage.
/// </summary>
uint AllocatePage();
/// <summary>
/// Executes FreePage.
/// </summary>
/// <param name="pageId">The page identifier.</param>
void FreePage(uint pageId);
/// <summary>
/// Executes ReadPage.
/// </summary>
@@ -25,6 +28,7 @@ internal interface IIndexStorage
/// <param name="transactionId">The optional transaction identifier.</param>
/// <param name="destination">The destination buffer.</param>
void ReadPage(uint pageId, ulong? transactionId, Span<byte> destination);
/// <summary>
/// Executes WritePage.
/// </summary>
@@ -32,6 +36,7 @@ internal interface IIndexStorage
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="data">The source page data.</param>
void WritePage(uint pageId, ulong transactionId, ReadOnlySpan<byte> data);
/// <summary>
/// Executes WritePageImmediate.
/// </summary>

3
src/CBDD.Core/Storage/IStorageEngine.cs
View File

@@ -61,7 +61,8 @@ internal interface IStorageEngine : IIndexStorage, IDisposable
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">A cancellation token.</param>
Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted, CancellationToken ct = default);
Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted,
CancellationToken ct = default);
/// <summary>
/// Gets collection metadata by name.

284
src/CBDD.Core/Storage/PageFile.cs
View File

@@ -59,37 +59,34 @@ public readonly struct PageFileConfig
/// </summary>
public sealed class PageFile : IDisposable
{
private readonly string _filePath;
private readonly PageFileConfig _config;
private FileStream? _fileStream;
private MemoryMappedFile? _mappedFile;
private readonly object _lock = new();
private bool _disposed;
private bool _wasCreated;
private uint _nextPageId;
private FileStream? _fileStream;
private uint _firstFreePageId;
private MemoryMappedFile? _mappedFile;
/// <summary>
/// Gets the next page identifier that will be allocated.
/// </summary>
public uint NextPageId => _nextPageId;
/// <summary>
/// Indicates whether this file was newly created on the current open call.
/// </summary>
public bool WasCreated => _wasCreated;
/// <summary>
/// Initializes a new instance of the <see cref="PageFile"/> class.
/// Initializes a new instance of the <see cref="PageFile" /> class.
/// </summary>
/// <param name="filePath">The file path for the page file.</param>
/// <param name="config">The page file configuration.</param>
public PageFile(string filePath, PageFileConfig config)
{
_filePath = filePath ?? throw new ArgumentNullException(nameof(filePath));
FilePath = filePath ?? throw new ArgumentNullException(nameof(filePath));
_config = config;
}
/// <summary>
/// Gets the next page identifier that will be allocated.
/// </summary>
public uint NextPageId { get; private set; }
/// <summary>
/// Indicates whether this file was newly created on the current open call.
/// </summary>
public bool WasCreated { get; private set; }
/// <summary>
/// Gets the configured page size in bytes.
/// </summary>
@@ -98,7 +95,7 @@ public sealed class PageFile : IDisposable
/// <summary>
/// Gets the underlying file path.
/// </summary>
public string FilePath => _filePath;
public string FilePath { get; }
/// <summary>
/// Gets the current physical file length in bytes.
@@ -120,6 +117,31 @@ public sealed class PageFile : IDisposable
/// </summary>
public PageFileConfig Config => _config;
/// <summary>
/// Releases resources used by the page file.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
lock (_lock)
{
// 1. Flush any pending writes from memory-mapped file
if (_fileStream != null) _fileStream.Flush(true);
// 2. Close memory-mapped file first
_mappedFile?.Dispose();
// 3. Then close file stream
_fileStream?.Dispose();
_disposed = true;
}
GC.SuppressFinalize(this);
}
/// <summary>
/// Opens the page file, creating it if it doesn't exist
/// </summary>
@@ -130,26 +152,28 @@ public sealed class PageFile : IDisposable
if (_fileStream != null)
return; // Already open
var fileExists = File.Exists(_filePath);
bool fileExists = File.Exists(FilePath);
_fileStream = new FileStream(
_filePath,
FilePath,
FileMode.OpenOrCreate,
FileAccess.ReadWrite,
FileShare.None,
bufferSize: 4096,
4096,
FileOptions.RandomAccess);
_wasCreated = !fileExists || _fileStream.Length == 0;
if (_wasCreated)
WasCreated = !fileExists || _fileStream.Length == 0;
if (WasCreated)
{
// Initialize new file with 2 pages (Header + Collection Metadata)
_fileStream.SetLength(_config.InitialFileSize < _config.PageSize * 2 ? _config.PageSize * 2 : _config.InitialFileSize);
_fileStream.SetLength(_config.InitialFileSize < _config.PageSize * 2
? _config.PageSize * 2
: _config.InitialFileSize);
InitializeHeader();
}
// Initialize next page ID based on file length
_nextPageId = (uint)(_fileStream.Length / _config.PageSize);
NextPageId = (uint)(_fileStream.Length / _config.PageSize);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -157,7 +181,7 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
// Read free list head from Page 0
if (_fileStream.Length >= _config.PageSize)
@@ -228,7 +252,7 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var offset = (long)pageId * _config.PageSize;
long offset = pageId * _config.PageSize;
using var accessor = _mappedFile.CreateViewAccessor(offset, _config.PageSize, MemoryMappedFileAccess.Read);
var temp = new byte[_config.PageSize];
@@ -249,16 +273,15 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var offset = (long)pageId * _config.PageSize;
long offset = pageId * _config.PageSize;
// Ensure file is large enough
if (offset + _config.PageSize > _fileStream!.Length)
{
lock (_lock)
{
if (offset + _config.PageSize > _fileStream.Length)
{
var newSize = Math.Max(offset + _config.PageSize, _fileStream.Length * 2);
long newSize = Math.Max(offset + _config.PageSize, _fileStream.Length * 2);
_fileStream.SetLength(newSize);
// Recreate memory-mapped file with new size
@@ -269,8 +292,7 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
}
true);
}
}
@@ -294,7 +316,7 @@ public sealed class PageFile : IDisposable
// 1. Try to reuse a free page
if (_firstFreePageId != 0)
{
var recycledPageId = _firstFreePageId;
uint recycledPageId = _firstFreePageId;
// Read the recycled page to update the free list head
var buffer = new byte[_config.PageSize];
@@ -311,13 +333,13 @@ public sealed class PageFile : IDisposable
}
// 2. No free pages, append new one
var pageId = _nextPageId++;
uint pageId = NextPageId++;
// Extend file if necessary
var requiredLength = (long)(pageId + 1) * _config.PageSize;
long requiredLength = (pageId + 1) * _config.PageSize;
if (requiredLength > _fileStream.Length)
{
var newSize = Math.Max(requiredLength, _fileStream.Length * 2);
long newSize = Math.Max(requiredLength, _fileStream.Length * 2);
_fileStream.SetLength(newSize);
// Recreate memory-mapped file with new size
@@ -328,7 +350,7 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
}
return pageId;
@@ -401,11 +423,13 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var absoluteOffset = 32 + extensionOffset;
int absoluteOffset = 32 + extensionOffset;
if (absoluteOffset + destination.Length > _config.PageSize)
throw new ArgumentOutOfRangeException(nameof(destination), "Requested range exceeds page 0 extension region.");
throw new ArgumentOutOfRangeException(nameof(destination),
"Requested range exceeds page 0 extension region.");
using var accessor = _mappedFile.CreateViewAccessor(absoluteOffset, destination.Length, MemoryMappedFileAccess.Read);
using var accessor =
_mappedFile.CreateViewAccessor(absoluteOffset, destination.Length, MemoryMappedFileAccess.Read);
var temp = new byte[destination.Length];
accessor.ReadArray(0, temp, 0, temp.Length);
temp.CopyTo(destination);
@@ -427,11 +451,12 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var absoluteOffset = 32 + extensionOffset;
int absoluteOffset = 32 + extensionOffset;
if (absoluteOffset + source.Length > _config.PageSize)
throw new ArgumentOutOfRangeException(nameof(source), "Requested range exceeds page 0 extension region.");
using var accessor = _mappedFile.CreateViewAccessor(absoluteOffset, source.Length, MemoryMappedFileAccess.Write);
using var accessor =
_mappedFile.CreateViewAccessor(absoluteOffset, source.Length, MemoryMappedFileAccess.Write);
accessor.WriteArray(0, source.ToArray(), 0, source.Length);
}
@@ -443,7 +468,7 @@ public sealed class PageFile : IDisposable
{
lock (_lock)
{
_fileStream?.Flush(flushToDisk: true);
_fileStream?.Flush(true);
}
}
@@ -460,14 +485,11 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
var directory = Path.GetDirectoryName(destinationPath);
if (!string.IsNullOrWhiteSpace(directory))
{
Directory.CreateDirectory(directory);
}
string? directory = Path.GetDirectoryName(destinationPath);
if (!string.IsNullOrWhiteSpace(directory)) Directory.CreateDirectory(directory);
_fileStream!.Flush(flushToDisk: true);
var originalPosition = _fileStream.Position;
_fileStream!.Flush(true);
long originalPosition = _fileStream.Position;
try
{
_fileStream.Position = 0;
@@ -476,10 +498,10 @@ public sealed class PageFile : IDisposable
FileMode.Create,
FileAccess.Write,
FileShare.None,
bufferSize: 128 * 1024,
128 * 1024,
FileOptions.SequentialScan | FileOptions.WriteThrough);
_fileStream.CopyTo(destination);
destination.Flush(flushToDisk: true);
destination.Flush(true);
}
finally
{
@@ -508,11 +530,12 @@ public sealed class PageFile : IDisposable
FileMode.Open,
FileAccess.Read,
FileShare.Read,
bufferSize: 128 * 1024,
128 * 1024,
FileOptions.SequentialScan);
if (source.Length <= 0 || source.Length % _config.PageSize != 0)
throw new InvalidDataException($"Replacement file length must be a positive multiple of page size ({_config.PageSize}).");
throw new InvalidDataException(
$"Replacement file length must be a positive multiple of page size ({_config.PageSize}).");
_mappedFile?.Dispose();
_mappedFile = null;
@@ -520,7 +543,7 @@ public sealed class PageFile : IDisposable
_fileStream!.SetLength(source.Length);
_fileStream.Position = 0;
source.CopyTo(_fileStream);
_fileStream.Flush(flushToDisk: true);
_fileStream.Flush(true);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -528,16 +551,17 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
_nextPageId = (uint)(_fileStream.Length / _config.PageSize);
NextPageId = (uint)(_fileStream.Length / _config.PageSize);
_firstFreePageId = 0;
if (_fileStream.Length >= _config.PageSize)
{
const int pageHeaderSizeBytes = 32;
var headerSpan = new byte[pageHeaderSizeBytes];
using var accessor = _mappedFile.CreateViewAccessor(0, pageHeaderSizeBytes, MemoryMappedFileAccess.Read);
using var accessor =
_mappedFile.CreateViewAccessor(0, pageHeaderSizeBytes, MemoryMappedFileAccess.Read);
accessor.ReadArray(0, headerSpan, 0, pageHeaderSizeBytes);
var header = PageHeader.ReadFrom(headerSpan);
_firstFreePageId = header.NextPageId;
@@ -548,7 +572,7 @@ public sealed class PageFile : IDisposable
/// <summary>
/// Enumerates all free pages, combining explicit free-list entries and reclaimable empty pages.
/// </summary>
/// <param name="includeEmptyPages">If set to <see langword="true"/>, includes all-zero pages as reclaimable.</param>
/// <param name="includeEmptyPages">If set to <see langword="true" />, includes all-zero pages as reclaimable.</param>
/// <returns>A sorted list of free page identifiers.</returns>
public IReadOnlyList<uint> EnumerateFreePages(bool includeEmptyPages = true)
{
@@ -563,7 +587,10 @@ public sealed class PageFile : IDisposable
/// <summary>
/// Normalizes the free-list by rebuilding it from a deterministic sorted free page set.
/// </summary>
/// <param name="includeEmptyPages">If set to <see langword="true"/>, all-zero pages are converted into explicit free-list pages.</param>
/// <param name="includeEmptyPages">
/// If set to <see langword="true" />, all-zero pages are converted into explicit free-list
/// pages.
/// </param>
/// <returns>The number of pages in the normalized free-list.</returns>
public int NormalizeFreeList(bool includeEmptyPages = true)
{
@@ -580,7 +607,10 @@ public sealed class PageFile : IDisposable
/// Truncates contiguous reclaimable pages at the end of the file.
/// Reclaimable tail pages include explicit free pages and truly empty pages.
/// </summary>
/// <param name="minimumPageCount">Minimum number of pages that must remain after truncation (defaults to header + metadata pages).</param>
/// <param name="minimumPageCount">
/// Minimum number of pages that must remain after truncation (defaults to header + metadata
/// pages).
/// </param>
/// <returns>Details about the truncation operation.</returns>
public TailTruncationResult TruncateReclaimableTailPages(uint minimumPageCount = 2)
{
@@ -588,31 +618,22 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
if (_nextPageId <= minimumPageCount)
{
return TailTruncationResult.None(_nextPageId);
}
if (NextPageId <= minimumPageCount) return TailTruncationResult.None(NextPageId);
var freePages = new HashSet<uint>(CollectFreePageIds(includeEmptyPages: true));
var originalPageCount = _nextPageId;
var newPageCount = _nextPageId;
var freePages = new HashSet<uint>(CollectFreePageIds(true));
uint originalPageCount = NextPageId;
uint newPageCount = NextPageId;
var pageBuffer = new byte[_config.PageSize];
while (newPageCount > minimumPageCount)
{
var candidatePageId = newPageCount - 1;
if (!IsReclaimableTailPage(candidatePageId, freePages, pageBuffer))
{
break;
}
uint candidatePageId = newPageCount - 1;
if (!IsReclaimableTailPage(candidatePageId, freePages, pageBuffer)) break;
newPageCount--;
}
if (newPageCount == originalPageCount)
{
return TailTruncationResult.None(originalPageCount);
}
if (newPageCount == originalPageCount) return TailTruncationResult.None(originalPageCount);
freePages.RemoveWhere(pageId => pageId >= newPageCount);
var remainingFreePages = freePages.ToList();
@@ -622,10 +643,10 @@ public sealed class PageFile : IDisposable
_mappedFile?.Dispose();
_mappedFile = null;
var previousLengthBytes = _fileStream!.Length;
var newLengthBytes = (long)newPageCount * _config.PageSize;
long previousLengthBytes = _fileStream!.Length;
long newLengthBytes = newPageCount * _config.PageSize;
_fileStream.SetLength(newLengthBytes);
_fileStream.Flush(flushToDisk: true);
_fileStream.Flush(true);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -633,9 +654,9 @@ public sealed class PageFile : IDisposable
newLengthBytes,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
_nextPageId = newPageCount;
NextPageId = newPageCount;
return new TailTruncationResult(
originalPageCount,
@@ -655,8 +676,8 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
var targetLengthBytes = (long)_nextPageId * _config.PageSize;
var currentLengthBytes = _fileStream!.Length;
long targetLengthBytes = NextPageId * _config.PageSize;
long currentLengthBytes = _fileStream!.Length;
if (currentLengthBytes <= targetLengthBytes)
return 0;
@@ -664,7 +685,7 @@ public sealed class PageFile : IDisposable
_mappedFile = null;
_fileStream.SetLength(targetLengthBytes);
_fileStream.Flush(flushToDisk: true);
_fileStream.Flush(true);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -672,7 +693,7 @@ public sealed class PageFile : IDisposable
targetLengthBytes,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
return currentLengthBytes - targetLengthBytes;
}
@@ -683,7 +704,7 @@ public sealed class PageFile : IDisposable
/// </summary>
/// <param name="pageId">The page identifier to defragment.</param>
/// <param name="reclaimedBytes">The number of free bytes reclaimed by defragmentation.</param>
/// <returns><see langword="true"/> when the page layout changed; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when the page layout changed; otherwise, <see langword="false" />.</returns>
public bool DefragmentSlottedPage(uint pageId, out int reclaimedBytes)
{
var result = DefragmentSlottedPageWithStats(pageId);
@@ -701,7 +722,7 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
if (pageId >= _nextPageId)
if (pageId >= NextPageId)
throw new ArgumentOutOfRangeException(nameof(pageId));
var pageBuffer = new byte[_config.PageSize];
@@ -721,7 +742,7 @@ public sealed class PageFile : IDisposable
/// </summary>
/// <param name="pageBuffer">The page buffer to compact in place.</param>
/// <param name="reclaimedBytes">The number of free bytes reclaimed by compaction.</param>
/// <returns><see langword="true"/> when compaction modified the page; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when compaction modified the page; otherwise, <see langword="false" />.</returns>
public static bool TryDefragmentSlottedPage(Span<byte> pageBuffer, out int reclaimedBytes)
{
var result = TryDefragmentSlottedPageWithStats(pageBuffer);
@@ -742,7 +763,7 @@ public sealed class PageFile : IDisposable
if (!IsSlottedPageType(header.PageType))
return SlottedPageDefragmentationResult.None;
var slotArrayEnd = SlottedPageHeader.Size + (header.SlotCount * SlotEntry.Size);
int slotArrayEnd = SlottedPageHeader.Size + header.SlotCount * SlotEntry.Size;
if (slotArrayEnd > pageBuffer.Length)
return SlottedPageDefragmentationResult.None;
@@ -750,29 +771,29 @@ public sealed class PageFile : IDisposable
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.Slice(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0 || slot.Length == 0)
continue;
var dataEnd = slot.Offset + slot.Length;
int dataEnd = slot.Offset + slot.Length;
if (slot.Offset < slotArrayEnd || dataEnd > pageBuffer.Length)
return SlottedPageDefragmentationResult.None;
var slotData = pageBuffer.Slice(slot.Offset, slot.Length).ToArray();
byte[] slotData = pageBuffer.Slice(slot.Offset, slot.Length).ToArray();
activeSlots.Add((i, slot, slotData));
}
var newFreeSpaceStart = (ushort)slotArrayEnd;
var writeCursor = pageBuffer.Length;
int writeCursor = pageBuffer.Length;
var changed = false;
var relocatedSlots = 0;
var oldFreeBytes = header.AvailableFreeSpace;
int oldFreeBytes = header.AvailableFreeSpace;
for (var i = 0; i < activeSlots.Count; i++)
{
var (slotIndex, slot, slotData) = activeSlots[i];
(ushort slotIndex, var slot, byte[] slotData) = activeSlots[i];
writeCursor -= slotData.Length;
if (writeCursor < newFreeSpaceStart)
return SlottedPageDefragmentationResult.None;
@@ -786,57 +807,24 @@ public sealed class PageFile : IDisposable
changed = true;
}
var slotOffset = SlottedPageHeader.Size + (slotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + slotIndex * SlotEntry.Size;
slot.WriteTo(pageBuffer.Slice(slotOffset, SlotEntry.Size));
}
if (writeCursor > newFreeSpaceStart)
{
pageBuffer.Slice(newFreeSpaceStart, writeCursor - newFreeSpaceStart).Clear();
}
if (header.FreeSpaceStart != newFreeSpaceStart || header.FreeSpaceEnd != writeCursor)
{
changed = true;
}
if (header.FreeSpaceStart != newFreeSpaceStart || header.FreeSpaceEnd != writeCursor) changed = true;
header.FreeSpaceStart = newFreeSpaceStart;
header.FreeSpaceEnd = (ushort)writeCursor;
header.WriteTo(pageBuffer);
var newFreeBytes = header.AvailableFreeSpace;
var reclaimedBytes = Math.Max(0, newFreeBytes - oldFreeBytes);
int newFreeBytes = header.AvailableFreeSpace;
int reclaimedBytes = Math.Max(0, newFreeBytes - oldFreeBytes);
return new SlottedPageDefragmentationResult(changed, reclaimedBytes, relocatedSlots);
}
/// <summary>
/// Releases resources used by the page file.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
lock (_lock)
{
// 1. Flush any pending writes from memory-mapped file
if (_fileStream != null)
{
_fileStream.Flush(flushToDisk: true);
}
// 2. Close memory-mapped file first
_mappedFile?.Dispose();
// 3. Then close file stream
_fileStream?.Dispose();
_disposed = true;
}
GC.SuppressFinalize(this);
}
private void EnsureFileOpen()
{
if (_fileStream == null || _mappedFile == null)
@@ -846,26 +834,23 @@ public sealed class PageFile : IDisposable
private List<uint> CollectFreePageIds(bool includeEmptyPages)
{
var freePages = new HashSet<uint>();
if (_nextPageId <= 2)
if (NextPageId <= 2)
return [];
var pageBuffer = new byte[_config.PageSize];
var seen = new HashSet<uint>();
var current = _firstFreePageId;
while (current != 0 && current < _nextPageId && seen.Add(current))
uint current = _firstFreePageId;
while (current != 0 && current < NextPageId && seen.Add(current))
{
if (current > 1)
{
freePages.Add(current);
}
if (current > 1) freePages.Add(current);
ReadPage(current, pageBuffer);
var header = PageHeader.ReadFrom(pageBuffer);
current = header.NextPageId;
}
for (uint pageId = 2; pageId < _nextPageId; pageId++)
for (uint pageId = 2; pageId < NextPageId; pageId++)
{
ReadPage(pageId, pageBuffer);
var header = PageHeader.ReadFrom(pageBuffer);
@@ -876,10 +861,7 @@ public sealed class PageFile : IDisposable
continue;
}
if (includeEmptyPages && IsTrulyEmptyPage(pageBuffer))
{
freePages.Add(pageId);
}
if (includeEmptyPages && IsTrulyEmptyPage(pageBuffer)) freePages.Add(pageId);
}
var ordered = freePages.ToList();
@@ -900,8 +882,8 @@ public sealed class PageFile : IDisposable
for (var i = 0; i < sortedFreePageIds.Count; i++)
{
var pageId = sortedFreePageIds[i];
var nextPageId = i + 1 < sortedFreePageIds.Count ? sortedFreePageIds[i + 1] : 0;
uint pageId = sortedFreePageIds[i];
uint nextPageId = i + 1 < sortedFreePageIds.Count ? sortedFreePageIds[i + 1] : 0;
Array.Clear(pageBuffer, 0, pageBuffer.Length);
var freeHeader = new PageHeader
@@ -928,7 +910,7 @@ public sealed class PageFile : IDisposable
private bool IsReclaimableTailPage(uint pageId, HashSet<uint> explicitFreePages, byte[] pageBuffer)
{
if (pageId <= 1 || pageId >= _nextPageId)
if (pageId <= 1 || pageId >= NextPageId)
return false;
if (explicitFreePages.Contains(pageId))
@@ -945,10 +927,8 @@ public sealed class PageFile : IDisposable
private static bool IsTrulyEmptyPage(ReadOnlySpan<byte> pageBuffer)
{
for (var i = 0; i < pageBuffer.Length; i++)
{
if (pageBuffer[i] != 0)
return false;
}
return true;
}
@@ -965,7 +945,7 @@ public readonly struct SlottedPageDefragmentationResult
public static SlottedPageDefragmentationResult None => new(false, 0, 0);
/// <summary>
/// Initializes a new instance of the <see cref="SlottedPageDefragmentationResult"/> struct.
/// Initializes a new instance of the <see cref="SlottedPageDefragmentationResult" /> struct.
/// </summary>
/// <param name="changed">Indicates whether the page layout changed.</param>
/// <param name="reclaimedBytes">The number of bytes reclaimed.</param>
@@ -999,7 +979,7 @@ public readonly struct SlottedPageDefragmentationResult
public readonly struct TailTruncationResult
{
/// <summary>
/// Initializes a new instance of the <see cref="TailTruncationResult"/> struct.
/// Initializes a new instance of the <see cref="TailTruncationResult" /> struct.
/// </summary>
/// <param name="prePageCount">The page count before truncation.</param>
/// <param name="postPageCount">The page count after truncation.</param>

36
src/CBDD.Core/Storage/PageHeader.cs
View File

@@ -11,43 +11,31 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
public struct PageHeader
{
/// <summary>Page ID (offset in pages from start of file)</summary>
[FieldOffset(0)]
public uint PageId;
[FieldOffset(0)] public uint PageId;
/// <summary>Type of this page</summary>
[FieldOffset(4)]
public PageType PageType;
[FieldOffset(4)] public PageType PageType;
/// <summary>Number of free bytes in this page</summary>
[FieldOffset(5)]
public ushort FreeBytes;
[FieldOffset(5)] public ushort FreeBytes;
/// <summary>ID of next page in linked list (0 if none). For Page 0 (Header), this points to the First Free Page.</summary>
[FieldOffset(7)]
public uint NextPageId;
[FieldOffset(7)] public uint NextPageId;
/// <summary>Transaction ID that last modified this page</summary>
[FieldOffset(11)]
public ulong TransactionId;
[FieldOffset(11)] public ulong TransactionId;
/// <summary>Checksum for data integrity (CRC32)</summary>
[FieldOffset(19)]
public uint Checksum;
[FieldOffset(19)] public uint Checksum;
/// <summary>Dictionary Root Page ID (Only used in Page 0 / File Header)</summary>
[FieldOffset(23)]
public uint DictionaryRootPageId;
[FieldOffset(23)] public uint DictionaryRootPageId;
[FieldOffset(27)]
private byte _reserved5;
[FieldOffset(28)]
private byte _reserved6;
[FieldOffset(29)]
private byte _reserved7;
[FieldOffset(30)]
private byte _reserved8;
[FieldOffset(31)]
private byte _reserved9;
[FieldOffset(27)] private byte _reserved5;
[FieldOffset(28)] private byte _reserved6;
[FieldOffset(29)] private byte _reserved7;
[FieldOffset(30)] private byte _reserved8;
[FieldOffset(31)] private byte _reserved9;
/// <summary>
/// Writes the header to a span

47
src/CBDD.Core/Storage/SlottedPage.cs
View File

@@ -1,3 +1,4 @@
using System.Buffers.Binary;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Storage;
@@ -10,36 +11,28 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
public struct SlottedPageHeader
{
/// <summary>Page ID</summary>
[FieldOffset(0)]
public uint PageId;
[FieldOffset(0)] public uint PageId;
/// <summary>Type of page (Data, Overflow, Index, Metadata)</summary>
[FieldOffset(4)]
public PageType PageType;
[FieldOffset(4)] public PageType PageType;
/// <summary>Number of slot entries in this page</summary>
[FieldOffset(8)]
public ushort SlotCount;
[FieldOffset(8)] public ushort SlotCount;
/// <summary>Offset where free space starts (grows down with slots)</summary>
[FieldOffset(10)]
public ushort FreeSpaceStart;
[FieldOffset(10)] public ushort FreeSpaceStart;
/// <summary>Offset where free space ends (grows up with data)</summary>
[FieldOffset(12)]
public ushort FreeSpaceEnd;
[FieldOffset(12)] public ushort FreeSpaceEnd;
/// <summary>Next overflow page ID (0 if none)</summary>
[FieldOffset(14)]
public uint NextOverflowPage;
[FieldOffset(14)] public uint NextOverflowPage;
/// <summary>Transaction ID that last modified this page</summary>
[FieldOffset(18)]
public uint TransactionId;
[FieldOffset(18)] public uint TransactionId;
/// <summary>Reserved for future use</summary>
[FieldOffset(22)]
public ushort Reserved;
[FieldOffset(22)] public ushort Reserved;
public const int Size = 24;
@@ -90,16 +83,13 @@ public struct SlottedPageHeader
public struct SlotEntry
{
/// <summary>Offset to document data within page</summary>
[FieldOffset(0)]
public ushort Offset;
[FieldOffset(0)] public ushort Offset;
/// <summary>Length of document data in bytes</summary>
[FieldOffset(2)]
public ushort Length;
[FieldOffset(2)] public ushort Length;
/// <summary>Slot flags (deleted, overflow, etc.)</summary>
[FieldOffset(4)]
public SlotFlags Flags;
[FieldOffset(4)] public SlotFlags Flags;
public const int Size = 8;
@@ -144,7 +134,7 @@ public enum SlotFlags : uint
HasOverflow = 1 << 1,
/// <summary>Document data is compressed</summary>
Compressed = 1 << 2,
Compressed = 1 << 2
}
/// <summary>
@@ -157,13 +147,14 @@ public readonly struct DocumentLocation
/// Gets the page identifier containing the document.
/// </summary>
public uint PageId { get; init; }
/// <summary>
/// Gets the slot index within the page.
/// </summary>
public ushort SlotIndex { get; init; }
/// <summary>
/// Initializes a new instance of the <see cref="DocumentLocation"/> struct.
/// Initializes a new instance of the <see cref="DocumentLocation" /> struct.
/// </summary>
/// <param name="pageId">The page identifier containing the document.</param>
/// <param name="slotIndex">The slot index within the page.</param>
@@ -182,8 +173,8 @@ public readonly struct DocumentLocation
if (destination.Length < 6)
throw new ArgumentException("Destination must be at least 6 bytes", nameof(destination));
System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(destination, PageId);
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(destination.Slice(4), SlotIndex);
BinaryPrimitives.WriteUInt32LittleEndian(destination, PageId);
BinaryPrimitives.WriteUInt16LittleEndian(destination.Slice(4), SlotIndex);
}
/// <summary>
@@ -195,8 +186,8 @@ public readonly struct DocumentLocation
if (source.Length < 6)
throw new ArgumentException("Source must be at least 6 bytes", nameof(source));
var pageId = System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(source);
var slotIndex = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(source.Slice(4));
uint pageId = BinaryPrimitives.ReadUInt32LittleEndian(source);
ushort slotIndex = BinaryPrimitives.ReadUInt16LittleEndian(source.Slice(4));
return new DocumentLocation(pageId, slotIndex);
}

49
src/CBDD.Core/Storage/SpatialPage.cs
View File

@@ -1,6 +1,5 @@
using System.Buffers.Binary;
using System.Runtime.InteropServices;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
namespace ZB.MOM.WW.CBDD.Core.Storage;
@@ -58,50 +57,71 @@ internal struct SpatialPage
/// Gets a value indicating whether the page is a leaf node.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns><see langword="true"/> if the page is a leaf node; otherwise, <see langword="false"/>.</returns>
public static bool GetIsLeaf(ReadOnlySpan<byte> page) => page[IsLeafOffset] == 1;
/// <returns><see langword="true" /> if the page is a leaf node; otherwise, <see langword="false" />.</returns>
public static bool GetIsLeaf(ReadOnlySpan<byte> page)
{
return page[IsLeafOffset] == 1;
}
/// <summary>
/// Gets the tree level stored in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The level value.</returns>
public static byte GetLevel(ReadOnlySpan<byte> page) => page[LevelOffset];
public static byte GetLevel(ReadOnlySpan<byte> page)
{
return page[LevelOffset];
}
/// <summary>
/// Gets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The number of entries.</returns>
public static ushort GetEntryCount(ReadOnlySpan<byte> page) => BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(EntryCountOffset));
public static ushort GetEntryCount(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(EntryCountOffset));
}
/// <summary>
/// Sets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="count">The entry count to set.</param>
public static void SetEntryCount(Span<byte> page, ushort count) => BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(EntryCountOffset), count);
public static void SetEntryCount(Span<byte> page, ushort count)
{
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(EntryCountOffset), count);
}
/// <summary>
/// Gets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The parent page identifier.</returns>
public static uint GetParentPageId(ReadOnlySpan<byte> page) => BinaryPrimitives.ReadUInt32LittleEndian(page.Slice(ParentPageIdOffset));
public static uint GetParentPageId(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadUInt32LittleEndian(page.Slice(ParentPageIdOffset));
}
/// <summary>
/// Sets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="parentId">The parent page identifier.</param>
public static void SetParentPageId(Span<byte> page, uint parentId) => BinaryPrimitives.WriteUInt32LittleEndian(page.Slice(ParentPageIdOffset), parentId);
public static void SetParentPageId(Span<byte> page, uint parentId)
{
BinaryPrimitives.WriteUInt32LittleEndian(page.Slice(ParentPageIdOffset), parentId);
}
/// <summary>
/// Gets the maximum number of entries that can fit in a page.
/// </summary>
/// <param name="pageSize">The page size in bytes.</param>
/// <returns>The maximum number of entries.</returns>
public static int GetMaxEntries(int pageSize) => (pageSize - DataOffset) / EntrySize;
public static int GetMaxEntries(int pageSize)
{
return (pageSize - DataOffset) / EntrySize;
}
/// <summary>
/// Writes an entry at the specified index.
@@ -112,7 +132,7 @@ internal struct SpatialPage
/// <param name="pointer">The document location pointer.</param>
public static void WriteEntry(Span<byte> page, int index, GeoBox mbr, DocumentLocation pointer)
{
int offset = DataOffset + (index * EntrySize);
int offset = DataOffset + index * EntrySize;
var entrySpan = page.Slice(offset, EntrySize);
// Write MBR (4 doubles)
@@ -135,7 +155,7 @@ internal struct SpatialPage
/// <param name="pointer">When this method returns, contains the entry document location.</param>
public static void ReadEntry(ReadOnlySpan<byte> page, int index, out GeoBox mbr, out DocumentLocation pointer)
{
int offset = DataOffset + (index * EntrySize);
int offset = DataOffset + index * EntrySize;
var entrySpan = page.Slice(offset, EntrySize);
var doubles = MemoryMarshal.Cast<byte, double>(entrySpan.Slice(0, 32));
@@ -147,19 +167,20 @@ internal struct SpatialPage
/// Calculates the combined MBR of all entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The combined MBR, or <see cref="GeoBox.Empty"/> when the page has no entries.</returns>
/// <returns>The combined MBR, or <see cref="GeoBox.Empty" /> when the page has no entries.</returns>
public static GeoBox CalculateMBR(ReadOnlySpan<byte> page)
{
ushort count = GetEntryCount(page);
if (count == 0) return GeoBox.Empty;
GeoBox result = GeoBox.Empty;
for (int i = 0; i < count; i++)
var result = GeoBox.Empty;
for (var i = 0; i < count; i++)
{
ReadEntry(page, i, out var mbr, out _);
if (i == 0) result = mbr;
else result = result.ExpandTo(mbr);
}
return result;
}
}

126
src/CBDD.Core/Storage/StorageEngine.Collections.cs
View File

@@ -1,9 +1,5 @@
using System;
using System.Collections.Generic;
using System.IO;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
@@ -81,38 +77,6 @@ public sealed partial class StorageEngine
.FirstOrDefault(x => string.Equals(x.Name, name, StringComparison.OrdinalIgnoreCase));
}
/// <summary>
/// Returns all collection metadata entries currently registered in page 1.
/// </summary>
public IReadOnlyList<CollectionMetadata> GetAllCollectionMetadata()
{
var result = new List<CollectionMetadata>();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
if (header.PageType != PageType.Collection || header.SlotCount == 0)
return result;
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0)
continue;
if (slot.Offset < SlottedPageHeader.Size || slot.Offset + slot.Length > buffer.Length)
continue;
if (TryDeserializeCollectionMetadata(buffer.AsSpan(slot.Offset, slot.Length), out var metadata) && metadata != null)
{
result.Add(metadata);
}
}
return result;
}
/// <summary>
/// Saves collection metadata to the metadata page.
/// </summary>
@@ -133,10 +97,7 @@ public sealed partial class StorageEngine
writer.Write((byte)idx.Type);
writer.Write(idx.RootPageId);
writer.Write(idx.PropertyPaths.Length);
foreach (var path in idx.PropertyPaths)
{
writer.Write(path);
}
foreach (string path in idx.PropertyPaths) writer.Write(path);
if (idx.Type == IndexType.Vector)
{
@@ -145,7 +106,7 @@ public sealed partial class StorageEngine
}
}
var newData = stream.ToArray();
byte[] newData = stream.ToArray();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
@@ -155,7 +116,7 @@ public sealed partial class StorageEngine
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0) continue;
@@ -163,7 +124,7 @@ public sealed partial class StorageEngine
{
using var ms = new MemoryStream(buffer, slot.Offset, slot.Length, false);
using var reader = new BinaryReader(ms);
var name = reader.ReadString();
string name = reader.ReadString();
if (string.Equals(name, metadata.Name, StringComparison.OrdinalIgnoreCase))
{
@@ -171,22 +132,23 @@ public sealed partial class StorageEngine
break;
}
}
catch { }
catch
{
}
}
if (existingSlotIndex >= 0)
{
var slotOffset = SlottedPageHeader.Size + (existingSlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + existingSlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
slot.Flags |= SlotFlags.Deleted;
slot.WriteTo(buffer.AsSpan(slotOffset));
}
if (header.AvailableFreeSpace < newData.Length + SlotEntry.Size)
{
// Compact logic omitted as per current architecture
throw new InvalidOperationException("Not enough space in Metadata Page (Page 1) to save collection metadata.");
}
throw new InvalidOperationException(
"Not enough space in Metadata Page (Page 1) to save collection metadata.");
int docOffset = header.FreeSpaceEnd - newData.Length;
newData.CopyTo(buffer.AsSpan(docOffset));
@@ -202,7 +164,7 @@ public sealed partial class StorageEngine
header.SlotCount++;
}
var newSlotEntryOffset = SlottedPageHeader.Size + (slotIndex * SlotEntry.Size);
int newSlotEntryOffset = SlottedPageHeader.Size + slotIndex * SlotEntry.Size;
var newSlot = new SlotEntry
{
Offset = (ushort)docOffset,
@@ -213,14 +175,52 @@ public sealed partial class StorageEngine
header.FreeSpaceEnd = (ushort)docOffset;
if (existingSlotIndex == -1)
{
header.FreeSpaceStart = (ushort)(SlottedPageHeader.Size + (header.SlotCount * SlotEntry.Size));
}
header.FreeSpaceStart = (ushort)(SlottedPageHeader.Size + header.SlotCount * SlotEntry.Size);
header.WriteTo(buffer);
WritePageImmediate(1, buffer);
}
/// <summary>
/// Registers all BSON keys used by a set of mappers into the global dictionary.
/// </summary>
/// <param name="mappers">The mappers whose keys should be registered.</param>
public void RegisterMappers(IEnumerable<IDocumentMapper> mappers)
{
var allKeys = mappers.SelectMany(m => m.UsedKeys).Distinct();
RegisterKeys(allKeys);
}
/// <summary>
/// Returns all collection metadata entries currently registered in page 1.
/// </summary>
public IReadOnlyList<CollectionMetadata> GetAllCollectionMetadata()
{
var result = new List<CollectionMetadata>();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
if (header.PageType != PageType.Collection || header.SlotCount == 0)
return result;
for (ushort i = 0; i < header.SlotCount; i++)
{
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0)
continue;
if (slot.Offset < SlottedPageHeader.Size || slot.Offset + slot.Length > buffer.Length)
continue;
if (TryDeserializeCollectionMetadata(buffer.AsSpan(slot.Offset, slot.Length), out var metadata) &&
metadata != null) result.Add(metadata);
}
return result;
}
private static bool TryDeserializeCollectionMetadata(ReadOnlySpan<byte> rawBytes, out CollectionMetadata? metadata)
{
metadata = null;
@@ -230,16 +230,16 @@ public sealed partial class StorageEngine
using var ms = new MemoryStream(rawBytes.ToArray());
using var reader = new BinaryReader(ms);
var collName = reader.ReadString();
string collName = reader.ReadString();
var parsed = new CollectionMetadata { Name = collName };
parsed.PrimaryRootPageId = reader.ReadUInt32();
parsed.SchemaRootPageId = reader.ReadUInt32();
var indexCount = reader.ReadInt32();
int indexCount = reader.ReadInt32();
if (indexCount < 0)
return false;
for (int j = 0; j < indexCount; j++)
for (var j = 0; j < indexCount; j++)
{
var idx = new IndexMetadata
{
@@ -249,12 +249,12 @@ public sealed partial class StorageEngine
RootPageId = reader.ReadUInt32()
};
var pathCount = reader.ReadInt32();
int pathCount = reader.ReadInt32();
if (pathCount < 0)
return false;
idx.PropertyPaths = new string[pathCount];
for (int k = 0; k < pathCount; k++)
for (var k = 0; k < pathCount; k++)
idx.PropertyPaths[k] = reader.ReadString();
if (idx.Type == IndexType.Vector)
@@ -274,14 +274,4 @@ public sealed partial class StorageEngine
return false;
}
}
/// <summary>
/// Registers all BSON keys used by a set of mappers into the global dictionary.
/// </summary>
/// <param name="mappers">The mappers whose keys should be registered.</param>
public void RegisterMappers(IEnumerable<IDocumentMapper> mappers)
{
var allKeys = mappers.SelectMany(m => m.UsedKeys).Distinct();
RegisterKeys(allKeys);
}
}

79
src/CBDD.Core/Storage/StorageEngine.Diagnostics.cs
View File

@@ -89,7 +89,8 @@ public sealed class CollectionCompressionRatioEntry
/// <summary>
/// Gets the compression ratio.
/// </summary>
public double CompressionRatio => BytesAfterCompression <= 0 ? 1.0 : (double)BytesBeforeCompression / BytesAfterCompression;
public double CompressionRatio =>
BytesAfterCompression <= 0 ? 1.0 : (double)BytesBeforeCompression / BytesAfterCompression;
}
/// <summary>
@@ -182,7 +183,7 @@ public sealed partial class StorageEngine
/// </summary>
public IReadOnlyList<PageTypeUsageEntry> GetPageUsageByPageType()
{
var pageCount = _pageFile.NextPageId;
uint pageCount = _pageFile.NextPageId;
var buffer = new byte[_pageFile.PageSize];
var counts = new Dictionary<PageType, int>();
@@ -190,7 +191,7 @@ public sealed partial class StorageEngine
{
_pageFile.ReadPage(pageId, buffer);
var pageType = PageHeader.ReadFrom(buffer).PageType;
counts[pageType] = counts.TryGetValue(pageType, out var count) ? count + 1 : 1;
counts[pageType] = counts.TryGetValue(pageType, out int count) ? count + 1 : 1;
}
return counts
@@ -221,27 +222,23 @@ public sealed partial class StorageEngine
pageIds.Add(metadata.SchemaRootPageId);
foreach (var indexMetadata in metadata.Indexes)
{
if (indexMetadata.RootPageId != 0)
pageIds.Add(indexMetadata.RootPageId);
}
foreach (var location in EnumeratePrimaryLocations(metadata))
{
pageIds.Add(location.PageId);
if (TryReadFirstOverflowPage(location, out var firstOverflowPage))
{
if (TryReadFirstOverflowPage(location, out uint firstOverflowPage))
AddOverflowChainPages(pageIds, firstOverflowPage);
}
}
int data = 0;
int overflow = 0;
int indexPages = 0;
int other = 0;
var data = 0;
var overflow = 0;
var indexPages = 0;
var other = 0;
var pageBuffer = new byte[_pageFile.PageSize];
foreach (var pageId in pageIds)
foreach (uint pageId in pageIds)
{
if (pageId >= _pageFile.NextPageId)
continue;
@@ -250,22 +247,14 @@ public sealed partial class StorageEngine
var pageType = PageHeader.ReadFrom(pageBuffer).PageType;
if (pageType == PageType.Data)
{
data++;
}
else if (pageType == PageType.Overflow)
{
overflow++;
}
else if (pageType == PageType.Index || pageType == PageType.Vector || pageType == PageType.Spatial)
{
indexPages++;
}
else
{
other++;
}
}
results.Add(new CollectionPageUsageEntry
{
@@ -298,7 +287,8 @@ public sealed partial class StorageEngine
foreach (var location in EnumeratePrimaryLocations(metadata))
{
if (!TryReadSlotPayloadStats(location, out var isCompressed, out var originalBytes, out var storedBytes))
if (!TryReadSlotPayloadStats(location, out bool isCompressed, out int originalBytes,
out int storedBytes))
continue;
docs++;
@@ -343,8 +333,8 @@ public sealed partial class StorageEngine
/// </summary>
public FragmentationMapReport GetFragmentationMap()
{
var freePageSet = new HashSet<uint>(_pageFile.EnumerateFreePages(includeEmptyPages: true));
var pageCount = _pageFile.NextPageId;
var freePageSet = new HashSet<uint>(_pageFile.EnumerateFreePages());
uint pageCount = _pageFile.NextPageId;
var buffer = new byte[_pageFile.PageSize];
var pages = new List<FragmentationPageEntry>((int)pageCount);
@@ -354,17 +344,12 @@ public sealed partial class StorageEngine
{
_pageFile.ReadPage(pageId, buffer);
var pageHeader = PageHeader.ReadFrom(buffer);
var isFreePage = freePageSet.Contains(pageId);
bool isFreePage = freePageSet.Contains(pageId);
int freeBytes = 0;
var freeBytes = 0;
if (isFreePage)
{
freeBytes = _pageFile.PageSize;
}
else if (TryReadSlottedFreeSpace(buffer, out var slottedFreeBytes))
{
freeBytes = slottedFreeBytes;
}
else if (TryReadSlottedFreeSpace(buffer, out int slottedFreeBytes)) freeBytes = slottedFreeBytes;
totalFreeBytes += freeBytes;
@@ -378,7 +363,7 @@ public sealed partial class StorageEngine
}
uint tailReclaimablePages = 0;
for (var i = pageCount; i > 2; i--)
for (uint i = pageCount; i > 2; i--)
{
if (!freePageSet.Contains(i - 1))
break;
@@ -386,12 +371,12 @@ public sealed partial class StorageEngine
tailReclaimablePages++;
}
var fileBytes = Math.Max(1L, _pageFile.FileLengthBytes);
long fileBytes = Math.Max(1L, _pageFile.FileLengthBytes);
return new FragmentationMapReport
{
Pages = pages,
TotalFreeBytes = totalFreeBytes,
FragmentationPercent = (totalFreeBytes * 100d) / fileBytes,
FragmentationPercent = totalFreeBytes * 100d / fileBytes,
TailReclaimablePages = tailReclaimablePages
};
}
@@ -403,11 +388,9 @@ public sealed partial class StorageEngine
var index = new BTreeIndex(this, IndexOptions.CreateUnique("_id"), metadata.PrimaryRootPageId);
foreach (var entry in index.Range(IndexKey.MinKey, IndexKey.MaxKey, IndexDirection.Forward, transactionId: 0))
{
foreach (var entry in index.Range(IndexKey.MinKey, IndexKey.MaxKey, IndexDirection.Forward, 0))
yield return entry.Location;
}
}
private bool TryReadFirstOverflowPage(in DocumentLocation location, out uint firstOverflowPage)
{
@@ -419,7 +402,7 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
@@ -441,7 +424,7 @@ public sealed partial class StorageEngine
var buffer = new byte[_pageFile.PageSize];
var visited = new HashSet<uint>();
var current = firstOverflowPage;
uint current = firstOverflowPage;
while (current != 0 && current < _pageFile.NextPageId && visited.Add(current))
{
@@ -472,12 +455,12 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
var hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
isCompressed = (slot.Flags & SlotFlags.Compressed) != 0;
if (!hasOverflow)
@@ -492,7 +475,8 @@ public sealed partial class StorageEngine
if (slot.Length < CompressedPayloadHeader.Size)
return false;
var compressedHeader = CompressedPayloadHeader.ReadFrom(pageBuffer.AsSpan(slot.Offset, CompressedPayloadHeader.Size));
var compressedHeader =
CompressedPayloadHeader.ReadFrom(pageBuffer.AsSpan(slot.Offset, CompressedPayloadHeader.Size));
originalBytes = compressedHeader.OriginalLength;
return true;
}
@@ -501,7 +485,7 @@ public sealed partial class StorageEngine
return false;
var primaryPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
var totalStoredBytes = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
int totalStoredBytes = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
if (totalStoredBytes < 0)
return false;
@@ -522,8 +506,8 @@ public sealed partial class StorageEngine
else
{
storedPrefix.CopyTo(headerBuffer);
var copied = storedPrefix.Length;
var nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
int copied = storedPrefix.Length;
uint nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
var overflowBuffer = new byte[_pageFile.PageSize];
while (copied < CompressedPayloadHeader.Size && nextOverflow != 0 && nextOverflow < _pageFile.NextPageId)
@@ -533,7 +517,8 @@ public sealed partial class StorageEngine
if (overflowHeader.PageType != PageType.Overflow)
return false;
var available = Math.Min(CompressedPayloadHeader.Size - copied, _pageFile.PageSize - SlottedPageHeader.Size);
int available = Math.Min(CompressedPayloadHeader.Size - copied,
_pageFile.PageSize - SlottedPageHeader.Size);
overflowBuffer.AsSpan(SlottedPageHeader.Size, available).CopyTo(headerBuffer.Slice(copied));
copied += available;
nextOverflow = overflowHeader.NextOverflowPage;

177
src/CBDD.Core/Storage/StorageEngine.Dictionary.cs
View File

@@ -1,17 +1,92 @@
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
private readonly ConcurrentDictionary<string, ushort> _dictionaryCache = new(StringComparer.OrdinalIgnoreCase);
// Lock for dictionary modifications (simple lock for now, could be RW lock)
private readonly object _dictionaryLock = new();
private readonly ConcurrentDictionary<ushort, string> _dictionaryReverseCache = new();
private uint _dictionaryRootPageId;
private ushort _nextDictionaryId;
// Lock for dictionary modifications (simple lock for now, could be RW lock)
private readonly object _dictionaryLock = new();
/// <summary>
/// Gets the key-to-id dictionary cache.
/// </summary>
/// <returns>The key-to-id map.</returns>
public ConcurrentDictionary<string, ushort> GetKeyMap()
{
return _dictionaryCache;
}
/// <summary>
/// Gets the id-to-key dictionary cache.
/// </summary>
/// <returns>The id-to-key map.</returns>
public ConcurrentDictionary<ushort, string> GetKeyReverseMap()
{
return _dictionaryReverseCache;
}
/// <summary>
/// Gets the ID for a dictionary key, creating it if it doesn't exist.
/// Thread-safe.
/// </summary>
/// <param name="key">The dictionary key.</param>
/// <returns>The dictionary identifier for the key.</returns>
public ushort GetOrAddDictionaryEntry(string key)
{
key = key.ToLowerInvariant();
if (_dictionaryCache.TryGetValue(key, out ushort id)) return id;
lock (_dictionaryLock)
{
// Double checked locking
if (_dictionaryCache.TryGetValue(key, out id)) return id;
// Try to find in storage (in case cache is incomplete or another process?)
// Note: FindAllGlobal loaded everything, so strict cache miss means it's not in DB.
// BUT if we support concurrent writers (multiple processed), we should re-check DB.
// Current CBDD seems to be single-process exclusive lock (FileShare.None).
// So in-memory cache is authoritative after load.
// Generate New ID
ushort nextId = _nextDictionaryId;
if (nextId == 0) nextId = DictionaryPage.ReservedValuesEnd + 1; // Should be init, but safety
// Insert into Page
// usage of default(ulong) or null transaction?
// Dictionary updates should ideally be transactional or immediate?
// "Immediate" for now to simplify, as dictionary is cross-collection.
// If we use transaction, we need to pass it in. For now, immediate write.
// We need to support "Insert Global" which handles overflow.
// DictionaryPage.Insert only handles single page.
// We need logic here to traverse chain and find space.
if (InsertDictionaryEntryGlobal(key, nextId))
{
_dictionaryCache[key] = nextId;
_dictionaryReverseCache[nextId] = key;
_nextDictionaryId++;
return nextId;
}
throw new InvalidOperationException("Failed to insert dictionary entry (Storage Full?)");
}
}
/// <summary>
/// Registers a set of keys in the global dictionary.
/// Ensures all keys are assigned an ID and persisted.
/// </summary>
/// <param name="keys">The keys to register.</param>
public void RegisterKeys(IEnumerable<string> keys)
{
foreach (string key in keys) GetOrAddDictionaryEntry(key.ToLowerInvariant());
}
private void InitializeDictionary()
{
@@ -57,13 +132,14 @@ public sealed partial class StorageEngine
// Warm cache
ushort maxId = DictionaryPage.ReservedValuesEnd;
foreach (var (key, val) in DictionaryPage.FindAllGlobal(this, _dictionaryRootPageId))
foreach ((string key, ushort val) in DictionaryPage.FindAllGlobal(this, _dictionaryRootPageId))
{
var lowerKey = key.ToLowerInvariant();
string lowerKey = key.ToLowerInvariant();
_dictionaryCache[lowerKey] = val;
_dictionaryReverseCache[val] = lowerKey;
if (val > maxId) maxId = val;
}
_nextDictionaryId = (ushort)(maxId + 1);
}
@@ -72,93 +148,25 @@ public sealed partial class StorageEngine
// Pre-register common array indices to avoid mapping during high-frequency writes
var indices = new List<string>(101);
for (int i = 0; i <= 100; i++) indices.Add(i.ToString());
for (var i = 0; i <= 100; i++) indices.Add(i.ToString());
RegisterKeys(indices);
}
/// <summary>
/// Gets the key-to-id dictionary cache.
/// </summary>
/// <returns>The key-to-id map.</returns>
public ConcurrentDictionary<string, ushort> GetKeyMap() => _dictionaryCache;
/// <summary>
/// Gets the id-to-key dictionary cache.
/// </summary>
/// <returns>The id-to-key map.</returns>
public ConcurrentDictionary<ushort, string> GetKeyReverseMap() => _dictionaryReverseCache;
/// <summary>
/// Gets the ID for a dictionary key, creating it if it doesn't exist.
/// Thread-safe.
/// </summary>
/// <param name="key">The dictionary key.</param>
/// <returns>The dictionary identifier for the key.</returns>
public ushort GetOrAddDictionaryEntry(string key)
{
key = key.ToLowerInvariant();
if (_dictionaryCache.TryGetValue(key, out var id))
{
return id;
}
lock (_dictionaryLock)
{
// Double checked locking
if (_dictionaryCache.TryGetValue(key, out id))
{
return id;
}
// Try to find in storage (in case cache is incomplete or another process?)
// Note: FindAllGlobal loaded everything, so strict cache miss means it's not in DB.
// BUT if we support concurrent writers (multiple processed), we should re-check DB.
// Current CBDD seems to be single-process exclusive lock (FileShare.None).
// So in-memory cache is authoritative after load.
// Generate New ID
ushort nextId = _nextDictionaryId;
if (nextId == 0) nextId = DictionaryPage.ReservedValuesEnd + 1; // Should be init, but safety
// Insert into Page
// usage of default(ulong) or null transaction?
// Dictionary updates should ideally be transactional or immediate?
// "Immediate" for now to simplify, as dictionary is cross-collection.
// If we use transaction, we need to pass it in. For now, immediate write.
// We need to support "Insert Global" which handles overflow.
// DictionaryPage.Insert only handles single page.
// We need logic here to traverse chain and find space.
if (InsertDictionaryEntryGlobal(key, nextId))
{
_dictionaryCache[key] = nextId;
_dictionaryReverseCache[nextId] = key;
_nextDictionaryId++;
return nextId;
}
else
{
throw new InvalidOperationException("Failed to insert dictionary entry (Storage Full?)");
}
}
}
/// <summary>
/// Gets the dictionary key for an identifier.
/// </summary>
/// <param name="id">The dictionary identifier.</param>
/// <returns>The dictionary key if found; otherwise, <see langword="null"/>.</returns>
/// <returns>The dictionary key if found; otherwise, <see langword="null" />.</returns>
public string? GetDictionaryKey(ushort id)
{
if (_dictionaryReverseCache.TryGetValue(id, out var key))
if (_dictionaryReverseCache.TryGetValue(id, out string? key))
return key;
return null;
}
private bool InsertDictionaryEntryGlobal(string key, ushort value)
{
var pageId = _dictionaryRootPageId;
uint pageId = _dictionaryRootPageId;
var pageBuffer = new byte[PageSize];
while (true)
@@ -182,7 +190,7 @@ public sealed partial class StorageEngine
}
// No Next Page - Allocate New
var newPageId = AllocatePage();
uint newPageId = AllocatePage();
var newPageBuffer = new byte[PageSize];
DictionaryPage.Initialize(newPageBuffer, newPageId);
@@ -203,17 +211,4 @@ public sealed partial class StorageEngine
return true;
}
}
/// <summary>
/// Registers a set of keys in the global dictionary.
/// Ensures all keys are assigned an ID and persisted.
/// </summary>
/// <param name="keys">The keys to register.</param>
public void RegisterKeys(IEnumerable<string> keys)
{
foreach (var key in keys)
{
GetOrAddDictionaryEntry(key.ToLowerInvariant());
}
}
}

17
src/CBDD.Core/Storage/StorageEngine.Format.cs
View File

@@ -39,7 +39,8 @@ internal readonly struct StorageFormatMetadata
/// </summary>
public bool CompressionCapabilityEnabled => (FeatureFlags & StorageFeatureFlags.CompressionCapability) != 0;
private StorageFormatMetadata(bool isPresent, byte version, StorageFeatureFlags featureFlags, CompressionCodec defaultCodec)
private StorageFormatMetadata(bool isPresent, byte version, StorageFeatureFlags featureFlags,
CompressionCodec defaultCodec)
{
IsPresent = isPresent;
Version = version;
@@ -53,7 +54,8 @@ internal readonly struct StorageFormatMetadata
/// <param name="version">The storage format version.</param>
/// <param name="featureFlags">Enabled feature flags.</param>
/// <param name="defaultCodec">The default compression codec.</param>
public static StorageFormatMetadata Present(byte version, StorageFeatureFlags featureFlags, CompressionCodec defaultCodec)
public static StorageFormatMetadata Present(byte version, StorageFeatureFlags featureFlags,
CompressionCodec defaultCodec)
{
return new StorageFormatMetadata(true, version, featureFlags, defaultCodec);
}
@@ -88,12 +90,13 @@ public sealed partial class StorageEngine
return metadata;
if (!_pageFile.WasCreated)
return StorageFormatMetadata.Legacy(_compressionOptions.Codec);
return StorageFormatMetadata.Legacy(CompressionOptions.Codec);
var featureFlags = _compressionOptions.EnableCompression
var featureFlags = CompressionOptions.EnableCompression
? StorageFeatureFlags.CompressionCapability
: StorageFeatureFlags.None;
var initialMetadata = StorageFormatMetadata.Present(CurrentStorageFormatVersion, featureFlags, _compressionOptions.Codec);
var initialMetadata =
StorageFormatMetadata.Present(CurrentStorageFormatVersion, featureFlags, CompressionOptions.Codec);
WriteStorageFormatMetadata(initialMetadata);
return initialMetadata;
}
@@ -104,11 +107,11 @@ public sealed partial class StorageEngine
if (source.Length < StorageFormatMetadata.WireSize)
return false;
var magic = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(0, 4));
uint magic = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(0, 4));
if (magic != StorageFormatMagic)
return false;
var version = source[4];
byte version = source[4];
var featureFlags = (StorageFeatureFlags)source[5];
var codec = (CompressionCodec)source[6];
if (!Enum.IsDefined(codec))

921
src/CBDD.Core/Storage/StorageEngine.Maintenance.cs
View File

File diff suppressed because it is too large Load Diff

2
src/CBDD.Core/Storage/StorageEngine.Memory.cs
View File

@@ -1,5 +1,3 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine

80
src/CBDD.Core/Storage/StorageEngine.Migration.cs
View File

@@ -117,7 +117,8 @@ public sealed partial class StorageEngine
/// </summary>
/// <param name="options">Optional compression migration options.</param>
/// <param name="ct">A token used to cancel the operation.</param>
public async Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null, CancellationToken ct = default)
public async Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null,
CancellationToken ct = default)
{
var normalized = NormalizeMigrationOptions(options);
@@ -147,13 +148,13 @@ public sealed partial class StorageEngine
{
ct.ThrowIfCancellationRequested();
if (!TryReadStoredPayload(location, out var storedPayload, out var isCompressed))
if (!TryReadStoredPayload(location, out byte[] storedPayload, out bool isCompressed))
{
docsSkipped++;
continue;
}
if (!TryGetLogicalPayload(storedPayload, isCompressed, out var logicalPayload))
if (!TryGetLogicalPayload(storedPayload, isCompressed, out byte[] logicalPayload))
{
docsSkipped++;
continue;
@@ -162,15 +163,14 @@ public sealed partial class StorageEngine
docsScanned++;
bytesBefore += logicalPayload.Length;
var targetStored = BuildTargetStoredPayload(logicalPayload, normalized, out var targetCompressed);
byte[] targetStored =
BuildTargetStoredPayload(logicalPayload, normalized, out bool targetCompressed);
bytesEstimatedAfter += targetStored.Length;
if (normalized.DryRun)
{
continue;
}
if (normalized.DryRun) continue;
if (!TryRewriteStoredPayloadAtLocation(location, targetStored, targetCompressed, out var actualStoredBytes))
if (!TryRewriteStoredPayloadAtLocation(location, targetStored, targetCompressed,
out int actualStoredBytes))
{
docsSkipped++;
continue;
@@ -184,9 +184,9 @@ public sealed partial class StorageEngine
if (!normalized.DryRun)
{
var metadata = StorageFormatMetadata.Present(
version: 1,
featureFlags: StorageFeatureFlags.CompressionCapability,
defaultCodec: normalized.Codec);
1,
StorageFeatureFlags.CompressionCapability,
normalized.Codec);
WriteStorageFormatMetadata(metadata);
_pageFile.Flush();
}
@@ -221,7 +221,8 @@ public sealed partial class StorageEngine
var normalized = options ?? new CompressionMigrationOptions();
if (!Enum.IsDefined(normalized.Codec) || normalized.Codec == CompressionCodec.None)
throw new ArgumentOutOfRangeException(nameof(options), "Migration codec must be a supported non-None codec.");
throw new ArgumentOutOfRangeException(nameof(options),
"Migration codec must be a supported non-None codec.");
if (normalized.MinSizeBytes < 0)
throw new ArgumentOutOfRangeException(nameof(options), "MinSizeBytes must be non-negative.");
@@ -250,7 +251,8 @@ public sealed partial class StorageEngine
.ToList();
}
private byte[] BuildTargetStoredPayload(ReadOnlySpan<byte> logicalPayload, CompressionMigrationOptions options, out bool compressed)
private byte[] BuildTargetStoredPayload(ReadOnlySpan<byte> logicalPayload, CompressionMigrationOptions options,
out bool compressed)
{
compressed = false;
@@ -259,10 +261,10 @@ public sealed partial class StorageEngine
try
{
var compressedPayload = _compressionService.Compress(logicalPayload, options.Codec, options.Level);
var storedLength = CompressedPayloadHeader.Size + compressedPayload.Length;
var savings = logicalPayload.Length - storedLength;
var savingsPercent = logicalPayload.Length == 0 ? 0 : (int)((savings * 100L) / logicalPayload.Length);
byte[] compressedPayload = CompressionService.Compress(logicalPayload, options.Codec, options.Level);
int storedLength = CompressedPayloadHeader.Size + compressedPayload.Length;
int savings = logicalPayload.Length - storedLength;
int savingsPercent = logicalPayload.Length == 0 ? 0 : (int)(savings * 100L / logicalPayload.Length);
if (savings <= 0 || savingsPercent < options.MinSavingsPercent)
return logicalPayload.ToArray();
@@ -308,11 +310,11 @@ public sealed partial class StorageEngine
try
{
logicalPayload = _compressionService.Decompress(
logicalPayload = CompressionService.Decompress(
compressedPayload,
header.Codec,
header.OriginalLength,
Math.Max(header.OriginalLength, _compressionOptions.MaxDecompressedSizeBytes));
Math.Max(header.OriginalLength, CompressionOptions.MaxDecompressedSizeBytes));
return true;
}
catch
@@ -336,13 +338,13 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
isCompressed = (slot.Flags & SlotFlags.Compressed) != 0;
var hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
if (!hasOverflow)
{
@@ -354,14 +356,14 @@ public sealed partial class StorageEngine
return false;
var primaryPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
var totalStoredLength = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
var nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
int totalStoredLength = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
uint nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
if (totalStoredLength < 0)
return false;
var output = new byte[totalStoredLength];
var primaryChunk = primaryPayload.Slice(8);
var copied = Math.Min(primaryChunk.Length, output.Length);
int copied = Math.Min(primaryChunk.Length, output.Length);
primaryChunk.Slice(0, copied).CopyTo(output);
var overflowBuffer = new byte[_pageFile.PageSize];
@@ -372,7 +374,7 @@ public sealed partial class StorageEngine
if (overflowHeader.PageType != PageType.Overflow)
return false;
var chunk = Math.Min(output.Length - copied, _pageFile.PageSize - SlottedPageHeader.Size);
int chunk = Math.Min(output.Length - copied, _pageFile.PageSize - SlottedPageHeader.Size);
overflowBuffer.AsSpan(SlottedPageHeader.Size, chunk).CopyTo(output.AsSpan(copied));
copied += chunk;
nextOverflow = overflowHeader.NextOverflowPage;
@@ -403,12 +405,12 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= pageHeader.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
var oldHasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool oldHasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
uint oldOverflowHead = 0;
if (oldHasOverflow)
{
@@ -442,12 +444,12 @@ public sealed partial class StorageEngine
if (slot.Length < 8)
return false;
var primaryChunkSize = slot.Length - 8;
int primaryChunkSize = slot.Length - 8;
if (primaryChunkSize < 0)
return false;
var remainder = newStoredPayload.Slice(primaryChunkSize);
var newOverflowHead = BuildOverflowChainForMigration(remainder);
uint newOverflowHead = BuildOverflowChainForMigration(remainder);
var slotPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
slotPayload.Clear();
@@ -475,22 +477,22 @@ public sealed partial class StorageEngine
if (overflowPayload.IsEmpty)
return 0;
var chunkSize = _pageFile.PageSize - SlottedPageHeader.Size;
int chunkSize = _pageFile.PageSize - SlottedPageHeader.Size;
uint nextOverflowPageId = 0;
var tailSize = overflowPayload.Length % chunkSize;
var fullPages = overflowPayload.Length / chunkSize;
int tailSize = overflowPayload.Length % chunkSize;
int fullPages = overflowPayload.Length / chunkSize;
if (tailSize > 0)
{
var tailOffset = fullPages * chunkSize;
int tailOffset = fullPages * chunkSize;
var tailSlice = overflowPayload.Slice(tailOffset, tailSize);
nextOverflowPageId = AllocateOverflowPageForMigration(tailSlice, nextOverflowPageId);
}
for (var i = fullPages - 1; i >= 0; i--)
for (int i = fullPages - 1; i >= 0; i--)
{
var chunkOffset = i * chunkSize;
int chunkOffset = i * chunkSize;
var chunk = overflowPayload.Slice(chunkOffset, chunkSize);
nextOverflowPageId = AllocateOverflowPageForMigration(chunk, nextOverflowPageId);
}
@@ -500,7 +502,7 @@ public sealed partial class StorageEngine
private uint AllocateOverflowPageForMigration(ReadOnlySpan<byte> payloadChunk, uint nextOverflowPageId)
{
var pageId = _pageFile.AllocatePage();
uint pageId = _pageFile.AllocatePage();
var buffer = new byte[_pageFile.PageSize];
var header = new SlottedPageHeader
@@ -524,13 +526,13 @@ public sealed partial class StorageEngine
{
var buffer = new byte[_pageFile.PageSize];
var visited = new HashSet<uint>();
var current = firstOverflowPage;
uint current = firstOverflowPage;
while (current != 0 && current < _pageFile.NextPageId && visited.Add(current))
{
_pageFile.ReadPage(current, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
var next = header.NextOverflowPage;
uint next = header.NextOverflowPage;
_pageFile.FreePage(current);
current = next;
}

14
src/CBDD.Core/Storage/StorageEngine.Pages.cs
View File

@@ -1,4 +1,4 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
using System.Collections.Concurrent;
namespace ZB.MOM.WW.CBDD.Core.Storage;
@@ -20,17 +20,17 @@ public sealed partial class StorageEngine
if (transactionId.HasValue &&
transactionId.Value != 0 &&
_walCache.TryGetValue(transactionId.Value, out var txnPages) &&
txnPages.TryGetValue(pageId, out var uncommittedData))
txnPages.TryGetValue(pageId, out byte[]? uncommittedData))
{
var length = Math.Min(uncommittedData.Length, destination.Length);
int length = Math.Min(uncommittedData.Length, destination.Length);
uncommittedData.AsSpan(0, length).CopyTo(destination);
return;
}
// 2. Check WAL index (committed but not checkpointed)
if (_walIndex.TryGetValue(pageId, out var committedData))
if (_walIndex.TryGetValue(pageId, out byte[]? committedData))
{
var length = Math.Min(committedData.Length, destination.Length);
int length = Math.Min(committedData.Length, destination.Length);
committedData.AsSpan(0, length).CopyTo(destination);
return;
}
@@ -54,10 +54,10 @@ public sealed partial class StorageEngine
// Get or create transaction-local cache
var txnPages = _walCache.GetOrAdd(transactionId,
_ => new System.Collections.Concurrent.ConcurrentDictionary<uint, byte[]>());
_ => new ConcurrentDictionary<uint, byte[]>());
// Store defensive copy
var copy = data.ToArray();
byte[] copy = data.ToArray();
txnPages[pageId] = copy;
}

59
src/CBDD.Core/Storage/StorageEngine.Recovery.cs
View File

@@ -50,7 +50,7 @@ public sealed partial class StorageEngine
lockAcquired = _commitLock.Wait(0);
if (!lockAcquired)
{
var walSize = _wal.GetCurrentSize();
long walSize = _wal.GetCurrentSize();
return new CheckpointResult(mode, false, 0, walSize, walSize, false, false);
}
}
@@ -66,19 +66,18 @@ public sealed partial class StorageEngine
}
finally
{
if (lockAcquired)
{
_commitLock.Release();
}
if (lockAcquired) _commitLock.Release();
}
}
private void CheckpointInternal()
=> _ = CheckpointInternal(CheckpointMode.Truncate);
{
_ = CheckpointInternal(CheckpointMode.Truncate);
}
private CheckpointResult CheckpointInternal(CheckpointMode mode)
{
var walBytesBefore = _wal.GetCurrentSize();
long walBytesBefore = _wal.GetCurrentSize();
var appliedPages = 0;
var truncated = false;
var restarted = false;
@@ -91,10 +90,7 @@ public sealed partial class StorageEngine
}
// 2. Flush PageFile to ensure durability.
if (appliedPages > 0)
{
_pageFile.Flush();
}
if (appliedPages > 0) _pageFile.Flush();
// 3. Clear in-memory WAL index (now persisted).
_walIndex.Clear();
@@ -109,6 +105,7 @@ public sealed partial class StorageEngine
_wal.WriteCheckpointRecord();
_wal.Flush();
}
break;
case CheckpointMode.Truncate:
if (walBytesBefore > 0)
@@ -116,6 +113,7 @@ public sealed partial class StorageEngine
_wal.Truncate();
truncated = true;
}
break;
case CheckpointMode.Restart:
_wal.Restart();
@@ -126,7 +124,7 @@ public sealed partial class StorageEngine
throw new ArgumentOutOfRangeException(nameof(mode), mode, "Unsupported checkpoint mode.");
}
var walBytesAfter = _wal.GetCurrentSize();
long walBytesAfter = _wal.GetCurrentSize();
return new CheckpointResult(mode, true, appliedPages, walBytesBefore, walBytesAfter, truncated, restarted);
}
@@ -153,7 +151,7 @@ public sealed partial class StorageEngine
lockAcquired = await _commitLock.WaitAsync(0, ct);
if (!lockAcquired)
{
var walSize = _wal.GetCurrentSize();
long walSize = _wal.GetCurrentSize();
return new CheckpointResult(mode, false, 0, walSize, walSize, false, false);
}
}
@@ -170,10 +168,7 @@ public sealed partial class StorageEngine
}
finally
{
if (lockAcquired)
{
_commitLock.Release();
}
if (lockAcquired) _commitLock.Release();
}
}
@@ -189,35 +184,28 @@ public sealed partial class StorageEngine
// 1. Read WAL and locate the latest checkpoint boundary.
var records = _wal.ReadAll();
var startIndex = 0;
for (var i = records.Count - 1; i >= 0; i--)
{
for (int i = records.Count - 1; i >= 0; i--)
if (records[i].Type == WalRecordType.Checkpoint)
{
startIndex = i + 1;
break;
}
}
// 2. Replay WAL in source order with deterministic commit application.
var pendingWrites = new Dictionary<ulong, List<(uint pageId, byte[] data)>>();
var appliedAny = false;
for (var i = startIndex; i < records.Count; i++)
for (int i = startIndex; i < records.Count; i++)
{
var record = records[i];
switch (record.Type)
{
case WalRecordType.Begin:
if (!pendingWrites.ContainsKey(record.TransactionId))
{
pendingWrites[record.TransactionId] = new List<(uint, byte[])>();
}
break;
case WalRecordType.Write:
if (record.AfterImage == null)
{
break;
}
if (record.AfterImage == null) break;
if (!pendingWrites.TryGetValue(record.TransactionId, out var writes))
{
@@ -228,12 +216,9 @@ public sealed partial class StorageEngine
writes.Add((record.PageId, record.AfterImage));
break;
case WalRecordType.Commit:
if (!pendingWrites.TryGetValue(record.TransactionId, out var committedWrites))
{
break;
}
if (!pendingWrites.TryGetValue(record.TransactionId, out var committedWrites)) break;
foreach (var (pageId, data) in committedWrites)
foreach ((uint pageId, byte[] data) in committedWrites)
{
_pageFile.WritePage(pageId, data);
appliedAny = true;
@@ -251,19 +236,13 @@ public sealed partial class StorageEngine
}
// 3. Flush PageFile to ensure durability.
if (appliedAny)
{
_pageFile.Flush();
}
if (appliedAny) _pageFile.Flush();
// 4. Clear in-memory WAL index (redundant since we just recovered).
_walIndex.Clear();
// 5. Truncate WAL (all changes now in PageFile).
if (_wal.GetCurrentSize() > 0)
{
_wal.Truncate();
}
if (_wal.GetCurrentSize() > 0) _wal.Truncate();
}
finally
{

10
src/CBDD.Core/Storage/StorageEngine.Schema.cs
View File

@@ -1,5 +1,3 @@
using System;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Bson.Schema;
@@ -17,7 +15,7 @@ public sealed partial class StorageEngine
var schemas = new List<BsonSchema>();
if (rootPageId == 0) return schemas;
var pageId = rootPageId;
uint pageId = rootPageId;
var buffer = new byte[PageSize];
while (pageId != 0)
@@ -33,7 +31,7 @@ public sealed partial class StorageEngine
var reader = new BsonSpanReader(buffer.AsSpan(32, used), GetKeyReverseMap());
while (reader.Remaining >= 4)
{
var docSize = reader.PeekInt32();
int docSize = reader.PeekInt32();
if (docSize <= 0 || docSize > reader.Remaining) break;
var schema = BsonSchema.FromBson(ref reader);
@@ -60,7 +58,7 @@ public sealed partial class StorageEngine
var tempBuffer = new byte[PageSize];
var tempWriter = new BsonSpanWriter(tempBuffer, GetKeyMap());
schema.ToBson(ref tempWriter);
var schemaSize = tempWriter.Position;
int schemaSize = tempWriter.Position;
if (rootPageId == 0)
{
@@ -106,7 +104,7 @@ public sealed partial class StorageEngine
else
{
// Allocate new page
var newPageId = AllocatePage();
uint newPageId = AllocatePage();
lastHeader.NextPageId = newPageId;
lastHeader.WriteTo(buffer);
WritePageImmediate(lastPageId, buffer);

441
src/CBDD.Core/Storage/StorageEngine.Transactions.cs
View File

@@ -4,6 +4,208 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Gets the number of active transactions (diagnostics).
/// </summary>
public int ActiveTransactionCount => _walCache.Count;
/// <summary>
/// Prepares a transaction: writes all changes to WAL but doesn't commit yet.
/// Part of 2-Phase Commit protocol.
/// </summary>
/// <param name="transactionId">Transaction ID</param>
/// <param name="writeSet">All writes to record in WAL</param>
/// <returns>True if preparation succeeded</returns>
public bool PrepareTransaction(ulong transactionId)
{
try
{
_wal.WriteBeginRecord(transactionId);
foreach (var walEntry in _walCache[transactionId])
_wal.WriteDataRecord(transactionId, walEntry.Key, walEntry.Value);
_wal.Flush(); // Ensure WAL is persisted
return true;
}
catch
{
// TODO: Log error?
return false;
}
}
/// <summary>
/// Prepares a transaction asynchronously by writing pending changes to the WAL.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns><see langword="true" /> if preparation succeeds; otherwise, <see langword="false" />.</returns>
public async Task<bool> PrepareTransactionAsync(ulong transactionId, CancellationToken ct = default)
{
try
{
await _wal.WriteBeginRecordAsync(transactionId, ct);
if (_walCache.TryGetValue(transactionId, out var changes))
foreach (var walEntry in changes)
await _wal.WriteDataRecordAsync(transactionId, walEntry.Key, walEntry.Value, ct);
await _wal.FlushAsync(ct); // Ensure WAL is persisted
return true;
}
catch
{
return false;
}
}
/// <summary>
/// Commits a transaction:
/// 1. Writes all changes to WAL (for durability)
/// 2. Writes commit record
/// 3. Flushes WAL to disk
/// 4. Moves pages from cache to WAL index (for future reads)
/// 5. Clears WAL cache
/// </summary>
/// <param name="transactionId">Transaction to commit</param>
/// <param name="writeSet">All writes performed in this transaction (unused, kept for compatibility)</param>
public void CommitTransaction(ulong transactionId)
{
_commitLock.Wait();
try
{
CommitTransactionCore(transactionId);
}
finally
{
_commitLock.Release();
}
}
private void CommitTransactionCore(ulong transactionId)
{
// Get ALL pages from WAL cache (includes both data and index pages)
if (!_walCache.TryGetValue(transactionId, out var pages))
{
// No writes for this transaction, just write commit record
_wal.WriteCommitRecord(transactionId);
_wal.Flush();
return;
}
// 1. Write all changes to WAL (from cache, not writeSet!)
_wal.WriteBeginRecord(transactionId);
foreach ((uint pageId, byte[] data) in pages) _wal.WriteDataRecord(transactionId, pageId, data);
// 2. Write commit record and flush
_wal.WriteCommitRecord(transactionId);
_wal.Flush(); // Durability: ensure WAL is on disk
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages) _walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize) CheckpointInternal();
}
/// <summary>
/// Commits a prepared transaction asynchronously by identifier.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
public async Task CommitTransactionAsync(ulong transactionId, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
await CommitTransactionCoreAsync(transactionId, ct);
}
finally
{
_commitLock.Release();
}
}
private async Task CommitTransactionCoreAsync(ulong transactionId, CancellationToken ct)
{
// Get ALL pages from WAL cache (includes both data and index pages)
if (!_walCache.TryGetValue(transactionId, out var pages))
{
// No writes for this transaction, just write commit record
await _wal.WriteCommitRecordAsync(transactionId, ct);
await _wal.FlushAsync(ct);
return;
}
// 1. Write all changes to WAL (from cache, not writeSet!)
await _wal.WriteBeginRecordAsync(transactionId, ct);
foreach ((uint pageId, byte[] data) in pages) await _wal.WriteDataRecordAsync(transactionId, pageId, data, ct);
// 2. Write commit record and flush
await _wal.WriteCommitRecordAsync(transactionId, ct);
await _wal.FlushAsync(ct); // Durability: ensure WAL is on disk
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages) _walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
// Checkpoint might be sync or async. For now sync inside the lock is "safe" but blocking.
// Ideally this should be async too.
CheckpointInternal();
}
/// <summary>
/// Marks a transaction as committed after WAL writes.
/// Used for 2PC: after Prepare() writes to WAL, this finalizes the commit.
/// </summary>
/// <param name="transactionId">Transaction to mark committed</param>
public void MarkTransactionCommitted(ulong transactionId)
{
_commitLock.Wait();
try
{
_wal.WriteCommitRecord(transactionId);
_wal.Flush();
// Move from cache to WAL index
if (_walCache.TryRemove(transactionId, out var pages))
foreach (var kvp in pages)
_walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize) CheckpointInternal();
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back a transaction: discards all uncommitted changes.
/// </summary>
/// <param name="transactionId">Transaction to rollback</param>
public void RollbackTransaction(ulong transactionId)
{
_walCache.TryRemove(transactionId, out _);
_wal.WriteAbortRecord(transactionId);
}
/// <summary>
/// Writes an abort record for the specified transaction.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
internal void WriteAbortRecord(ulong transactionId)
{
_wal.WriteAbortRecord(transactionId);
}
#region Transaction Management
/// <summary>
@@ -16,7 +218,7 @@ public sealed partial class StorageEngine
_commitLock.Wait();
try
{
var txnId = _nextTransactionId++;
ulong txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
@@ -33,12 +235,13 @@ public sealed partial class StorageEngine
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The started transaction.</returns>
public async Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted, CancellationToken ct = default)
public async Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted,
CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
var txnId = _nextTransactionId++;
ulong txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
@@ -121,236 +324,4 @@ public sealed partial class StorageEngine
// but for consistency we might consider it. For now, sync is fine as it's not the happy path bottleneck.
#endregion
/// <summary>
/// Prepares a transaction: writes all changes to WAL but doesn't commit yet.
/// Part of 2-Phase Commit protocol.
/// </summary>
/// <param name="transactionId">Transaction ID</param>
/// <param name="writeSet">All writes to record in WAL</param>
/// <returns>True if preparation succeeded</returns>
public bool PrepareTransaction(ulong transactionId)
{
try
{
_wal.WriteBeginRecord(transactionId);
foreach (var walEntry in _walCache[transactionId])
{
_wal.WriteDataRecord(transactionId, walEntry.Key, walEntry.Value);
}
_wal.Flush(); // Ensure WAL is persisted
return true;
}
catch
{
// TODO: Log error?
return false;
}
}
/// <summary>
/// Prepares a transaction asynchronously by writing pending changes to the WAL.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns><see langword="true"/> if preparation succeeds; otherwise, <see langword="false"/>.</returns>
public async Task<bool> PrepareTransactionAsync(ulong transactionId, CancellationToken ct = default)
{
try
{
await _wal.WriteBeginRecordAsync(transactionId, ct);
if (_walCache.TryGetValue(transactionId, out var changes))
{
foreach (var walEntry in changes)
{
await _wal.WriteDataRecordAsync(transactionId, walEntry.Key, walEntry.Value, ct);
}
}
await _wal.FlushAsync(ct); // Ensure WAL is persisted
return true;
}
catch
{
return false;
}
}
/// <summary>
/// Commits a transaction:
/// 1. Writes all changes to WAL (for durability)
/// 2. Writes commit record
/// 3. Flushes WAL to disk
/// 4. Moves pages from cache to WAL index (for future reads)
/// 5. Clears WAL cache
/// </summary>
/// <param name="transactionId">Transaction to commit</param>
/// <param name="writeSet">All writes performed in this transaction (unused, kept for compatibility)</param>
public void CommitTransaction(ulong transactionId)
{
_commitLock.Wait();
try
{
CommitTransactionCore(transactionId);
}
finally
{
_commitLock.Release();
}
}
private void CommitTransactionCore(ulong transactionId)
{
// Get ALL pages from WAL cache (includes both data and index pages)
if (!_walCache.TryGetValue(transactionId, out var pages))
{
// No writes for this transaction, just write commit record
_wal.WriteCommitRecord(transactionId);
_wal.Flush();
return;
}
// 1. Write all changes to WAL (from cache, not writeSet!)
_wal.WriteBeginRecord(transactionId);
foreach (var (pageId, data) in pages)
{
_wal.WriteDataRecord(transactionId, pageId, data);
}
// 2. Write commit record and flush
_wal.WriteCommitRecord(transactionId);
_wal.Flush(); // Durability: ensure WAL is on disk
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
CheckpointInternal();
}
}
/// <summary>
/// Commits a prepared transaction asynchronously by identifier.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
public async Task CommitTransactionAsync(ulong transactionId, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
await CommitTransactionCoreAsync(transactionId, ct);
}
finally
{
_commitLock.Release();
}
}
private async Task CommitTransactionCoreAsync(ulong transactionId, CancellationToken ct)
{
// Get ALL pages from WAL cache (includes both data and index pages)
if (!_walCache.TryGetValue(transactionId, out var pages))
{
// No writes for this transaction, just write commit record
await _wal.WriteCommitRecordAsync(transactionId, ct);
await _wal.FlushAsync(ct);
return;
}
// 1. Write all changes to WAL (from cache, not writeSet!)
await _wal.WriteBeginRecordAsync(transactionId, ct);
foreach (var (pageId, data) in pages)
{
await _wal.WriteDataRecordAsync(transactionId, pageId, data, ct);
}
// 2. Write commit record and flush
await _wal.WriteCommitRecordAsync(transactionId, ct);
await _wal.FlushAsync(ct); // Durability: ensure WAL is on disk
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
// Checkpoint might be sync or async. For now sync inside the lock is "safe" but blocking.
// Ideally this should be async too.
CheckpointInternal();
}
}
/// <summary>
/// Marks a transaction as committed after WAL writes.
/// Used for 2PC: after Prepare() writes to WAL, this finalizes the commit.
/// </summary>
/// <param name="transactionId">Transaction to mark committed</param>
public void MarkTransactionCommitted(ulong transactionId)
{
_commitLock.Wait();
try
{
_wal.WriteCommitRecord(transactionId);
_wal.Flush();
// Move from cache to WAL index
if (_walCache.TryRemove(transactionId, out var pages))
{
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
}
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
CheckpointInternal();
}
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back a transaction: discards all uncommitted changes.
/// </summary>
/// <param name="transactionId">Transaction to rollback</param>
public void RollbackTransaction(ulong transactionId)
{
_walCache.TryRemove(transactionId, out _);
_wal.WriteAbortRecord(transactionId);
}
/// <summary>
/// Writes an abort record for the specified transaction.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
internal void WriteAbortRecord(ulong transactionId)
{
_wal.WriteAbortRecord(transactionId);
}
/// <summary>
/// Gets the number of active transactions (diagnostics).
/// </summary>
public int ActiveTransactionCount => _walCache.Count;
}

131
src/CBDD.Core/Storage/StorageEngine.cs
View File

@@ -1,4 +1,5 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Compression;
using ZB.MOM.WW.CBDD.Core.Transactions;
@@ -6,7 +7,6 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Central storage engine managing page-based storage with WAL for durability.
///
/// Architecture (WAL-based like SQLite/PostgreSQL):
/// - PageFile: Committed baseline (persistent on disk)
/// - WAL Cache: Uncommitted transaction writes (in-memory)
@@ -16,14 +16,15 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
/// </summary>
public sealed partial class StorageEngine : IStorageEngine, IDisposable
{
private const long MaxWalSize = 4 * 1024 * 1024; // 4MB
// Transaction Management
private readonly ConcurrentDictionary<ulong, Transaction> _activeTransactions;
// Global lock for commit/checkpoint synchronization
private readonly SemaphoreSlim _commitLock = new(1, 1);
private readonly PageFile _pageFile;
private readonly WriteAheadLog _wal;
private readonly CompressionOptions _compressionOptions;
private readonly CompressionService _compressionService;
private readonly CompressionTelemetry _compressionTelemetry;
private readonly StorageFormatMetadata _storageFormatMetadata;
private readonly MaintenanceOptions _maintenanceOptions;
private CDC.ChangeStreamDispatcher? _cdc;
// WAL cache: TransactionId → (PageId → PageData)
// Stores uncommitted writes for "Read Your Own Writes" isolation
@@ -32,18 +33,10 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
// WAL index cache: PageId → PageData (from latest committed transaction)
// Lazily populated on first read after commit
private readonly ConcurrentDictionary<uint, byte[]> _walIndex;
// Global lock for commit/checkpoint synchronization
private readonly SemaphoreSlim _commitLock = new(1, 1);
// Transaction Management
private readonly ConcurrentDictionary<ulong, Transaction> _activeTransactions;
private ulong _nextTransactionId;
private const long MaxWalSize = 4 * 1024 * 1024; // 4MB
/// <summary>
/// Initializes a new instance of the <see cref="StorageEngine"/> class.
/// Initializes a new instance of the <see cref="StorageEngine" /> class.
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="config">The page file configuration.</param>
@@ -55,13 +48,13 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
CompressionOptions? compressionOptions = null,
MaintenanceOptions? maintenanceOptions = null)
{
_compressionOptions = CompressionOptions.Normalize(compressionOptions);
_compressionService = new CompressionService();
_compressionTelemetry = new CompressionTelemetry();
_maintenanceOptions = maintenanceOptions ?? new MaintenanceOptions();
CompressionOptions = CompressionOptions.Normalize(compressionOptions);
CompressionService = new CompressionService();
CompressionTelemetry = new CompressionTelemetry();
MaintenanceOptions = maintenanceOptions ?? new MaintenanceOptions();
// Auto-derive WAL path
var walPath = Path.ChangeExtension(databasePath, ".wal");
string walPath = Path.ChangeExtension(databasePath, ".wal");
// Initialize storage infrastructure
_pageFile = new PageFile(databasePath, config);
@@ -72,14 +65,11 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
_walIndex = new ConcurrentDictionary<uint, byte[]>();
_activeTransactions = new ConcurrentDictionary<ulong, Transaction>();
_nextTransactionId = 1;
_storageFormatMetadata = InitializeStorageFormatMetadata();
StorageFormatMetadata = InitializeStorageFormatMetadata();
// Recover from WAL if exists (crash recovery or resume after close)
// This replays any committed transactions not yet checkpointed
if (_wal.GetCurrentSize() > 0)
{
Recover();
}
if (_wal.GetCurrentSize() > 0) Recover();
_ = ResumeCompactionIfNeeded();
@@ -91,35 +81,35 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
// _checkpointManager.StartAutoCheckpoint();
}
/// <summary>
/// Page size for this storage engine
/// </summary>
public int PageSize => _pageFile.PageSize;
/// <summary>
/// Compression options for this engine instance.
/// </summary>
public CompressionOptions CompressionOptions => _compressionOptions;
public CompressionOptions CompressionOptions { get; }
/// <summary>
/// Compression codec service for payload roundtrip operations.
/// </summary>
public CompressionService CompressionService => _compressionService;
public CompressionService CompressionService { get; }
/// <summary>
/// Compression telemetry counters for this engine instance.
/// </summary>
public CompressionTelemetry CompressionTelemetry => _compressionTelemetry;
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats() => _compressionTelemetry.GetSnapshot();
public CompressionTelemetry CompressionTelemetry { get; }
/// <summary>
/// Gets storage format metadata associated with the current database.
/// </summary>
internal StorageFormatMetadata StorageFormatMetadata => _storageFormatMetadata;
internal StorageFormatMetadata StorageFormatMetadata { get; }
/// <summary>
/// Gets the registered change stream dispatcher, if available.
/// </summary>
internal ChangeStreamDispatcher? Cdc { get; private set; }
/// <summary>
/// Page size for this storage engine
/// </summary>
public int PageSize => _pageFile.PageSize;
/// <summary>
/// Checks if a page is currently being modified by another active transaction.
@@ -127,18 +117,19 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
/// </summary>
/// <param name="pageId">The page identifier to check.</param>
/// <param name="excludingTxId">The transaction identifier to exclude from the check.</param>
/// <returns><see langword="true"/> if another transaction holds the page; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if another transaction holds the page; otherwise, <see langword="false" />.</returns>
public bool IsPageLocked(uint pageId, ulong excludingTxId)
{
foreach (var kvp in _walCache)
{
var txId = kvp.Key;
ulong txId = kvp.Key;
if (txId == excludingTxId) continue;
var txnPages = kvp.Value;
if (txnPages.ContainsKey(pageId))
return true;
}
return false;
}
@@ -151,13 +142,15 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
if (_activeTransactions != null)
{
foreach (var txn in _activeTransactions.Values)
{
try
{
RollbackTransaction(txn.TransactionId);
}
catch { /* Ignore errors during dispose */ }
catch
{
/* Ignore errors during dispose */
}
_activeTransactions.Clear();
}
@@ -168,32 +161,38 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
_commitLock?.Dispose();
}
/// <inheritdoc />
void IStorageEngine.RegisterCdc(ChangeStreamDispatcher cdc)
{
RegisterCdc(cdc);
}
/// <inheritdoc />
ChangeStreamDispatcher? IStorageEngine.Cdc => Cdc;
/// <inheritdoc />
CompressionOptions IStorageEngine.CompressionOptions => CompressionOptions;
/// <inheritdoc />
CompressionService IStorageEngine.CompressionService => CompressionService;
/// <inheritdoc />
CompressionTelemetry IStorageEngine.CompressionTelemetry => CompressionTelemetry;
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats()
{
return CompressionTelemetry.GetSnapshot();
}
/// <summary>
/// Registers the change stream dispatcher used for CDC notifications.
/// </summary>
/// <param name="cdc">The change stream dispatcher instance.</param>
internal void RegisterCdc(CDC.ChangeStreamDispatcher cdc)
internal void RegisterCdc(ChangeStreamDispatcher cdc)
{
_cdc = cdc;
Cdc = cdc;
}
/// <summary>
/// Gets the registered change stream dispatcher, if available.
/// </summary>
internal CDC.ChangeStreamDispatcher? Cdc => _cdc;
/// <inheritdoc />
void IStorageEngine.RegisterCdc(CDC.ChangeStreamDispatcher cdc) => RegisterCdc(cdc);
/// <inheritdoc />
CDC.ChangeStreamDispatcher? IStorageEngine.Cdc => _cdc;
/// <inheritdoc />
CompressionOptions IStorageEngine.CompressionOptions => _compressionOptions;
/// <inheritdoc />
CompressionService IStorageEngine.CompressionService => _compressionService;
/// <inheritdoc />
CompressionTelemetry IStorageEngine.CompressionTelemetry => _compressionTelemetry;
}

54
src/CBDD.Core/Storage/VectorPage.cs
View File

@@ -1,5 +1,5 @@
using System.Buffers.Binary;
using System.Runtime.InteropServices;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
@@ -30,7 +30,7 @@ public struct VectorPage
public static void IncrementNodeCount(Span<byte> page)
{
int count = GetNodeCount(page);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), count + 1);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), count + 1);
}
/// <summary>
@@ -52,17 +52,17 @@ public struct VectorPage
};
header.WriteTo(page);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(DimensionsOffset), dimensions);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(MaxMOffset), maxM);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(DimensionsOffset), dimensions);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(MaxMOffset), maxM);
// Node Size Calculation:
// Location (6) + MaxLevel (1) + Vector (dim * 4) + Links (maxM * 10 * 6) -- estimating 10 levels for simplicity
// Better: Node size is variable? No, let's keep it fixed per index configuration to avoid fragmentation.
// HNSW standard: level 0 has 2*M links, levels > 0 have M links.
// Max level is typically < 16. Let's reserve space for 16 levels.
int nodeSize = 6 + 1 + (dimensions * 4) + (maxM * (2 + 15) * 6);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeSizeOffset), nodeSize);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), 0);
int nodeSize = 6 + 1 + dimensions * 4 + maxM * (2 + 15) * 6;
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeSizeOffset), nodeSize);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), 0);
}
/// <summary>
@@ -70,24 +70,30 @@ public struct VectorPage
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The node count.</returns>
public static int GetNodeCount(ReadOnlySpan<byte> page) =>
System.Buffers.Binary.BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeCountOffset));
public static int GetNodeCount(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeCountOffset));
}
/// <summary>
/// Gets the configured node size for the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The node size in bytes.</returns>
public static int GetNodeSize(ReadOnlySpan<byte> page) =>
System.Buffers.Binary.BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeSizeOffset));
public static int GetNodeSize(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeSizeOffset));
}
/// <summary>
/// Gets the maximum number of nodes that can fit in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The maximum node count.</returns>
public static int GetMaxNodes(ReadOnlySpan<byte> page) =>
(page.Length - DataOffset) / GetNodeSize(page);
public static int GetMaxNodes(ReadOnlySpan<byte> page)
{
return (page.Length - DataOffset) / GetNodeSize(page);
}
/// <summary>
/// Writes a node to the page at the specified index.
@@ -98,10 +104,11 @@ public struct VectorPage
/// <param name="maxLevel">The maximum graph level for the node.</param>
/// <param name="vector">The vector values to store.</param>
/// <param name="dimensions">The vector dimensionality.</param>
public static void WriteNode(Span<byte> page, int nodeIndex, DocumentLocation loc, int maxLevel, ReadOnlySpan<float> vector, int dimensions)
public static void WriteNode(Span<byte> page, int nodeIndex, DocumentLocation loc, int maxLevel,
ReadOnlySpan<float> vector, int dimensions)
{
int nodeSize = GetNodeSize(page);
int offset = DataOffset + (nodeIndex * nodeSize);
int offset = DataOffset + nodeIndex * nodeSize;
var nodeSpan = page.Slice(offset, nodeSize);
// 1. Document Location
@@ -127,10 +134,11 @@ public struct VectorPage
/// <param name="loc">When this method returns, contains the node document location.</param>
/// <param name="maxLevel">When this method returns, contains the node max level.</param>
/// <param name="vector">The destination span for vector values.</param>
public static void ReadNodeData(ReadOnlySpan<byte> page, int nodeIndex, out DocumentLocation loc, out int maxLevel, Span<float> vector)
public static void ReadNodeData(ReadOnlySpan<byte> page, int nodeIndex, out DocumentLocation loc, out int maxLevel,
Span<float> vector)
{
int nodeSize = GetNodeSize(page);
int offset = DataOffset + (nodeIndex * nodeSize);
int offset = DataOffset + nodeIndex * nodeSize;
var nodeSpan = page.Slice(offset, nodeSize);
loc = DocumentLocation.ReadFrom(nodeSpan.Slice(0, 6));
@@ -152,23 +160,19 @@ public struct VectorPage
public static Span<byte> GetLinksSpan(Span<byte> page, int nodeIndex, int level, int dimensions, int maxM)
{
int nodeSize = GetNodeSize(page);
int nodeOffset = DataOffset + (nodeIndex * nodeSize);
int nodeOffset = DataOffset + nodeIndex * nodeSize;
// Link offset: Location(6) + MaxLevel(1) + Vector(dim*4)
int linkBaseOffset = nodeOffset + 7 + (dimensions * 4);
int linkBaseOffset = nodeOffset + 7 + dimensions * 4;
int levelOffset;
if (level == 0)
{
levelOffset = 0;
}
else
{
// Level 0 has 2*M links
levelOffset = (2 * maxM * 6) + ((level - 1) * maxM * 6);
}
levelOffset = 2 * maxM * 6 + (level - 1) * maxM * 6;
int count = (level == 0) ? (2 * maxM) : maxM;
int count = level == 0 ? 2 * maxM : maxM;
return page.Slice(linkBaseOffset + levelOffset, count * 6);
}
}

4
src/CBDD.Core/Transactions/CheckpointMode.cs
View File

@@ -20,13 +20,13 @@ public enum CheckpointMode
Full = 1,
/// <summary>
/// Truncate checkpoint: same as <see cref="Full"/> but truncates WAL after
/// Truncate checkpoint: same as <see cref="Full" /> but truncates WAL after
/// successfully applying committed pages. Use this to reclaim disk space.
/// </summary>
Truncate = 2,
/// <summary>
/// Restart checkpoint: same as <see cref="Truncate"/> and then reinitializes
/// Restart checkpoint: same as <see cref="Truncate" /> and then reinitializes
/// the WAL stream for a fresh writer session.
/// </summary>
Restart = 3

4
src/CBDD.Core/Transactions/ITransaction.cs
View File

@@ -1,7 +1,3 @@
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>

20
src/CBDD.Core/Transactions/ITransactionHolder.cs
View File

@@ -3,24 +3,30 @@ namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Defines a contract for managing and providing access to the current transaction context.
/// </summary>
/// <remarks>Implementations of this interface are responsible for tracking the current transaction and starting a
/// <remarks>
/// Implementations of this interface are responsible for tracking the current transaction and starting a
/// new one if none exists. This is typically used in scenarios where transactional consistency is required across
/// multiple operations.</remarks>
/// multiple operations.
/// </remarks>
public interface ITransactionHolder
{
/// <summary>
/// Gets the current transaction if one exists; otherwise, starts a new transaction.
/// </summary>
/// <remarks>Use this method to ensure that a transaction context is available for the current operation.
/// <remarks>
/// Use this method to ensure that a transaction context is available for the current operation.
/// If a transaction is already in progress, it is returned; otherwise, a new transaction is started and returned.
/// The caller is responsible for managing the transaction's lifetime as appropriate.</remarks>
/// <returns>An <see cref="ITransaction"/> representing the current transaction, or a new transaction if none is active.</returns>
/// The caller is responsible for managing the transaction's lifetime as appropriate.
/// </remarks>
/// <returns>An <see cref="ITransaction" /> representing the current transaction, or a new transaction if none is active.</returns>
ITransaction GetCurrentTransactionOrStart();
/// <summary>
/// Gets the current transaction if one exists; otherwise, starts a new transaction asynchronously.
/// </summary>
/// <returns>A task that represents the asynchronous operation. The task result contains an <see cref="ITransaction"/>
/// representing the current or newly started transaction.</returns>
/// <returns>
/// A task that represents the asynchronous operation. The task result contains an <see cref="ITransaction" />
/// representing the current or newly started transaction.
/// </returns>
Task<ITransaction> GetCurrentTransactionOrStartAsync();
}

136
src/CBDD.Core/Transactions/Transaction.cs
View File

@@ -1,8 +1,6 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
@@ -12,12 +10,8 @@ namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// </summary>
public sealed class Transaction : ITransaction
{
private readonly ulong _transactionId;
private readonly IsolationLevel _isolationLevel;
private readonly DateTime _startTime;
private readonly List<InternalChangeEvent> _pendingChanges = new();
private readonly StorageEngine _storage;
private readonly List<CDC.InternalChangeEvent> _pendingChanges = new();
private TransactionState _state;
private bool _disposed;
/// <summary>
@@ -30,41 +24,32 @@ public sealed class Transaction : ITransaction
StorageEngine storage,
IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
{
_transactionId = transactionId;
TransactionId = transactionId;
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_isolationLevel = isolationLevel;
_startTime = DateTime.UtcNow;
_state = TransactionState.Active;
IsolationLevel = isolationLevel;
StartTime = DateTime.UtcNow;
State = TransactionState.Active;
}
/// <summary>
/// Adds a pending CDC change to be published after commit.
/// </summary>
/// <param name="change">The change event to buffer.</param>
internal void AddChange(CDC.InternalChangeEvent change)
{
_pendingChanges.Add(change);
}
/// <summary>
/// Gets the unique transaction identifier.
/// </summary>
public ulong TransactionId => _transactionId;
/// <summary>
/// Gets the current transaction state.
/// </summary>
public TransactionState State => _state;
/// <summary>
/// Gets the configured transaction isolation level.
/// </summary>
public IsolationLevel IsolationLevel => _isolationLevel;
public IsolationLevel IsolationLevel { get; }
/// <summary>
/// Gets the UTC start time of the transaction.
/// </summary>
public DateTime StartTime => _startTime;
public DateTime StartTime { get; }
/// <summary>
/// Gets the unique transaction identifier.
/// </summary>
public ulong TransactionId { get; }
/// <summary>
/// Gets the current transaction state.
/// </summary>
public TransactionState State { get; private set; }
/// <summary>
/// Adds a write operation to the transaction's write set.
@@ -74,13 +59,13 @@ public sealed class Transaction : ITransaction
/// <param name="operation">The write operation to add.</param>
public void AddWrite(WriteOperation operation)
{
if (_state != TransactionState.Active)
throw new InvalidOperationException($"Cannot add writes to transaction in state {_state}");
if (State != TransactionState.Active)
throw new InvalidOperationException($"Cannot add writes to transaction in state {State}");
// Defensive copy: necessary to prevent use-after-return if caller uses pooled buffers
byte[] ownedCopy = operation.NewValue.ToArray();
// StorageEngine gestisce tutte le scritture transazionali
_storage.WritePage(operation.PageId, _transactionId, ownedCopy);
_storage.WritePage(operation.PageId, TransactionId, ownedCopy);
}
/// <summary>
@@ -88,13 +73,13 @@ public sealed class Transaction : ITransaction
/// </summary>
public bool Prepare()
{
if (_state != TransactionState.Active)
if (State != TransactionState.Active)
return false;
_state = TransactionState.Preparing;
State = TransactionState.Preparing;
// StorageEngine handles WAL writes
return _storage.PrepareTransaction(_transactionId);
return _storage.PrepareTransaction(TransactionId);
}
/// <summary>
@@ -104,23 +89,19 @@ public sealed class Transaction : ITransaction
/// </summary>
public void Commit()
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine handles WAL writes and buffer management
_storage.CommitTransaction(_transactionId);
_storage.CommitTransaction(TransactionId);
_state = TransactionState.Committed;
State = TransactionState.Committed;
// Publish CDC events after successful commit
if (_pendingChanges.Count > 0 && _storage.Cdc != null)
{
foreach (var change in _pendingChanges)
{
_storage.Cdc.Publish(change);
}
}
}
/// <summary>
/// Asynchronously commits the transaction.
@@ -128,38 +109,19 @@ public sealed class Transaction : ITransaction
/// <param name="ct">A cancellation token.</param>
public async Task CommitAsync(CancellationToken ct = default)
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine handles WAL writes and buffer management
await _storage.CommitTransactionAsync(_transactionId, ct);
await _storage.CommitTransactionAsync(TransactionId, ct);
_state = TransactionState.Committed;
State = TransactionState.Committed;
// Publish CDC events after successful commit
if (_pendingChanges.Count > 0 && _storage.Cdc != null)
{
foreach (var change in _pendingChanges)
{
_storage.Cdc.Publish(change);
}
}
}
/// <summary>
/// Marks the transaction as committed without writing to PageFile.
/// Used by TransactionManager with lazy checkpointing.
/// </summary>
internal void MarkCommitted()
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
// StorageEngine marks transaction as committed and moves to committed buffer
_storage.MarkTransactionCommitted(_transactionId);
_state = TransactionState.Committed;
}
/// <summary>
/// Rolls back the transaction (discards all writes)
@@ -171,12 +133,12 @@ public sealed class Transaction : ITransaction
/// </summary>
public void Rollback()
{
if (_state == TransactionState.Committed)
if (State == TransactionState.Committed)
throw new InvalidOperationException("Cannot rollback committed transaction");
_pendingChanges.Clear();
_storage.RollbackTransaction(_transactionId);
_state = TransactionState.Aborted;
_storage.RollbackTransaction(TransactionId);
State = TransactionState.Aborted;
OnRollback?.Invoke();
}
@@ -189,15 +151,37 @@ public sealed class Transaction : ITransaction
if (_disposed)
return;
if (_state == TransactionState.Active || _state == TransactionState.Preparing)
{
if (State == TransactionState.Active || State == TransactionState.Preparing)
// Auto-rollback if not committed
Rollback();
}
_disposed = true;
GC.SuppressFinalize(this);
}
/// <summary>
/// Adds a pending CDC change to be published after commit.
/// </summary>
/// <param name="change">The change event to buffer.</param>
internal void AddChange(InternalChangeEvent change)
{
_pendingChanges.Add(change);
}
/// <summary>
/// Marks the transaction as committed without writing to PageFile.
/// Used by TransactionManager with lazy checkpointing.
/// </summary>
internal void MarkCommitted()
{
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine marks transaction as committed and moves to committed buffer
_storage.MarkTransactionCommitted(TransactionId);
State = TransactionState.Committed;
}
}
/// <summary>

116
src/CBDD.Core/Transactions/WriteAheadLog.cs
View File

@@ -1,3 +1,5 @@
using System.Buffers;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
@@ -18,13 +20,13 @@ public enum WalRecordType : byte
/// </summary>
public sealed class WriteAheadLog : IDisposable
{
private readonly string _walPath;
private FileStream? _walStream;
private readonly SemaphoreSlim _lock = new(1, 1);
private readonly string _walPath;
private bool _disposed;
private FileStream? _walStream;
/// <summary>
/// Initializes a new instance of the <see cref="WriteAheadLog"/> class.
/// Initializes a new instance of the <see cref="WriteAheadLog" /> class.
/// </summary>
/// <param name="walPath">The file path of the write-ahead log.</param>
public WriteAheadLog(string walPath)
@@ -36,11 +38,34 @@ public sealed class WriteAheadLog : IDisposable
FileMode.OpenOrCreate,
FileAccess.ReadWrite,
FileShare.None, // Exclusive access like PageFile
bufferSize: 64 * 1024); // 64KB buffer for better sequential write performance
64 * 1024); // 64KB buffer for better sequential write performance
// REMOVED FileOptions.WriteThrough for SQLite-style lazy checkpointing
// Durability is ensured by explicit Flush() calls
}
/// <summary>
/// Releases resources used by the write-ahead log.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
_lock.Wait();
try
{
_walStream?.Dispose();
_disposed = true;
}
finally
{
_lock.Release();
_lock.Dispose();
}
GC.SuppressFinalize(this);
}
/// <summary>
/// Writes a begin transaction record
/// </summary>
@@ -70,7 +95,7 @@ public sealed class WriteAheadLog : IDisposable
try
{
// Use ArrayPool for async I/O compatibility (cannot use stackalloc with async)
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Begin;
@@ -81,7 +106,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -132,7 +157,7 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Commit;
@@ -143,7 +168,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -194,7 +219,7 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Abort;
@@ -205,7 +230,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -251,7 +276,7 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Checkpoint;
@@ -261,7 +286,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -310,15 +335,16 @@ public sealed class WriteAheadLog : IDisposable
/// <param name="afterImage">The page contents after modification.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task that represents the asynchronous write operation.</returns>
public async ValueTask WriteDataRecordAsync(ulong transactionId, uint pageId, ReadOnlyMemory<byte> afterImage, CancellationToken ct = default)
public async ValueTask WriteDataRecordAsync(ulong transactionId, uint pageId, ReadOnlyMemory<byte> afterImage,
CancellationToken ct = default)
{
await _lock.WaitAsync(ct);
try
{
var headerSize = 17;
var totalSize = headerSize + afterImage.Length;
int totalSize = headerSize + afterImage.Length;
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(totalSize);
byte[] buffer = ArrayPool<byte>.Shared.Rent(totalSize);
try
{
buffer[0] = (byte)WalRecordType.Write;
@@ -332,7 +358,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -345,9 +371,9 @@ public sealed class WriteAheadLog : IDisposable
{
// Header: type(1) + txnId(8) + pageId(4) + afterSize(4) = 17 bytes
var headerSize = 17;
var totalSize = headerSize + afterImage.Length;
int totalSize = headerSize + afterImage.Length;
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(totalSize);
byte[] buffer = ArrayPool<byte>.Shared.Rent(totalSize);
try
{
buffer[0] = (byte)WalRecordType.Write;
@@ -361,7 +387,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
@@ -377,7 +403,7 @@ public sealed class WriteAheadLog : IDisposable
_lock.Wait();
try
{
_walStream?.Flush(flushToDisk: true);
_walStream?.Flush(true);
}
finally
{
@@ -395,9 +421,7 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
if (_walStream != null)
{
await _walStream.FlushAsync(ct);
if (_walStream != null) await _walStream.FlushAsync(ct);
// FlushAsync doesn't guarantee flushToDisk on all platforms/implementations in the same way as Flush(true)
// but FileStream in .NET 6+ handles this reasonable well.
// For strict durability, we might still want to invoke a sync flush or check platform specifics,
@@ -407,7 +431,6 @@ public sealed class WriteAheadLog : IDisposable
// To be safe for WAL, we might care about fsync.
// For now, just FlushAsync();
}
}
finally
{
_lock.Release();
@@ -445,7 +468,7 @@ public sealed class WriteAheadLog : IDisposable
{
_walStream.SetLength(0);
_walStream.Position = 0;
_walStream.Flush(flushToDisk: true);
_walStream.Flush(true);
}
}
finally
@@ -468,7 +491,7 @@ public sealed class WriteAheadLog : IDisposable
FileMode.Create,
FileAccess.ReadWrite,
FileShare.None,
bufferSize: 64 * 1024);
64 * 1024);
}
finally
{
@@ -498,17 +521,15 @@ public sealed class WriteAheadLog : IDisposable
while (_walStream.Position < _walStream.Length)
{
var typeByte = _walStream.ReadByte();
int typeByte = _walStream.ReadByte();
if (typeByte == -1) break;
var type = (WalRecordType)typeByte;
// Check for invalid record type (file padding or corruption)
if (typeByte == 0 || !Enum.IsDefined(typeof(WalRecordType), type))
{
// Reached end of valid records (file may have padding)
break;
}
WalRecord record;
@@ -519,14 +540,12 @@ public sealed class WriteAheadLog : IDisposable
case WalRecordType.Abort:
case WalRecordType.Checkpoint:
// Read common fields (txnId + timestamp = 16 bytes)
var bytesRead = _walStream.Read(headerBuf);
int bytesRead = _walStream.Read(headerBuf);
if (bytesRead < 16)
{
// Incomplete record, stop reading
return records;
}
var txnId = BitConverter.ToUInt64(headerBuf[0..8]);
var txnId = BitConverter.ToUInt64(headerBuf[..8]);
var timestamp = BitConverter.ToInt64(headerBuf[8..16]);
record = new WalRecord
@@ -542,30 +561,24 @@ public sealed class WriteAheadLog : IDisposable
// Read txnId + pageId + afterSize = 16 bytes
bytesRead = _walStream.Read(headerBuf);
if (bytesRead < 16)
{
// Incomplete write record header, stop reading
return records;
}
txnId = BitConverter.ToUInt64(headerBuf[0..8]);
txnId = BitConverter.ToUInt64(headerBuf[..8]);
var pageId = BitConverter.ToUInt32(headerBuf[8..12]);
var afterSize = BitConverter.ToInt32(headerBuf[12..16]);
// Validate afterSize to prevent overflow or corruption
if (afterSize < 0 || afterSize > 100 * 1024 * 1024) // Max 100MB per record
{
// Corrupted size, stop reading
return records;
}
var afterImage = new byte[afterSize];
// Read afterImage
if (_walStream.Read(afterImage) < afterSize)
{
// Incomplete after image, stop reading
return records;
}
record = new WalRecord
{
@@ -592,29 +605,6 @@ public sealed class WriteAheadLog : IDisposable
_lock.Release();
}
}
/// <summary>
/// Releases resources used by the write-ahead log.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
_lock.Wait();
try
{
_walStream?.Dispose();
_disposed = true;
}
finally
{
_lock.Release();
_lock.Dispose();
}
GC.SuppressFinalize(this);
}
}
/// <summary>

4
src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj
View File

@@ -22,7 +22,7 @@
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj" />
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj"/>
</ItemGroup>
<ItemGroup>
@@ -32,7 +32,7 @@
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

66
src/CBDD.SourceGenerators/Analysis/EntityAnalyzer.cs
View File

@@ -1,14 +1,13 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
namespace ZB.MOM.WW.CBDD.SourceGenerators
namespace ZB.MOM.WW.CBDD.SourceGenerators;
public static class EntityAnalyzer
{
public static class EntityAnalyzer
{
/// <summary>
/// Analyzes an entity symbol and builds source-generation metadata.
/// </summary>
@@ -28,14 +27,13 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
var tableAttr = AttributeHelper.GetAttribute(entityType, "Table");
if (tableAttr != null)
{
var tableName = tableAttr.ConstructorArguments.Length > 0 ? tableAttr.ConstructorArguments[0].Value?.ToString() : null;
var schema = AttributeHelper.GetNamedArgumentValue(tableAttr, "Schema");
string? tableName = tableAttr.ConstructorArguments.Length > 0
? tableAttr.ConstructorArguments[0].Value?.ToString()
: null;
string? schema = AttributeHelper.GetNamedArgumentValue(tableAttr, "Schema");
var collectionName = !string.IsNullOrEmpty(tableName) ? tableName! : entityInfo.Name;
if (!string.IsNullOrEmpty(schema))
{
collectionName = $"{schema}.{collectionName}";
}
string collectionName = !string.IsNullOrEmpty(tableName) ? tableName! : entityInfo.Name;
if (!string.IsNullOrEmpty(schema)) collectionName = $"{schema}.{collectionName}";
entityInfo.CollectionName = collectionName;
}
@@ -44,10 +42,11 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Check if entity needs reflection-based deserialization
// Include properties with private setters or init-only setters (which can't be set outside initializers)
entityInfo.HasPrivateSetters = entityInfo.Properties.Any(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter);
entityInfo.HasPrivateSetters =
entityInfo.Properties.Any(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter);
// Check if entity has public parameterless constructor
var hasPublicParameterlessConstructor = entityType.Constructors
bool hasPublicParameterlessConstructor = entityType.Constructors
.Any(c => c.DeclaredAccessibility == Accessibility.Public && c.Parameters.Length == 0);
entityInfo.HasPrivateOrNoConstructor = !hasPublicParameterlessConstructor;
@@ -63,20 +62,14 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
{
// Fallback to convention: property named "Id"
var idProp = entityInfo.Properties.FirstOrDefault(p => p.Name == "Id");
if (idProp != null)
{
idProp.IsKey = true;
}
if (idProp != null) idProp.IsKey = true;
}
// Check for AutoId (int/long keys)
if (entityInfo.IdProperty != null)
{
var idType = entityInfo.IdProperty.TypeName.TrimEnd('?');
if (idType == "int" || idType == "Int32" || idType == "long" || idType == "Int64")
{
entityInfo.AutoId = true;
}
string idType = entityInfo.IdProperty.TypeName.TrimEnd('?');
if (idType == "int" || idType == "Int32" || idType == "long" || idType == "Int64") entityInfo.AutoId = true;
}
return entityInfo;
@@ -109,20 +102,22 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
continue;
var columnAttr = AttributeHelper.GetAttribute(prop, "Column");
var bsonFieldName = AttributeHelper.GetAttributeStringValue(prop, "BsonProperty") ??
string? bsonFieldName = AttributeHelper.GetAttributeStringValue(prop, "BsonProperty") ??
AttributeHelper.GetAttributeStringValue(prop, "JsonPropertyName");
if (bsonFieldName == null && columnAttr != null)
{
bsonFieldName = columnAttr.ConstructorArguments.Length > 0 ? columnAttr.ConstructorArguments[0].Value?.ToString() : null;
}
bsonFieldName = columnAttr.ConstructorArguments.Length > 0
? columnAttr.ConstructorArguments[0].Value?.ToString()
: null;
var propInfo = new PropertyInfo
{
Name = prop.Name,
TypeName = SyntaxHelper.GetTypeName(prop.Type),
BsonFieldName = bsonFieldName ?? prop.Name.ToLowerInvariant(),
ColumnTypeName = columnAttr != null ? AttributeHelper.GetNamedArgumentValue(columnAttr, "TypeName") : null,
ColumnTypeName = columnAttr != null
? AttributeHelper.GetNamedArgumentValue(columnAttr, "TypeName")
: null,
IsNullable = SyntaxHelper.IsNullableType(prop.Type),
IsKey = AttributeHelper.IsKey(prop),
IsRequired = AttributeHelper.HasAttribute(prop, "Required"),
@@ -131,7 +126,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
HasInitOnlySetter = prop.SetMethod?.IsInitOnly == true,
HasAnySetter = prop.SetMethod != null,
IsReadOnlyGetter = isReadOnlyGetter,
BackingFieldName = (prop.SetMethod?.DeclaredAccessibility != Accessibility.Public)
BackingFieldName = prop.SetMethod?.DeclaredAccessibility != Accessibility.Public
? $"<{prop.Name}>k__BackingField"
: null
};
@@ -143,11 +138,12 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
var stringLengthAttr = AttributeHelper.GetAttribute(prop, "StringLength");
if (stringLengthAttr != null)
{
if (stringLengthAttr.ConstructorArguments.Length > 0 && stringLengthAttr.ConstructorArguments[0].Value is int max)
if (stringLengthAttr.ConstructorArguments.Length > 0 &&
stringLengthAttr.ConstructorArguments[0].Value is int max)
propInfo.MaxLength = max;
var minLenStr = AttributeHelper.GetNamedArgumentValue(stringLengthAttr, "MinimumLength");
if (int.TryParse(minLenStr, out var min))
string? minLenStr = AttributeHelper.GetNamedArgumentValue(stringLengthAttr, "MinimumLength");
if (int.TryParse(minLenStr, out int min))
propInfo.MinLength = min;
}
@@ -215,8 +211,8 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
foreach (var prop in nestedProps)
{
var fullTypeName = prop.NestedTypeFullName!;
var simpleName = prop.NestedTypeName!;
string fullTypeName = prop.NestedTypeFullName!;
string simpleName = prop.NestedTypeName!;
// Avoid cycles
if (analyzedTypes.Contains(fullTypeName)) continue;
@@ -254,8 +250,8 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
targetNestedTypes[fullTypeName] = nestedInfo;
// Recurse
AnalyzeNestedTypesRecursive(nestedInfo.Properties, nestedInfo.NestedTypes, semanticModel, analyzedTypes, currentDepth + 1, maxDepth);
}
AnalyzeNestedTypesRecursive(nestedInfo.Properties, nestedInfo.NestedTypes, semanticModel, analyzedTypes,
currentDepth + 1, maxDepth);
}
}
}

483
src/CBDD.SourceGenerators/Generators/CodeGenerator.cs
View File

@@ -1,14 +1,12 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Text;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
namespace ZB.MOM.WW.CBDD.SourceGenerators
namespace ZB.MOM.WW.CBDD.SourceGenerators;
public static class CodeGenerator
{
public static class CodeGenerator
{
/// <summary>
/// Generates the mapper class source code for an entity.
/// </summary>
@@ -18,26 +16,27 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
public static string GenerateMapperClass(EntityInfo entity, string mapperNamespace)
{
var sb = new StringBuilder();
var mapperName = GetMapperName(entity.FullTypeName);
string mapperName = GetMapperName(entity.FullTypeName);
var keyProp = entity.Properties.FirstOrDefault(p => p.IsKey);
var isRoot = entity.IdProperty != null;
bool isRoot = entity.IdProperty != null;
sb.AppendLine("#pragma warning disable CS8604");
// Class Declaration
if (isRoot)
{
var baseClass = GetBaseMapperClass(keyProp, entity);
string baseClass = GetBaseMapperClass(keyProp, entity);
// Ensure FullTypeName has global:: prefix if not already present (assuming FullTypeName is fully qualified)
var entityType = $"global::{entity.FullTypeName}";
sb.AppendLine($" public class {mapperName} : global::ZB.MOM.WW.CBDD.Core.Collections.{baseClass}{entityType}>");
sb.AppendLine(
$" public class {mapperName} : global::ZB.MOM.WW.CBDD.Core.Collections.{baseClass}{entityType}>");
}
else
{
sb.AppendLine($" public class {mapperName}");
}
sb.AppendLine($" {{");
sb.AppendLine(" {");
// Converter instance
if (keyProp?.ConverterTypeName != null)
@@ -47,26 +46,34 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
// Generate static setters for private properties (Expression Trees)
var privateSetterProps = entity.Properties.Where(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter).ToList();
var privateSetterProps = entity.Properties
.Where(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter).ToList();
if (privateSetterProps.Any())
{
sb.AppendLine($" // Cached Expression Tree setters for private properties");
sb.AppendLine(" // Cached Expression Tree setters for private properties");
foreach (var prop in privateSetterProps)
{
var entityType = $"global::{entity.FullTypeName}";
var propType = QualifyType(prop.TypeName);
sb.AppendLine($" private static readonly global::System.Action<{entityType}, {propType}> _setter_{prop.Name} = CreateSetter<{entityType}, {propType}>(\"{prop.Name}\");");
string propType = QualifyType(prop.TypeName);
sb.AppendLine(
$" private static readonly global::System.Action<{entityType}, {propType}> _setter_{prop.Name} = CreateSetter<{entityType}, {propType}>(\"{prop.Name}\");");
}
sb.AppendLine();
sb.AppendLine($" private static global::System.Action<TObj, TVal> CreateSetter<TObj, TVal>(string propertyName)");
sb.AppendLine($" {{");
sb.AppendLine($" var param = global::System.Linq.Expressions.Expression.Parameter(typeof(TObj), \"obj\");");
sb.AppendLine($" var value = global::System.Linq.Expressions.Expression.Parameter(typeof(TVal), \"val\");");
sb.AppendLine($" var prop = global::System.Linq.Expressions.Expression.Property(param, propertyName);");
sb.AppendLine($" var assign = global::System.Linq.Expressions.Expression.Assign(prop, value);");
sb.AppendLine($" return global::System.Linq.Expressions.Expression.Lambda<global::System.Action<TObj, TVal>>(assign, param, value).Compile();");
sb.AppendLine($" }}");
sb.AppendLine(
" private static global::System.Action<TObj, TVal> CreateSetter<TObj, TVal>(string propertyName)");
sb.AppendLine(" {");
sb.AppendLine(
" var param = global::System.Linq.Expressions.Expression.Parameter(typeof(TObj), \"obj\");");
sb.AppendLine(
" var value = global::System.Linq.Expressions.Expression.Parameter(typeof(TVal), \"val\");");
sb.AppendLine(
" var prop = global::System.Linq.Expressions.Expression.Property(param, propertyName);");
sb.AppendLine(" var assign = global::System.Linq.Expressions.Expression.Assign(prop, value);");
sb.AppendLine(
" return global::System.Linq.Expressions.Expression.Lambda<global::System.Action<TObj, TVal>>(assign, param, value).Compile();");
sb.AppendLine(" }");
sb.AppendLine();
}
@@ -78,7 +85,8 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
else if (entity.Properties.All(p => !p.IsKey))
{
sb.AppendLine($"// #warning Entity '{entity.Name}' has no defined primary key. Mapper may not support all features.");
sb.AppendLine(
$"// #warning Entity '{entity.Name}' has no defined primary key. Mapper may not support all features.");
}
// Serialize Method
@@ -95,39 +103,41 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
GenerateIdAccessors(sb, entity);
}
sb.AppendLine($" }}");
sb.AppendLine(" }");
sb.AppendLine("#pragma warning restore CS8604");
return sb.ToString();
}
private static void GenerateSerializeMethod(StringBuilder sb, EntityInfo entity, bool isRoot, string mapperNamespace)
private static void GenerateSerializeMethod(StringBuilder sb, EntityInfo entity, bool isRoot,
string mapperNamespace)
{
var entityType = $"global::{entity.FullTypeName}";
// Always generate SerializeFields (writes only fields, no document wrapper)
// This is needed even for root entities, as they may be used as nested objects
// Note: BsonSpanWriter is a ref struct, so it must be passed by ref
sb.AppendLine($" public void SerializeFields({entityType} entity, ref global::ZB.MOM.WW.CBDD.Bson.BsonSpanWriter writer)");
sb.AppendLine($" {{");
sb.AppendLine(
$" public void SerializeFields({entityType} entity, ref global::ZB.MOM.WW.CBDD.Bson.BsonSpanWriter writer)");
sb.AppendLine(" {");
GenerateFieldWritesCore(sb, entity, mapperNamespace);
sb.AppendLine($" }}");
sb.AppendLine(" }");
sb.AppendLine();
// Generate Serialize method (with document wrapper)
var methodSig = isRoot
string methodSig = isRoot
? $"public override int Serialize({entityType} entity, global::ZB.MOM.WW.CBDD.Bson.BsonSpanWriter writer)"
: $"public int Serialize({entityType} entity, global::ZB.MOM.WW.CBDD.Bson.BsonSpanWriter writer)";
sb.AppendLine($" {methodSig}");
sb.AppendLine($" {{");
sb.AppendLine($" var startingPos = writer.BeginDocument();");
sb.AppendLine(" {");
sb.AppendLine(" var startingPos = writer.BeginDocument();");
sb.AppendLine();
GenerateFieldWritesCore(sb, entity, mapperNamespace);
sb.AppendLine();
sb.AppendLine($" writer.EndDocument(startingPos);");
sb.AppendLine($" return writer.Position;");
sb.AppendLine($" }}");
sb.AppendLine(" writer.EndDocument(startingPos);");
sb.AppendLine(" return writer.Position;");
sb.AppendLine(" }");
}
private static void GenerateFieldWritesCore(StringBuilder sb, EntityInfo entity, string mapperNamespace)
@@ -140,37 +150,41 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
if (prop.ConverterTypeName != null)
{
var providerProp = new PropertyInfo { TypeName = prop.ProviderTypeName ?? "string" };
var idWriteMethod = GetPrimitiveWriteMethod(providerProp, allowKey: true);
string? idWriteMethod = GetPrimitiveWriteMethod(providerProp, true);
if (idWriteMethod == "WriteString")
{
sb.AppendLine($" var convertedId = _idConverter.ConvertToProvider(entity.{prop.Name});");
sb.AppendLine($" if (convertedId != null)");
sb.AppendLine($" {{");
sb.AppendLine($" writer.WriteString(\"_id\", convertedId);");
sb.AppendLine($" }}");
sb.AppendLine($" else");
sb.AppendLine($" {{");
sb.AppendLine($" writer.WriteNull(\"_id\");");
sb.AppendLine($" }}");
sb.AppendLine(
$" var convertedId = _idConverter.ConvertToProvider(entity.{prop.Name});");
sb.AppendLine(" if (convertedId != null)");
sb.AppendLine(" {");
sb.AppendLine(" writer.WriteString(\"_id\", convertedId);");
sb.AppendLine(" }");
sb.AppendLine(" else");
sb.AppendLine(" {");
sb.AppendLine(" writer.WriteNull(\"_id\");");
sb.AppendLine(" }");
}
else
{
sb.AppendLine($" writer.{idWriteMethod}(\"_id\", _idConverter.ConvertToProvider(entity.{prop.Name}));");
sb.AppendLine(
$" writer.{idWriteMethod}(\"_id\", _idConverter.ConvertToProvider(entity.{prop.Name}));");
}
}
else
{
var idWriteMethod = GetPrimitiveWriteMethod(prop, allowKey: true);
string? idWriteMethod = GetPrimitiveWriteMethod(prop, true);
if (idWriteMethod != null)
{
sb.AppendLine($" writer.{idWriteMethod}(\"_id\", entity.{prop.Name});");
}
else
{
sb.AppendLine($"#warning Unsupported Id type for '{prop.Name}': {prop.TypeName}. Serialization of '_id' will fail.");
sb.AppendLine(
$"#warning Unsupported Id type for '{prop.Name}': {prop.TypeName}. Serialization of '_id' will fail.");
sb.AppendLine($" // Unsupported Id type: {prop.TypeName}");
}
}
continue;
}
@@ -181,40 +195,37 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
private static void GenerateValidation(StringBuilder sb, PropertyInfo prop)
{
var isString = prop.TypeName == "string" || prop.TypeName == "String";
bool isString = prop.TypeName == "string" || prop.TypeName == "String";
if (prop.IsRequired)
{
if (isString)
{
sb.AppendLine($" if (string.IsNullOrEmpty(entity.{prop.Name})) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is required.\");");
}
sb.AppendLine(
$" if (string.IsNullOrEmpty(entity.{prop.Name})) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is required.\");");
else if (prop.IsNullable)
{
sb.AppendLine($" if (entity.{prop.Name} == null) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is required.\");");
}
sb.AppendLine(
$" if (entity.{prop.Name} == null) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is required.\");");
}
if (prop.MaxLength.HasValue && isString)
{
sb.AppendLine($" if ((entity.{prop.Name}?.Length ?? 0) > {prop.MaxLength}) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} exceeds max length {prop.MaxLength}.\");");
}
sb.AppendLine(
$" if ((entity.{prop.Name}?.Length ?? 0) > {prop.MaxLength}) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} exceeds max length {prop.MaxLength}.\");");
if (prop.MinLength.HasValue && isString)
{
sb.AppendLine($" if ((entity.{prop.Name}?.Length ?? 0) < {prop.MinLength}) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is below min length {prop.MinLength}.\");");
}
sb.AppendLine(
$" if ((entity.{prop.Name}?.Length ?? 0) < {prop.MinLength}) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is below min length {prop.MinLength}.\");");
if (prop.RangeMin.HasValue || prop.RangeMax.HasValue)
{
var minStr = prop.RangeMin?.ToString(System.Globalization.CultureInfo.InvariantCulture) ?? "double.MinValue";
var maxStr = prop.RangeMax?.ToString(System.Globalization.CultureInfo.InvariantCulture) ?? "double.MaxValue";
sb.AppendLine($" if ((double)entity.{prop.Name} < {minStr} || (double)entity.{prop.Name} > {maxStr}) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is outside range [{minStr}, {maxStr}].\");");
string minStr = prop.RangeMin?.ToString(CultureInfo.InvariantCulture) ?? "double.MinValue";
string maxStr = prop.RangeMax?.ToString(CultureInfo.InvariantCulture) ?? "double.MaxValue";
sb.AppendLine(
$" if ((double)entity.{prop.Name} < {minStr} || (double)entity.{prop.Name} > {maxStr}) throw new global::System.ComponentModel.DataAnnotations.ValidationException(\"Property {prop.Name} is outside range [{minStr}, {maxStr}].\");");
}
}
private static void GenerateWriteProperty(StringBuilder sb, PropertyInfo prop, string mapperNamespace)
{
var fieldName = prop.BsonFieldName;
string fieldName = prop.BsonFieldName;
if (prop.IsCollection)
{
@@ -222,11 +233,11 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
if (prop.IsNullable)
{
sb.AppendLine($" if (entity.{prop.Name} != null)");
sb.AppendLine($" {{");
sb.AppendLine(" {");
}
var arrayVar = $"{prop.Name.ToLower()}Array";
var indent = prop.IsNullable ? " " : "";
string indent = prop.IsNullable ? " " : "";
sb.AppendLine($" {indent}var {arrayVar}Pos = writer.BeginArray(\"{fieldName}\");");
sb.AppendLine($" {indent}var {prop.Name.ToLower()}Index = 0;");
sb.AppendLine($" {indent}foreach (var item in entity.{prop.Name})");
@@ -236,23 +247,26 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
if (prop.IsCollectionItemNested)
{
sb.AppendLine($" {indent} // Nested Object in List");
var nestedMapperTypes = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine($" {indent} var {prop.Name.ToLower()}ItemMapper = new global::{mapperNamespace}.{nestedMapperTypes}();");
string nestedMapperTypes = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine(
$" {indent} var {prop.Name.ToLower()}ItemMapper = new global::{mapperNamespace}.{nestedMapperTypes}();");
sb.AppendLine($" {indent} var itemStartPos = writer.BeginDocument({prop.Name.ToLower()}Index.ToString());");
sb.AppendLine($" {indent} {prop.Name.ToLower()}ItemMapper.SerializeFields(item, ref writer);");
sb.AppendLine(
$" {indent} var itemStartPos = writer.BeginDocument({prop.Name.ToLower()}Index.ToString());");
sb.AppendLine(
$" {indent} {prop.Name.ToLower()}ItemMapper.SerializeFields(item, ref writer);");
sb.AppendLine($" {indent} writer.EndDocument(itemStartPos);");
}
else
{
// Simplified: pass a dummy PropertyInfo with the item type for primitive collection items
var dummyProp = new PropertyInfo { TypeName = prop.CollectionItemType! };
var writeMethod = GetPrimitiveWriteMethod(dummyProp);
string? writeMethod = GetPrimitiveWriteMethod(dummyProp);
if (writeMethod != null)
{
sb.AppendLine($" {indent} writer.{writeMethod}({prop.Name.ToLower()}Index.ToString(), item);");
}
sb.AppendLine(
$" {indent} writer.{writeMethod}({prop.Name.ToLower()}Index.ToString(), item);");
}
sb.AppendLine($" {indent} {prop.Name.ToLower()}Index++;");
sb.AppendLine($" {indent}}}");
@@ -261,49 +275,51 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Close the null check if block
if (prop.IsNullable)
{
sb.AppendLine($" }}");
sb.AppendLine($" else");
sb.AppendLine($" {{");
sb.AppendLine(" }");
sb.AppendLine(" else");
sb.AppendLine(" {");
sb.AppendLine($" writer.WriteNull(\"{fieldName}\");");
sb.AppendLine($" }}");
sb.AppendLine(" }");
}
}
else if (prop.IsNestedObject)
{
sb.AppendLine($" if (entity.{prop.Name} != null)");
sb.AppendLine($" {{");
sb.AppendLine(" {");
sb.AppendLine($" var {prop.Name.ToLower()}Pos = writer.BeginDocument(\"{fieldName}\");");
var nestedMapperType = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine($" var {prop.Name.ToLower()}Mapper = new global::{mapperNamespace}.{nestedMapperType}();");
sb.AppendLine($" {prop.Name.ToLower()}Mapper.SerializeFields(entity.{prop.Name}, ref writer);");
string nestedMapperType = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine(
$" var {prop.Name.ToLower()}Mapper = new global::{mapperNamespace}.{nestedMapperType}();");
sb.AppendLine(
$" {prop.Name.ToLower()}Mapper.SerializeFields(entity.{prop.Name}, ref writer);");
sb.AppendLine($" writer.EndDocument({prop.Name.ToLower()}Pos);");
sb.AppendLine($" }}");
sb.AppendLine($" else");
sb.AppendLine($" {{");
sb.AppendLine(" }");
sb.AppendLine(" else");
sb.AppendLine(" {");
sb.AppendLine($" writer.WriteNull(\"{fieldName}\");");
sb.AppendLine($" }}");
sb.AppendLine(" }");
}
else
{
var writeMethod = GetPrimitiveWriteMethod(prop, allowKey: false);
string? writeMethod = GetPrimitiveWriteMethod(prop);
if (writeMethod != null)
{
if (prop.IsNullable || prop.TypeName == "string" || prop.TypeName == "String")
{
sb.AppendLine($" if (entity.{prop.Name} != null)");
sb.AppendLine($" {{");
sb.AppendLine(" {");
// For nullable value types, use .Value to unwrap
// String is a reference type and doesn't need .Value
var isValueTypeNullable = prop.IsNullable && IsValueType(prop.TypeName);
var valueAccess = isValueTypeNullable
bool isValueTypeNullable = prop.IsNullable && IsValueType(prop.TypeName);
string valueAccess = isValueTypeNullable
? $"entity.{prop.Name}.Value"
: $"entity.{prop.Name}";
sb.AppendLine($" writer.{writeMethod}(\"{fieldName}\", {valueAccess});");
sb.AppendLine($" }}");
sb.AppendLine($" else");
sb.AppendLine($" {{");
sb.AppendLine(" }");
sb.AppendLine(" else");
sb.AppendLine(" {");
sb.AppendLine($" writer.WriteNull(\"{fieldName}\");");
sb.AppendLine($" }}");
sb.AppendLine(" }");
}
else
{
@@ -312,16 +328,18 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
else
{
sb.AppendLine($"#warning Property '{prop.Name}' of type '{prop.TypeName}' is not directly supported and has no converter. It will be skipped during serialization.");
sb.AppendLine(
$"#warning Property '{prop.Name}' of type '{prop.TypeName}' is not directly supported and has no converter. It will be skipped during serialization.");
sb.AppendLine($" // Unsupported type: {prop.TypeName} for {prop.Name}");
}
}
}
private static void GenerateDeserializeMethod(StringBuilder sb, EntityInfo entity, bool isRoot, string mapperNamespace)
private static void GenerateDeserializeMethod(StringBuilder sb, EntityInfo entity, bool isRoot,
string mapperNamespace)
{
var entityType = $"global::{entity.FullTypeName}";
var needsReflection = entity.HasPrivateSetters || entity.HasPrivateOrNoConstructor;
bool needsReflection = entity.HasPrivateSetters || entity.HasPrivateOrNoConstructor;
// Always generate a public Deserialize method that accepts ref (for nested/internal usage)
GenerateDeserializeCore(sb, entity, entityType, needsReflection, mapperNamespace);
@@ -330,18 +348,21 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
if (isRoot)
{
sb.AppendLine();
sb.AppendLine($" public override {entityType} Deserialize(global::ZB.MOM.WW.CBDD.Bson.BsonSpanReader reader)");
sb.AppendLine($" {{");
sb.AppendLine($" return Deserialize(ref reader);");
sb.AppendLine($" }}");
sb.AppendLine(
$" public override {entityType} Deserialize(global::ZB.MOM.WW.CBDD.Bson.BsonSpanReader reader)");
sb.AppendLine(" {");
sb.AppendLine(" return Deserialize(ref reader);");
sb.AppendLine(" }");
}
}
private static void GenerateDeserializeCore(StringBuilder sb, EntityInfo entity, string entityType, bool needsReflection, string mapperNamespace)
private static void GenerateDeserializeCore(StringBuilder sb, EntityInfo entity, string entityType,
bool needsReflection, string mapperNamespace)
{
// Public method that always accepts ref for internal/nested usage
sb.AppendLine($" public {entityType} Deserialize(ref global::ZB.MOM.WW.CBDD.Bson.BsonSpanReader reader)");
sb.AppendLine($" {{");
sb.AppendLine(
$" public {entityType} Deserialize(ref global::ZB.MOM.WW.CBDD.Bson.BsonSpanReader reader)");
sb.AppendLine(" {");
// Use object initializer if possible or constructor, but for now standard new()
// To support required properties, we might need a different approach or verify if source generators can detect required.
// For now, let's assume standard creation and property setting.
@@ -351,14 +372,16 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Declare temp variables for all properties
foreach (var prop in entity.Properties)
{
var baseType = QualifyType(prop.TypeName.TrimEnd('?'));
string baseType = QualifyType(prop.TypeName.TrimEnd('?'));
// Handle collections init
if (prop.IsCollection)
{
var itemType = prop.CollectionItemType;
if (prop.IsCollectionItemNested) itemType = $"global::{prop.NestedTypeFullName}"; // Use full name with global::
sb.AppendLine($" var {prop.Name.ToLower()} = new global::System.Collections.Generic.List<{itemType}>();");
string? itemType = prop.CollectionItemType;
if (prop.IsCollectionItemNested)
itemType = $"global::{prop.NestedTypeFullName}"; // Use full name with global::
sb.AppendLine(
$" var {prop.Name.ToLower()} = new global::System.Collections.Generic.List<{itemType}>();");
}
else
{
@@ -368,49 +391,50 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Read document size and track boundaries
sb.AppendLine($" var docSize = reader.ReadDocumentSize();");
sb.AppendLine($" var docEndPos = reader.Position + docSize - 4; // -4 because size includes itself");
sb.AppendLine(" var docSize = reader.ReadDocumentSize();");
sb.AppendLine(" var docEndPos = reader.Position + docSize - 4; // -4 because size includes itself");
sb.AppendLine();
sb.AppendLine($" while (reader.Position < docEndPos)");
sb.AppendLine($" {{");
sb.AppendLine($" var bsonType = reader.ReadBsonType();");
sb.AppendLine($" if (bsonType == global::ZB.MOM.WW.CBDD.Bson.BsonType.EndOfDocument) break;");
sb.AppendLine(" while (reader.Position < docEndPos)");
sb.AppendLine(" {");
sb.AppendLine(" var bsonType = reader.ReadBsonType();");
sb.AppendLine(" if (bsonType == global::ZB.MOM.WW.CBDD.Bson.BsonType.EndOfDocument) break;");
sb.AppendLine();
sb.AppendLine($" var elementName = reader.ReadElementHeader();");
sb.AppendLine($" switch (elementName)");
sb.AppendLine($" {{");
sb.AppendLine(" var elementName = reader.ReadElementHeader();");
sb.AppendLine(" switch (elementName)");
sb.AppendLine(" {");
foreach (var prop in entity.Properties)
{
var caseName = prop.IsKey ? "_id" : prop.BsonFieldName;
string caseName = prop.IsKey ? "_id" : prop.BsonFieldName;
sb.AppendLine($" case \"{caseName}\":");
// Read Logic -> assign to local var
GenerateReadPropertyToLocal(sb, prop, "bsonType", mapperNamespace);
sb.AppendLine($" break;");
sb.AppendLine(" break;");
}
sb.AppendLine($" default:");
sb.AppendLine($" reader.SkipValue(bsonType);");
sb.AppendLine($" break;");
sb.AppendLine($" }}");
sb.AppendLine($" }}");
sb.AppendLine(" default:");
sb.AppendLine(" reader.SkipValue(bsonType);");
sb.AppendLine(" break;");
sb.AppendLine(" }");
sb.AppendLine(" }");
sb.AppendLine();
// Construct object - different approach if needs reflection
if (needsReflection)
{
// Use GetUninitializedObject + Expression Trees for private setters
sb.AppendLine($" // Creating instance without calling constructor (has private members)");
sb.AppendLine($" var entity = (global::{entity.FullTypeName})global::System.Runtime.CompilerServices.RuntimeHelpers.GetUninitializedObject(typeof(global::{entity.FullTypeName}));");
sb.AppendLine(" // Creating instance without calling constructor (has private members)");
sb.AppendLine(
$" var entity = (global::{entity.FullTypeName})global::System.Runtime.CompilerServices.RuntimeHelpers.GetUninitializedObject(typeof(global::{entity.FullTypeName}));");
sb.AppendLine();
// Set properties using setters (Expression Trees for private, direct for public)
foreach (var prop in entity.Properties)
{
var varName = prop.Name.ToLower();
var propValue = varName;
string varName = prop.Name.ToLower();
string propValue = varName;
if (prop.IsCollection)
{
@@ -421,8 +445,10 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
else if (prop.CollectionConcreteTypeName != null)
{
var concreteType = prop.CollectionConcreteTypeName;
var itemType = prop.IsCollectionItemNested ? $"global::{prop.NestedTypeFullName}" : prop.CollectionItemType;
string? concreteType = prop.CollectionConcreteTypeName;
string? itemType = prop.IsCollectionItemNested
? $"global::{prop.NestedTypeFullName}"
: prop.CollectionItemType;
if (concreteType.Contains("HashSet"))
propValue = $"new global::System.Collections.Generic.HashSet<{itemType}>({propValue})";
@@ -441,27 +467,24 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Use appropriate setter
if ((!prop.HasPublicSetter && prop.HasAnySetter) || prop.HasInitOnlySetter)
{
// Use Expression Tree setter (for private or init-only setters)
sb.AppendLine($" _setter_{prop.Name}(entity, {propValue} ?? default!);");
}
else
{
// Direct property assignment
sb.AppendLine($" entity.{prop.Name} = {propValue} ?? default!;");
}
}
sb.AppendLine();
sb.AppendLine($" return entity;");
sb.AppendLine(" return entity;");
}
else
{
// Standard object initializer approach
sb.AppendLine($" return new {entityType}");
sb.AppendLine($" {{");
sb.AppendLine(" {");
foreach (var prop in entity.Properties)
{
var val = prop.Name.ToLower();
string val = prop.Name.ToLower();
if (prop.IsCollection)
{
// Convert to appropriate collection type
@@ -471,128 +494,128 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
else if (prop.CollectionConcreteTypeName != null)
{
var concreteType = prop.CollectionConcreteTypeName;
var itemType = prop.IsCollectionItemNested ? $"global::{prop.NestedTypeFullName}" : prop.CollectionItemType;
string? concreteType = prop.CollectionConcreteTypeName;
string? itemType = prop.IsCollectionItemNested
? $"global::{prop.NestedTypeFullName}"
: prop.CollectionItemType;
// Check if it needs conversion from List
if (concreteType.Contains("HashSet"))
{
val = $"new global::System.Collections.Generic.HashSet<{itemType}>({val})";
}
else if (concreteType.Contains("ISet"))
{
val = $"new global::System.Collections.Generic.HashSet<{itemType}>({val})";
}
else if (concreteType.Contains("LinkedList"))
{
val = $"new global::System.Collections.Generic.LinkedList<{itemType}>({val})";
}
else if (concreteType.Contains("Queue"))
{
val = $"new global::System.Collections.Generic.Queue<{itemType}>({val})";
}
else if (concreteType.Contains("Stack"))
{
val = $"new global::System.Collections.Generic.Stack<{itemType}>({val})";
}
else if (concreteType.Contains("IReadOnlyList") || concreteType.Contains("IReadOnlyCollection"))
{
val += ".AsReadOnly()";
}
// Otherwise keep as List (works for List<T>, IList<T>, ICollection<T>, IEnumerable<T>)
}
}
// For nullable properties, don't use ?? default! since null is a valid value
if (prop.IsNullable)
{
sb.AppendLine($" {prop.Name} = {val},");
}
else
{
sb.AppendLine($" {prop.Name} = {val} ?? default!,");
}
}
sb.AppendLine($" }};");
}
sb.AppendLine($" }}");
sb.AppendLine(" };");
}
private static void GenerateReadPropertyToLocal(StringBuilder sb, PropertyInfo prop, string bsonTypeVar, string mapperNamespace)
sb.AppendLine(" }");
}
private static void GenerateReadPropertyToLocal(StringBuilder sb, PropertyInfo prop, string bsonTypeVar,
string mapperNamespace)
{
var localVar = prop.Name.ToLower();
string localVar = prop.Name.ToLower();
if (prop.IsCollection)
{
var arrVar = prop.Name.ToLower();
string arrVar = prop.Name.ToLower();
sb.AppendLine($" // Read Array {prop.Name}");
sb.AppendLine($" var {arrVar}ArrSize = reader.ReadDocumentSize();");
sb.AppendLine($" var {arrVar}ArrEndPos = reader.Position + {arrVar}ArrSize - 4;");
sb.AppendLine($" while (reader.Position < {arrVar}ArrEndPos)");
sb.AppendLine($" {{");
sb.AppendLine($" var itemType = reader.ReadBsonType();");
sb.AppendLine($" if (itemType == global::ZB.MOM.WW.CBDD.Bson.BsonType.EndOfDocument) break;");
sb.AppendLine($" reader.ReadElementHeader(); // Skip index key");
sb.AppendLine(" {");
sb.AppendLine(" var itemType = reader.ReadBsonType();");
sb.AppendLine(
" if (itemType == global::ZB.MOM.WW.CBDD.Bson.BsonType.EndOfDocument) break;");
sb.AppendLine(" reader.ReadElementHeader(); // Skip index key");
if (prop.IsCollectionItemNested)
{
var nestedMapperTypes = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine($" var {prop.Name.ToLower()}ItemMapper = new global::{mapperNamespace}.{nestedMapperTypes}();");
sb.AppendLine($" var item = {prop.Name.ToLower()}ItemMapper.Deserialize(ref reader);");
string nestedMapperTypes = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine(
$" var {prop.Name.ToLower()}ItemMapper = new global::{mapperNamespace}.{nestedMapperTypes}();");
sb.AppendLine(
$" var item = {prop.Name.ToLower()}ItemMapper.Deserialize(ref reader);");
sb.AppendLine($" {localVar}.Add(item);");
}
else
{
var readMethod = GetPrimitiveReadMethod(new PropertyInfo { TypeName = prop.CollectionItemType! });
string? readMethod = GetPrimitiveReadMethod(new PropertyInfo { TypeName = prop.CollectionItemType! });
if (readMethod != null)
{
var cast = (prop.CollectionItemType == "float" || prop.CollectionItemType == "Single") ? "(float)" : "";
string cast = prop.CollectionItemType == "float" || prop.CollectionItemType == "Single"
? "(float)"
: "";
sb.AppendLine($" var item = {cast}reader.{readMethod}();");
sb.AppendLine($" {localVar}.Add(item);");
}
else
{
sb.AppendLine($" reader.SkipValue(itemType);");
sb.AppendLine(" reader.SkipValue(itemType);");
}
}
sb.AppendLine($" }}");
sb.AppendLine(" }");
}
else if (prop.IsKey && prop.ConverterTypeName != null)
{
var providerProp = new PropertyInfo { TypeName = prop.ProviderTypeName ?? "string" };
var readMethod = GetPrimitiveReadMethod(providerProp);
sb.AppendLine($" {localVar} = _idConverter.ConvertFromProvider(reader.{readMethod}());");
string? readMethod = GetPrimitiveReadMethod(providerProp);
sb.AppendLine(
$" {localVar} = _idConverter.ConvertFromProvider(reader.{readMethod}());");
}
else if (prop.IsNestedObject)
{
sb.AppendLine($" if ({bsonTypeVar} == global::ZB.MOM.WW.CBDD.Bson.BsonType.Null)");
sb.AppendLine($" {{");
sb.AppendLine(" {");
sb.AppendLine($" {localVar} = null;");
sb.AppendLine($" }}");
sb.AppendLine($" else");
sb.AppendLine($" {{");
var nestedMapperType = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine($" var {prop.Name.ToLower()}Mapper = new global::{mapperNamespace}.{nestedMapperType}();");
sb.AppendLine($" {localVar} = {prop.Name.ToLower()}Mapper.Deserialize(ref reader);");
sb.AppendLine($" }}");
sb.AppendLine(" }");
sb.AppendLine(" else");
sb.AppendLine(" {");
string nestedMapperType = GetMapperName(prop.NestedTypeFullName!);
sb.AppendLine(
$" var {prop.Name.ToLower()}Mapper = new global::{mapperNamespace}.{nestedMapperType}();");
sb.AppendLine(
$" {localVar} = {prop.Name.ToLower()}Mapper.Deserialize(ref reader);");
sb.AppendLine(" }");
}
else
{
var readMethod = GetPrimitiveReadMethod(prop);
string? readMethod = GetPrimitiveReadMethod(prop);
if (readMethod != null)
{
var cast = (prop.TypeName == "float" || prop.TypeName == "Single") ? "(float)" : "";
string cast = prop.TypeName == "float" || prop.TypeName == "Single" ? "(float)" : "";
// Handle nullable types - check for null in BSON stream
if (prop.IsNullable)
{
sb.AppendLine($" if ({bsonTypeVar} == global::ZB.MOM.WW.CBDD.Bson.BsonType.Null)");
sb.AppendLine($" {{");
sb.AppendLine(
$" if ({bsonTypeVar} == global::ZB.MOM.WW.CBDD.Bson.BsonType.Null)");
sb.AppendLine(" {");
sb.AppendLine($" {localVar} = null;");
sb.AppendLine($" }}");
sb.AppendLine($" else");
sb.AppendLine($" {{");
sb.AppendLine(" }");
sb.AppendLine(" else");
sb.AppendLine(" {");
sb.AppendLine($" {localVar} = {cast}reader.{readMethod}();");
sb.AppendLine($" }}");
sb.AppendLine(" }");
}
else
{
@@ -615,7 +638,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
{
if (string.IsNullOrEmpty(fullTypeName)) return "UnknownMapper";
// Remove global:: prefix
var cleanName = fullTypeName.Replace("global::", "");
string cleanName = fullTypeName.Replace("global::", "");
// Replace dots, plus (nested classes), and colons (global::) with underscores
return cleanName.Replace(".", "_").Replace("+", "_").Replace(":", "_") + "Mapper";
}
@@ -627,14 +650,10 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Use CollectionIdTypeFullName if available (from DocumentCollection<TId, T> declaration)
string keyType;
if (!string.IsNullOrEmpty(entity.CollectionIdTypeFullName))
{
// Remove "global::" prefix if present
keyType = entity.CollectionIdTypeFullName!.Replace("global::", "");
}
else
{
keyType = keyProp?.TypeName ?? "ObjectId";
}
// Normalize keyType - remove nullable suffix for the methods
// We expect Id to have a value during serialization/deserialization
@@ -655,34 +674,31 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
var entityType = $"global::{entity.FullTypeName}";
var qualifiedKeyType = keyType.StartsWith("global::") ? keyType : (keyProp?.ConverterTypeName != null ? $"global::{keyProp.TypeName.TrimEnd('?')}" : keyType);
string qualifiedKeyType = keyType.StartsWith("global::") ? keyType :
keyProp?.ConverterTypeName != null ? $"global::{keyProp.TypeName.TrimEnd('?')}" : keyType;
var propName = keyProp?.Name ?? "Id";
string propName = keyProp?.Name ?? "Id";
// GetId can return nullable if the property is nullable, but we add ! to assert non-null
// This helps catch bugs where entities are created without an Id
if (keyProp?.IsNullable == true)
{
sb.AppendLine($" public override {qualifiedKeyType} GetId({entityType} entity) => entity.{propName}!;");
}
sb.AppendLine(
$" public override {qualifiedKeyType} GetId({entityType} entity) => entity.{propName}!;");
else
{
sb.AppendLine($" public override {qualifiedKeyType} GetId({entityType} entity) => entity.{propName};");
}
sb.AppendLine(
$" public override {qualifiedKeyType} GetId({entityType} entity) => entity.{propName};");
// If the ID property has a private or init-only setter, use the compiled setter
if (entity.HasPrivateSetters && keyProp != null && (!keyProp.HasPublicSetter || keyProp.HasInitOnlySetter))
{
sb.AppendLine($" public override void SetId({entityType} entity, {qualifiedKeyType} id) => _setter_{propName}(entity, id);");
}
sb.AppendLine(
$" public override void SetId({entityType} entity, {qualifiedKeyType} id) => _setter_{propName}(entity, id);");
else
{
sb.AppendLine($" public override void SetId({entityType} entity, {qualifiedKeyType} id) => entity.{propName} = id;");
}
sb.AppendLine(
$" public override void SetId({entityType} entity, {qualifiedKeyType} id) => entity.{propName} = id;");
if (keyProp?.ConverterTypeName != null)
{
var providerType = keyProp.ProviderTypeName ?? "string";
string providerType = keyProp.ProviderTypeName ?? "string";
// Normalize providerType
switch (providerType)
{
@@ -694,36 +710,32 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
sb.AppendLine();
sb.AppendLine($" public override global::ZB.MOM.WW.CBDD.Core.Indexing.IndexKey ToIndexKey({qualifiedKeyType} id) => ");
sb.AppendLine($" global::ZB.MOM.WW.CBDD.Core.Indexing.IndexKey.Create(_idConverter.ConvertToProvider(id));");
sb.AppendLine(
$" public override global::ZB.MOM.WW.CBDD.Core.Indexing.IndexKey ToIndexKey({qualifiedKeyType} id) => ");
sb.AppendLine(
" global::ZB.MOM.WW.CBDD.Core.Indexing.IndexKey.Create(_idConverter.ConvertToProvider(id));");
sb.AppendLine();
sb.AppendLine($" public override {qualifiedKeyType} FromIndexKey(global::ZB.MOM.WW.CBDD.Core.Indexing.IndexKey key) => ");
sb.AppendLine(
$" public override {qualifiedKeyType} FromIndexKey(global::ZB.MOM.WW.CBDD.Core.Indexing.IndexKey key) => ");
sb.AppendLine($" _idConverter.ConvertFromProvider(key.As<{providerType}>());");
}
}
private static string GetBaseMapperClass(PropertyInfo? keyProp, EntityInfo entity)
{
if (keyProp?.ConverterTypeName != null)
{
return $"DocumentMapperBase<global::{keyProp.TypeName}, ";
}
if (keyProp?.ConverterTypeName != null) return $"DocumentMapperBase<global::{keyProp.TypeName}, ";
// Use CollectionIdTypeFullName if available (from DocumentCollection<TId, T> declaration)
string keyType;
if (!string.IsNullOrEmpty(entity.CollectionIdTypeFullName))
{
// Remove "global::" prefix if present
keyType = entity.CollectionIdTypeFullName!.Replace("global::", "");
}
else
{
keyType = keyProp?.TypeName ?? "ObjectId";
}
// Normalize type by removing nullable suffix (?) for comparison
// At serialization time, we expect the Id to always have a value
var normalizedKeyType = keyType.TrimEnd('?');
string normalizedKeyType = keyType.TrimEnd('?');
if (normalizedKeyType.EndsWith("Int32") || normalizedKeyType == "int") return "Int32MapperBase<";
if (normalizedKeyType.EndsWith("Int64") || normalizedKeyType == "long") return "Int64MapperBase<";
@@ -736,18 +748,14 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
private static string? GetPrimitiveWriteMethod(PropertyInfo prop, bool allowKey = false)
{
var typeName = prop.TypeName;
string typeName = prop.TypeName;
if (prop.ColumnTypeName == "point" || prop.ColumnTypeName == "coordinate" || prop.ColumnTypeName == "geopoint")
{
return "WriteCoordinates";
}
if (typeName.Contains("double") && typeName.Contains(",") && typeName.StartsWith("(") && typeName.EndsWith(")"))
{
return "WriteCoordinates";
}
var cleanType = typeName.TrimEnd('?').Trim();
string cleanType = typeName.TrimEnd('?').Trim();
if (cleanType.EndsWith("Int32") || cleanType == "int") return "WriteInt32";
if (cleanType.EndsWith("Int64") || cleanType == "long") return "WriteInt64";
@@ -769,18 +777,14 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
private static string? GetPrimitiveReadMethod(PropertyInfo prop)
{
var typeName = prop.TypeName;
string typeName = prop.TypeName;
if (prop.ColumnTypeName == "point" || prop.ColumnTypeName == "coordinate" || prop.ColumnTypeName == "geopoint")
{
return "ReadCoordinates";
}
if (typeName.Contains("double") && typeName.Contains(",") && typeName.StartsWith("(") && typeName.EndsWith(")"))
{
return "ReadCoordinates";
}
var cleanType = typeName.TrimEnd('?').Trim();
string cleanType = typeName.TrimEnd('?').Trim();
if (cleanType.EndsWith("Int32") || cleanType == "int") return "ReadInt32";
if (cleanType.EndsWith("Int64") || cleanType == "long") return "ReadInt64";
@@ -804,7 +808,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
{
// Check if the type is a value type (struct) that requires .Value unwrapping when nullable
// String is a reference type and doesn't need .Value
var cleanType = typeName.TrimEnd('?').Trim();
string cleanType = typeName.TrimEnd('?').Trim();
// Common value types
if (cleanType.EndsWith("Int32") || cleanType == "int") return true;
@@ -830,8 +834,8 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
if (string.IsNullOrEmpty(typeName)) return "object";
if (typeName.StartsWith("global::")) return typeName;
var isNullable = typeName.EndsWith("?");
var baseType = typeName.TrimEnd('?').Trim();
bool isNullable = typeName.EndsWith("?");
string baseType = typeName.TrimEnd('?').Trim();
if (baseType.StartsWith("(") && baseType.EndsWith(")")) return typeName; // Tuple
@@ -869,7 +873,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
private static bool IsPrimitive(string typeName)
{
var cleanType = typeName.TrimEnd('?').Trim();
string cleanType = typeName.TrimEnd('?').Trim();
if (cleanType.StartsWith("(") && cleanType.EndsWith(")")) return true;
switch (cleanType)
@@ -894,5 +898,4 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
return false;
}
}
}
}

195
src/CBDD.SourceGenerators/Generators/MapperGenerator.cs
View File

@@ -1,15 +1,17 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
namespace ZB.MOM.WW.CBDD.SourceGenerators
namespace ZB.MOM.WW.CBDD.SourceGenerators;
public class DbContextInfo
{
public class DbContextInfo
{
/// <summary>
/// Gets or sets the simple class name of the DbContext.
/// </summary>
@@ -43,22 +45,23 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
/// <summary>
/// Gets or sets a value indicating whether the DbContext inherits from another DbContext.
/// </summary>
public bool HasBaseDbContext { get; set; } // True if inherits from another DbContext (not DocumentDbContext directly)
public bool
HasBaseDbContext { get; set; } // True if inherits from another DbContext (not DocumentDbContext directly)
/// <summary>
/// Gets or sets the entities discovered for this DbContext.
/// </summary>
public List<EntityInfo> Entities { get; set; } = new List<EntityInfo>();
public List<EntityInfo> Entities { get; set; } = new();
/// <summary>
/// Gets or sets the collected nested types keyed by full type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; set; } = new Dictionary<string, NestedTypeInfo>();
}
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; set; } = new();
}
[Generator]
public class MapperGenerator : IIncrementalGenerator
{
[Generator]
public class MapperGenerator : IIncrementalGenerator
{
/// <summary>
/// Initializes the mapper source generator pipeline.
/// </summary>
@@ -68,8 +71,8 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Find all classes that inherit from DocumentDbContext
var dbContextClasses = context.SyntaxProvider
.CreateSyntaxProvider(
predicate: static (node, _) => IsPotentialDbContext(node),
transform: static (ctx, _) => GetDbContextInfo(ctx))
static (node, _) => IsPotentialDbContext(node),
static (ctx, _) => GetDbContextInfo(ctx))
.Where(static context => context is not null)
.Collect()
.SelectMany(static (contexts, _) => contexts.GroupBy(c => c!.FullClassName).Select(g => g.First())!);
@@ -81,14 +84,13 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
var sb = new StringBuilder();
sb.AppendLine($"// Found DbContext: {dbContext.ClassName}");
sb.AppendLine($"// BaseType: {(dbContext.HasBaseDbContext ? "inherits from another DbContext" : "inherits from DocumentDbContext directly")}");
sb.AppendLine(
$"// BaseType: {(dbContext.HasBaseDbContext ? "inherits from another DbContext" : "inherits from DocumentDbContext directly")}");
foreach (var entity in dbContext.Entities)
{
// Aggregate nested types recursively
CollectNestedTypes(entity.NestedTypes, dbContext.GlobalNestedTypes);
}
// Collect namespaces
var namespaces = new HashSet<string>
@@ -101,169 +103,149 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Add Entity namespaces
foreach (var entity in dbContext.Entities)
{
if (!string.IsNullOrEmpty(entity.Namespace))
namespaces.Add(entity.Namespace);
}
foreach (var nested in dbContext.GlobalNestedTypes.Values)
{
if (!string.IsNullOrEmpty(nested.Namespace))
namespaces.Add(nested.Namespace);
}
// Sanitize file path for name uniqueness
var safeName = dbContext.ClassName;
string safeName = dbContext.ClassName;
if (!string.IsNullOrEmpty(dbContext.FilePath))
{
var fileName = System.IO.Path.GetFileNameWithoutExtension(dbContext.FilePath);
string fileName = Path.GetFileNameWithoutExtension(dbContext.FilePath);
safeName += $"_{fileName}";
}
sb.AppendLine("// <auto-generated/>");
sb.AppendLine("#nullable enable");
foreach (var ns in namespaces.OrderBy(n => n))
{
sb.AppendLine($"using {ns};");
}
foreach (string ns in namespaces.OrderBy(n => n)) sb.AppendLine($"using {ns};");
sb.AppendLine();
// Use safeName (Context + Filename) to avoid collisions
var mapperNamespace = $"{dbContext.Namespace}.{safeName}_Mappers";
sb.AppendLine($"namespace {mapperNamespace}");
sb.AppendLine($"{{");
sb.AppendLine("{");
var generatedMappers = new HashSet<string>();
// Generate Entity Mappers
foreach (var entity in dbContext.Entities)
{
if (generatedMappers.Add(entity.FullTypeName))
{
sb.AppendLine(CodeGenerator.GenerateMapperClass(entity, mapperNamespace));
}
}
// Generate Nested Mappers
foreach (var nested in dbContext.GlobalNestedTypes.Values)
{
if (generatedMappers.Add(nested.FullTypeName))
{
var nestedEntity = new EntityInfo
{
Name = nested.Name,
Namespace = nested.Namespace,
FullTypeName = nested.FullTypeName, // Ensure FullTypeName is copied
FullTypeName = nested.FullTypeName // Ensure FullTypeName is copied
// Helper to copy properties
};
nestedEntity.Properties.AddRange(nested.Properties);
sb.AppendLine(CodeGenerator.GenerateMapperClass(nestedEntity, mapperNamespace));
}
}
sb.AppendLine($"}}");
sb.AppendLine("}");
sb.AppendLine();
// Partial DbContext for InitializeCollections (Only for top-level partial classes)
if (!dbContext.IsNested && dbContext.IsPartial)
{
sb.AppendLine($"namespace {dbContext.Namespace}");
sb.AppendLine($"{{");
sb.AppendLine("{");
sb.AppendLine($" public partial class {dbContext.ClassName}");
sb.AppendLine($" {{");
sb.AppendLine($" protected override void InitializeCollections()");
sb.AppendLine($" {{");
sb.AppendLine(" {");
sb.AppendLine(" protected override void InitializeCollections()");
sb.AppendLine(" {");
// Call base.InitializeCollections() if this context inherits from another DbContext
if (dbContext.HasBaseDbContext)
{
sb.AppendLine($" base.InitializeCollections();");
}
if (dbContext.HasBaseDbContext) sb.AppendLine(" base.InitializeCollections();");
foreach (var entity in dbContext.Entities)
{
if (!string.IsNullOrEmpty(entity.CollectionPropertyName))
{
var mapperName = $"global::{mapperNamespace}.{CodeGenerator.GetMapperName(entity.FullTypeName)}";
sb.AppendLine($" this.{entity.CollectionPropertyName} = CreateCollection(new {mapperName}());");
}
var mapperName =
$"global::{mapperNamespace}.{CodeGenerator.GetMapperName(entity.FullTypeName)}";
sb.AppendLine(
$" this.{entity.CollectionPropertyName} = CreateCollection(new {mapperName}());");
}
sb.AppendLine($" }}");
sb.AppendLine(" }");
sb.AppendLine();
// Generate Set<TId, T>() override
var collectionsWithProperties = dbContext.Entities
.Where(e => !string.IsNullOrEmpty(e.CollectionPropertyName) && !string.IsNullOrEmpty(e.CollectionIdTypeFullName))
.Where(e => !string.IsNullOrEmpty(e.CollectionPropertyName) &&
!string.IsNullOrEmpty(e.CollectionIdTypeFullName))
.ToList();
if (collectionsWithProperties.Any())
{
sb.AppendLine($" public override global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T> Set<TId, T>()");
sb.AppendLine($" {{");
sb.AppendLine(
" public override global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T> Set<TId, T>()");
sb.AppendLine(" {");
foreach (var entity in collectionsWithProperties)
{
var entityTypeStr = $"global::{entity.FullTypeName}";
var idTypeStr = entity.CollectionIdTypeFullName;
sb.AppendLine($" if (typeof(TId) == typeof({idTypeStr}) && typeof(T) == typeof({entityTypeStr}))");
sb.AppendLine($" return (global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T>)(object)this.{entity.CollectionPropertyName};");
string? idTypeStr = entity.CollectionIdTypeFullName;
sb.AppendLine(
$" if (typeof(TId) == typeof({idTypeStr}) && typeof(T) == typeof({entityTypeStr}))");
sb.AppendLine(
$" return (global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T>)(object)this.{entity.CollectionPropertyName};");
}
if (dbContext.HasBaseDbContext)
{
sb.AppendLine($" return base.Set<TId, T>();");
}
sb.AppendLine(" return base.Set<TId, T>();");
else
{
sb.AppendLine($" throw new global::System.InvalidOperationException($\"No collection registered for entity type '{{typeof(T).Name}}' with key type '{{typeof(TId).Name}}'.\");");
sb.AppendLine(
" throw new global::System.InvalidOperationException($\"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.\");");
sb.AppendLine(" }");
}
sb.AppendLine($" }}");
}
sb.AppendLine($" }}");
sb.AppendLine($"}}");
sb.AppendLine(" }");
sb.AppendLine("}");
}
spc.AddSource($"{dbContext.Namespace}.{safeName}.Mappers.g.cs", sb.ToString());
});
}
private static void CollectNestedTypes(Dictionary<string, NestedTypeInfo> source, Dictionary<string, NestedTypeInfo> target)
private static void CollectNestedTypes(Dictionary<string, NestedTypeInfo> source,
Dictionary<string, NestedTypeInfo> target)
{
foreach (var kvp in source)
{
if (!target.ContainsKey(kvp.Value.FullTypeName))
{
target[kvp.Value.FullTypeName] = kvp.Value;
CollectNestedTypes(kvp.Value.NestedTypes, target);
}
}
}
private static void PrintNestedTypes(StringBuilder sb, Dictionary<string, NestedTypeInfo> nestedTypes, string indent)
private static void PrintNestedTypes(StringBuilder sb, Dictionary<string, NestedTypeInfo> nestedTypes,
string indent)
{
foreach (var nt in nestedTypes.Values)
{
sb.AppendLine($"//{indent}- {nt.Name} (Depth: {nt.Depth})");
if (nt.Properties.Count > 0)
{
// Print properties for nested type to be sure
foreach (var p in nt.Properties)
{
var flags = new List<string>();
if (p.IsCollection) flags.Add($"Collection<{p.CollectionItemType}>");
if (p.IsNestedObject) flags.Add($"Nested<{p.NestedTypeName}>");
var flagStr = flags.Any() ? $" [{string.Join(", ", flags)}]" : "";
string flagStr = flags.Any() ? $" [{string.Join(", ", flags)}]" : "";
sb.AppendLine($"//{indent} - {p.Name}: {p.TypeName}{flagStr}");
}
}
if (nt.NestedTypes.Any())
{
PrintNestedTypes(sb, nt.NestedTypes, indent + " ");
}
if (nt.NestedTypes.Any()) PrintNestedTypes(sb, nt.NestedTypes, indent + " ");
}
}
@@ -281,7 +263,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
var classDecl = (ClassDeclarationSyntax)context.Node;
var semanticModel = context.SemanticModel;
var classSymbol = semanticModel.GetDeclaredSymbol(classDecl) as INamedTypeSymbol;
var classSymbol = ModelExtensions.GetDeclaredSymbol(semanticModel, classDecl) as INamedTypeSymbol;
if (classSymbol == null) return null;
if (!SyntaxHelper.InheritsFrom(classSymbol, "DocumentDbContext"))
@@ -299,7 +281,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
Namespace = classSymbol.ContainingNamespace.ToDisplayString(),
FilePath = classDecl.SyntaxTree.FilePath,
IsNested = classSymbol.ContainingType != null,
IsPartial = classDecl.Modifiers.Any(m => m.IsKind(Microsoft.CodeAnalysis.CSharp.SyntaxKind.PartialKeyword)),
IsPartial = classDecl.Modifiers.Any(m => m.IsKind(SyntaxKind.PartialKeyword)),
HasBaseDbContext = hasBaseDbContext
};
@@ -313,7 +295,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
var entityCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "Entity");
foreach (var call in entityCalls)
{
var typeName = SyntaxHelper.GetGenericTypeArgument(call);
string? typeName = SyntaxHelper.GetGenericTypeArgument(call);
if (typeName != null)
{
// Try to find the symbol
@@ -324,15 +306,12 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
entityType = symbols.OfType<INamedTypeSymbol>().FirstOrDefault();
// 2. Try by metadata name (if fully qualified)
if (entityType == null)
{
entityType = semanticModel.Compilation.GetTypeByMetadataName(typeName);
}
if (entityType == null) entityType = semanticModel.Compilation.GetTypeByMetadataName(typeName);
if (entityType != null)
{
// Check for duplicates
var fullTypeName = SyntaxHelper.GetFullName(entityType);
string fullTypeName = SyntaxHelper.GetFullName(entityType);
if (!info.Entities.Any(e => e.FullTypeName == fullTypeName))
{
var entityInfo = EntityAnalyzer.Analyze(entityType, semanticModel);
@@ -349,35 +328,53 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
var conversionCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "HasConversion");
foreach (var call in conversionCalls)
{
var converterName = SyntaxHelper.GetGenericTypeArgument(call);
string? converterName = SyntaxHelper.GetGenericTypeArgument(call);
if (converterName == null) continue;
// Trace back: .Property(x => x.Id).HasConversion<T>() or .HasKey(x => x.Id).HasConversion<T>()
if (call.Expression is MemberAccessExpressionSyntax { Expression: InvocationExpressionSyntax propertyCall } &&
propertyCall.Expression is MemberAccessExpressionSyntax { Name: IdentifierNameSyntax { Identifier: { Text: var propertyMethod } } } &&
if (call.Expression is MemberAccessExpressionSyntax
{
Expression: InvocationExpressionSyntax propertyCall
} &&
propertyCall.Expression is MemberAccessExpressionSyntax
{
Name: IdentifierNameSyntax { Identifier: { Text: var propertyMethod } }
} &&
(propertyMethod == "Property" || propertyMethod == "HasKey"))
{
var propertyName = SyntaxHelper.GetPropertyName(propertyCall.ArgumentList.Arguments.FirstOrDefault()?.Expression);
string? propertyName =
SyntaxHelper.GetPropertyName(propertyCall.ArgumentList.Arguments.FirstOrDefault()?.Expression);
if (propertyName == null) continue;
// Trace further back: Entity<T>().Property(...)
if (propertyCall.Expression is MemberAccessExpressionSyntax { Expression: InvocationExpressionSyntax entityCall } &&
entityCall.Expression is MemberAccessExpressionSyntax { Name: GenericNameSyntax { Identifier: { Text: "Entity" } } })
if (propertyCall.Expression is MemberAccessExpressionSyntax
{
var entityTypeName = SyntaxHelper.GetGenericTypeArgument(entityCall);
Expression: InvocationExpressionSyntax entityCall
} &&
entityCall.Expression is MemberAccessExpressionSyntax
{
Name: GenericNameSyntax { Identifier: { Text: "Entity" } }
})
{
string? entityTypeName = SyntaxHelper.GetGenericTypeArgument(entityCall);
if (entityTypeName != null)
{
var entity = info.Entities.FirstOrDefault(e => e.Name == entityTypeName || e.FullTypeName.EndsWith("." + entityTypeName));
var entity = info.Entities.FirstOrDefault(e =>
e.Name == entityTypeName || e.FullTypeName.EndsWith("." + entityTypeName));
if (entity != null)
{
var prop = entity.Properties.FirstOrDefault(p => p.Name == propertyName);
if (prop != null)
{
// Resolve TProvider from ValueConverter<TModel, TProvider>
var converterType = semanticModel.Compilation.GetTypeByMetadataName(converterName) ??
semanticModel.Compilation.GetSymbolsWithName(converterName).OfType<INamedTypeSymbol>().FirstOrDefault();
var converterType =
semanticModel.Compilation.GetTypeByMetadataName(converterName) ??
semanticModel.Compilation.GetSymbolsWithName(converterName)
.OfType<INamedTypeSymbol>().FirstOrDefault();
prop.ConverterTypeName = converterType != null ? SyntaxHelper.GetFullName(converterType) : converterName;
prop.ConverterTypeName = converterType != null
? SyntaxHelper.GetFullName(converterType)
: converterName;
if (converterType != null && converterType.BaseType != null &&
converterType.BaseType.Name == "ValueConverter" &&
@@ -391,11 +388,13 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
var converterBaseType = converterType.BaseType;
while (converterBaseType != null)
{
if (converterBaseType.Name == "ValueConverter" && converterBaseType.TypeArguments.Length == 2)
if (converterBaseType.Name == "ValueConverter" &&
converterBaseType.TypeArguments.Length == 2)
{
prop.ProviderTypeName = converterBaseType.TypeArguments[1].Name;
break;
}
converterBaseType = converterBaseType.BaseType;
}
}
@@ -410,10 +409,8 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
// Analyze properties to find DocumentCollection<TId, TEntity>
var properties = classSymbol.GetMembers().OfType<IPropertySymbol>();
foreach (var prop in properties)
{
if (prop.Type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.Name == "DocumentCollection")
{
// Expecting 2 type arguments: TId, TEntity
if (namedType.TypeArguments.Length == 2)
{
@@ -424,13 +421,11 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
if (entityInfo != null)
{
entityInfo.CollectionPropertyName = prop.Name;
entityInfo.CollectionIdTypeFullName = namedType.TypeArguments[0].ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
}
}
entityInfo.CollectionIdTypeFullName = namedType.TypeArguments[0]
.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
}
}
return info;
}
}
}

34
src/CBDD.SourceGenerators/Helpers/AttributeHelper.cs
View File

@@ -1,16 +1,15 @@
using System;
using System.Linq;
using Microsoft.CodeAnalysis;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
public static class AttributeHelper
{
public static class AttributeHelper
{
/// <summary>
/// Determines whether a property should be ignored during mapping.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true"/> when the property has an ignore attribute; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when the property has an ignore attribute; otherwise, <see langword="false" />.</returns>
public static bool ShouldIgnore(IPropertySymbol property)
{
return HasAttribute(property, "BsonIgnore") ||
@@ -22,7 +21,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// Determines whether a property is marked as a key.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true"/> when the property has a key attribute; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when the property has a key attribute; otherwise, <see langword="false" />.</returns>
public static bool IsKey(IPropertySymbol property)
{
return HasAttribute(property, "Key") ||
@@ -34,14 +33,11 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static string? GetAttributeStringValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
return attr.ConstructorArguments[0].Value?.ToString();
}
if (attr != null && attr.ConstructorArguments.Length > 0) return attr.ConstructorArguments[0].Value?.ToString();
return null;
}
@@ -51,14 +47,13 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static int? GetAttributeIntValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
if (attr.ConstructorArguments[0].Value is int val) return val;
}
if (attr.ConstructorArguments[0].Value is int val)
return val;
return null;
}
@@ -68,7 +63,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static double? GetAttributeDoubleValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
@@ -87,7 +82,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// </summary>
/// <param name="attr">The attribute data.</param>
/// <param name="name">The named argument key.</param>
/// <returns>The named argument value if present; otherwise, <see langword="null"/>.</returns>
/// <returns>The named argument value if present; otherwise, <see langword="null" />.</returns>
public static string? GetNamedArgumentValue(AttributeData attr, string name)
{
return attr.NamedArguments.FirstOrDefault(a => a.Key == name).Value.Value?.ToString();
@@ -98,7 +93,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The matching attribute data if found; otherwise, <see langword="null"/>.</returns>
/// <returns>The matching attribute data if found; otherwise, <see langword="null" />.</returns>
public static AttributeData? GetAttribute(ISymbol symbol, string attributeName)
{
return symbol.GetAttributes().FirstOrDefault(a =>
@@ -112,10 +107,9 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns><see langword="true"/> when a matching attribute exists; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when a matching attribute exists; otherwise, <see langword="false" />.</returns>
public static bool HasAttribute(ISymbol symbol, string attributeName)
{
return GetAttribute(symbol, attributeName) != null;
}
}
}

68
src/CBDD.SourceGenerators/Helpers/SyntaxHelper.cs
View File

@@ -1,19 +1,18 @@
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
public static class SyntaxHelper
{
public static class SyntaxHelper
{
/// <summary>
/// Determines whether a symbol inherits from a base type with the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="baseTypeName">The base type name to match.</param>
/// <returns><see langword="true"/> if the symbol inherits from the base type; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the symbol inherits from the base type; otherwise, <see langword="false" />.</returns>
public static bool InheritsFrom(INamedTypeSymbol symbol, string baseTypeName)
{
var current = symbol.BaseType;
@@ -23,6 +22,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
return true;
current = current.BaseType;
}
return false;
}
@@ -39,9 +39,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
.Where(invocation =>
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess)
{
return memberAccess.Name.Identifier.Text == methodName;
}
return false;
})
.ToList();
@@ -51,15 +49,13 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// Gets the first generic type argument from an invocation, if present.
/// </summary>
/// <param name="invocation">The invocation to inspect.</param>
/// <returns>The generic type argument text, or <see langword="null"/> when not available.</returns>
/// <returns>The generic type argument text, or <see langword="null" /> when not available.</returns>
public static string? GetGenericTypeArgument(InvocationExpressionSyntax invocation)
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess &&
memberAccess.Name is GenericNameSyntax genericName &&
genericName.TypeArgumentList.Arguments.Count > 0)
{
return genericName.TypeArgumentList.Arguments[0].ToString();
}
return null;
}
@@ -67,26 +63,17 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// Extracts a property name from an expression.
/// </summary>
/// <param name="expression">The expression to analyze.</param>
/// <returns>The property name when resolved; otherwise, <see langword="null"/>.</returns>
/// <returns>The property name when resolved; otherwise, <see langword="null" />.</returns>
public static string? GetPropertyName(ExpressionSyntax? expression)
{
if (expression == null) return null;
if (expression is LambdaExpressionSyntax lambda)
{
return GetPropertyName(lambda.Body as ExpressionSyntax);
}
if (expression is MemberAccessExpressionSyntax memberAccess)
{
return memberAccess.Name.Identifier.Text;
}
if (expression is PrefixUnaryExpressionSyntax prefixUnary && prefixUnary.Operand is MemberAccessExpressionSyntax prefixMember)
{
return prefixMember.Name.Identifier.Text;
}
if (expression is PostfixUnaryExpressionSyntax postfixUnary && postfixUnary.Operand is MemberAccessExpressionSyntax postfixMember)
{
if (expression is LambdaExpressionSyntax lambda) return GetPropertyName(lambda.Body as ExpressionSyntax);
if (expression is MemberAccessExpressionSyntax memberAccess) return memberAccess.Name.Identifier.Text;
if (expression is PrefixUnaryExpressionSyntax prefixUnary &&
prefixUnary.Operand is MemberAccessExpressionSyntax prefixMember) return prefixMember.Name.Identifier.Text;
if (expression is PostfixUnaryExpressionSyntax postfixUnary &&
postfixUnary.Operand is MemberAccessExpressionSyntax postfixMember)
return postfixMember.Name.Identifier.Text;
}
return null;
}
@@ -115,15 +102,9 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
return GetTypeName(underlyingType) + "?";
}
if (type is IArrayTypeSymbol arrayType)
{
return GetTypeName(arrayType.ElementType) + "[]";
}
if (type is IArrayTypeSymbol arrayType) return GetTypeName(arrayType.ElementType) + "[]";
if (type is INamedTypeSymbol nt && nt.IsTupleType)
{
return type.ToDisplayString();
}
if (type is INamedTypeSymbol nt && nt.IsTupleType) return type.ToDisplayString();
return type.ToDisplayString();
}
@@ -132,14 +113,12 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// Determines whether a type is nullable.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is nullable; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the type is nullable; otherwise, <see langword="false" />.</returns>
public static bool IsNullableType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
return true;
}
return type.NullableAnnotation == NullableAnnotation.Annotated;
}
@@ -148,7 +127,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <param name="itemType">When this method returns, contains the collection item type if the type is a collection.</param>
/// <returns><see langword="true"/> if the type is a collection; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the type is a collection; otherwise, <see langword="false" />.</returns>
public static bool IsCollectionType(ITypeSymbol type, out ITypeSymbol? itemType)
{
itemType = null;
@@ -167,7 +146,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
// Check if the type itself is IEnumerable<T>
if (type is INamedTypeSymbol namedType && namedType.IsGenericType)
{
var typeDefName = namedType.OriginalDefinition.ToDisplayString();
string typeDefName = namedType.OriginalDefinition.ToDisplayString();
if (typeDefName == "System.Collections.Generic.IEnumerable<T>" && namedType.TypeArguments.Length == 1)
{
itemType = namedType.TypeArguments[0];
@@ -193,19 +172,17 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// Determines whether a type should be treated as a primitive value.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is primitive-like; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the type is primitive-like; otherwise, <see langword="false" />.</returns>
public static bool IsPrimitiveType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
type = namedType.TypeArguments[0];
}
if (type.SpecialType != SpecialType.None && type.SpecialType != SpecialType.System_Object)
return true;
var typeName = type.Name;
string typeName = type.Name;
if (typeName == "Guid" || typeName == "DateTime" || typeName == "DateTimeOffset" ||
typeName == "TimeSpan" || typeName == "DateOnly" || typeName == "TimeOnly" ||
typeName == "Decimal" || typeName == "ObjectId")
@@ -224,7 +201,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// Determines whether a type should be treated as a nested object.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is a nested object; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the type is a nested object; otherwise, <see langword="false" />.</returns>
public static bool IsNestedObjectType(ITypeSymbol type)
{
if (IsPrimitiveType(type)) return false;
@@ -240,7 +217,7 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
/// Determines whether a property has an associated backing field.
/// </summary>
/// <param name="property">The property to inspect.</param>
/// <returns><see langword="true"/> if a backing field is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if a backing field is found; otherwise, <see langword="false" />.</returns>
public static bool HasBackingField(IPropertySymbol property)
{
// Auto-properties have compiler-generated backing fields
@@ -249,5 +226,4 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
.OfType<IFieldSymbol>()
.Any(f => f.AssociatedSymbol?.Equals(property, SymbolEqualityComparer.Default) == true);
}
}
}

11
src/CBDD.SourceGenerators/Models/DbContextInfo.cs
View File

@@ -1,9 +1,9 @@
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models;
public class DbContextInfo
{
public class DbContextInfo
{
/// <summary>
/// Gets or sets the DbContext class name.
/// </summary>
@@ -22,11 +22,10 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// <summary>
/// Gets the entity types discovered for the DbContext.
/// </summary>
public List<EntityInfo> Entities { get; } = new List<EntityInfo>();
public List<EntityInfo> Entities { get; } = new();
/// <summary>
/// Gets global nested types keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
}
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; } = new();
}

82
src/CBDD.SourceGenerators/Models/EntityInfo.cs
View File

@@ -1,33 +1,38 @@
using System.Collections.Generic;
using System.Linq;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models;
/// <summary>
/// Contains metadata describing an entity discovered by source generation.
/// </summary>
public class EntityInfo
{
/// <summary>
/// Contains metadata describing an entity discovered by source generation.
/// </summary>
public class EntityInfo
{
/// <summary>
/// Gets or sets the entity name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the entity namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified entity type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the collection name for the entity.
/// </summary>
public string CollectionName { get; set; } = "";
/// <summary>
/// Gets or sets the collection property name.
/// </summary>
public string? CollectionPropertyName { get; set; }
/// <summary>
/// Gets or sets the fully qualified collection identifier type name.
/// </summary>
@@ -37,14 +42,17 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// Gets the key property for the entity if one exists.
/// </summary>
public PropertyInfo? IdProperty => Properties.FirstOrDefault(p => p.IsKey);
/// <summary>
/// Gets or sets a value indicating whether IDs are automatically generated.
/// </summary>
public bool AutoId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity uses private setters.
/// </summary>
public bool HasPrivateSetters { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity has a private or missing constructor.
/// </summary>
@@ -53,38 +61,44 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// <summary>
/// Gets the entity properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new List<PropertyInfo>();
public List<PropertyInfo> Properties { get; } = new();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new();
/// <summary>
/// Gets property names that should be ignored by mapping.
/// </summary>
public HashSet<string> IgnoredProperties { get; } = new HashSet<string>();
}
public HashSet<string> IgnoredProperties { get; } = new();
}
/// <summary>
/// Contains metadata describing a mapped property.
/// </summary>
public class PropertyInfo
{
/// <summary>
/// Contains metadata describing a mapped property.
/// </summary>
public class PropertyInfo
{
/// <summary>
/// Gets or sets the property name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the property type name.
/// </summary>
public string TypeName { get; set; } = "";
/// <summary>
/// Gets or sets the BSON field name.
/// </summary>
public string BsonFieldName { get; set; } = "";
/// <summary>
/// Gets or sets the database column type name.
/// </summary>
public string? ColumnTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is nullable.
/// </summary>
@@ -94,18 +108,22 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// Gets or sets a value indicating whether the property has a public setter.
/// </summary>
public bool HasPublicSetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property uses an init-only setter.
/// </summary>
public bool HasInitOnlySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has any setter.
/// </summary>
public bool HasAnySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the getter is read-only.
/// </summary>
public bool IsReadOnlyGetter { get; set; }
/// <summary>
/// Gets or sets the backing field name if available.
/// </summary>
@@ -115,22 +133,27 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// Gets or sets a value indicating whether the property is the key.
/// </summary>
public bool IsKey { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is required.
/// </summary>
public bool IsRequired { get; set; }
/// <summary>
/// Gets or sets the maximum allowed length.
/// </summary>
public int? MaxLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed length.
/// </summary>
public int? MinLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed range value.
/// </summary>
public double? RangeMin { get; set; }
/// <summary>
/// Gets or sets the maximum allowed range value.
/// </summary>
@@ -140,14 +163,17 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// Gets or sets a value indicating whether the property is a collection.
/// </summary>
public bool IsCollection { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is an array.
/// </summary>
public bool IsArray { get; set; }
/// <summary>
/// Gets or sets the collection item type name.
/// </summary>
public string? CollectionItemType { get; set; }
/// <summary>
/// Gets or sets the concrete collection type name.
/// </summary>
@@ -157,45 +183,53 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// Gets or sets a value indicating whether the property is a nested object.
/// </summary>
public bool IsNestedObject { get; set; }
/// <summary>
/// Gets or sets a value indicating whether collection items are nested objects.
/// </summary>
public bool IsCollectionItemNested { get; set; }
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string? NestedTypeName { get; set; }
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string? NestedTypeFullName { get; set; }
/// <summary>
/// Gets or sets the converter type name.
/// </summary>
public string? ConverterTypeName { get; set; }
/// <summary>
/// Gets or sets the provider type name used by the converter.
/// </summary>
public string? ProviderTypeName { get; set; }
}
}
/// <summary>
/// Contains metadata describing a nested type.
/// </summary>
public class NestedTypeInfo
{
/// <summary>
/// Contains metadata describing a nested type.
/// </summary>
public class NestedTypeInfo
{
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the nested type namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the depth of the nested type.
/// </summary>
@@ -204,10 +238,10 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
/// <summary>
/// Gets the nested type properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new List<PropertyInfo>();
public List<PropertyInfo> Properties { get; } = new();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
}
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new();
}

8
src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj
View File

@@ -23,16 +23,16 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.8.0" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.8.0" PrivateAssets="all"/>
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" PrivateAssets="all"/>
</ItemGroup>
<ItemGroup>
<None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
<None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false"/>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

8
src/CBDD/ZB.MOM.WW.CBDD.csproj
View File

@@ -22,13 +22,13 @@
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Core\ZB.MOM.WW.CBDD.Core.csproj" />
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj" />
<ProjectReference Include="..\CBDD.SourceGenerators\ZB.MOM.WW.CBDD.SourceGenerators.csproj" PrivateAssets="none" />
<ProjectReference Include="..\CBDD.Core\ZB.MOM.WW.CBDD.Core.csproj"/>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj"/>
<ProjectReference Include="..\CBDD.SourceGenerators\ZB.MOM.WW.CBDD.SourceGenerators.csproj" PrivateAssets="none"/>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

24
tests/CBDD.Tests.Benchmark/Compaction/CompactionBenchmarks.cs
View File

@@ -1,6 +1,6 @@
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
@@ -15,19 +15,20 @@ namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
[JsonExporterAttribute.Full]
public class CompactionBenchmarks
{
private readonly List<ObjectId> _insertedIds = [];
private DocumentCollection<Person> _collection = null!;
private string _dbPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private string _walPath = string.Empty;
/// <summary>
/// Gets or sets the number of documents used per benchmark iteration.
/// </summary>
[Params(2_000)]
public int DocumentCount { get; set; }
private string _dbPath = string.Empty;
private string _walPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private DocumentCollection<Person> _collection = null!;
private List<ObjectId> _insertedIds = [];
/// <summary>
/// Prepares benchmark state and seed data for each iteration.
/// </summary>
@@ -53,10 +54,7 @@ public class CompactionBenchmarks
_transactionHolder.CommitAndReset();
_storage.Checkpoint();
for (var i = _insertedIds.Count - 1; i >= _insertedIds.Count / 3; i--)
{
_collection.Delete(_insertedIds[i]);
}
for (int i = _insertedIds.Count - 1; i >= _insertedIds.Count / 3; i--) _collection.Delete(_insertedIds[i]);
_transactionHolder.CommitAndReset();
_storage.Checkpoint();
@@ -135,7 +133,7 @@ public class CompactionBenchmarks
private static string BuildPayload(int seed)
{
var builder = new System.Text.StringBuilder(2500);
var builder = new StringBuilder(2500);
for (var i = 0; i < 80; i++)
{
builder.Append("compact-");

35
tests/CBDD.Tests.Benchmark/Compression/CompressionBenchmarks.cs
View File

@@ -1,7 +1,7 @@
using System.IO.Compression;
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using System.IO.Compression;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Compression;
@@ -19,6 +19,15 @@ public class CompressionBenchmarks
{
private const int SeedCount = 300;
private const int WorkloadCount = 100;
private DocumentCollection<Person> _collection = null!;
private string _dbPath = string.Empty;
private Person[] _insertBatch = Array.Empty<Person>();
private ObjectId[] _seedIds = Array.Empty<ObjectId>();
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private string _walPath = string.Empty;
/// <summary>
/// Gets or sets whether compression is enabled for the benchmark run.
@@ -38,15 +47,6 @@ public class CompressionBenchmarks
[Params(CompressionLevel.Fastest, CompressionLevel.Optimal)]
public CompressionLevel Level { get; set; }
private string _dbPath = string.Empty;
private string _walPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private DocumentCollection<Person> _collection = null!;
private Person[] _insertBatch = Array.Empty<Person>();
private ObjectId[] _seedIds = Array.Empty<ObjectId>();
/// <summary>
/// Prepares benchmark storage and seed data for each iteration.
/// </summary>
@@ -73,14 +73,14 @@ public class CompressionBenchmarks
_seedIds = new ObjectId[SeedCount];
for (var i = 0; i < SeedCount; i++)
{
var doc = CreatePerson(i, includeLargeBio: true);
var doc = CreatePerson(i, true);
_seedIds[i] = _collection.Insert(doc);
}
_transactionHolder.CommitAndReset();
_insertBatch = Enumerable.Range(SeedCount, WorkloadCount)
.Select(i => CreatePerson(i, includeLargeBio: true))
.Select(i => CreatePerson(i, true))
.ToArray();
}
@@ -141,10 +141,7 @@ public class CompressionBenchmarks
for (var i = 0; i < WorkloadCount; i++)
{
var person = _collection.FindById(_seedIds[i]);
if (person != null)
{
checksum += person.Age;
}
if (person != null) checksum += person.Age;
}
_transactionHolder.CommitAndReset();
@@ -158,7 +155,7 @@ public class CompressionBenchmarks
Id = ObjectId.NewObjectId(),
FirstName = $"First_{i}",
LastName = $"Last_{i}",
Age = 20 + (i % 50),
Age = 20 + i % 50,
Bio = includeLargeBio ? BuildBio(i) : $"bio-{i}",
CreatedAt = DateTime.UnixEpoch.AddMinutes(i),
Balance = 100 + i,
@@ -183,7 +180,7 @@ public class CompressionBenchmarks
private static string BuildBio(int seed)
{
var builder = new System.Text.StringBuilder(4500);
var builder = new StringBuilder(4500);
for (var i = 0; i < 150; i++)
{
builder.Append("bio-");

14
tests/CBDD.Tests.Benchmark/Fixtures/Person.cs
View File

@@ -1,6 +1,4 @@
using ZB.MOM.WW.CBDD.Bson;
using System;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
@@ -10,10 +8,12 @@ public class Address
/// Gets or sets the Street.
/// </summary>
public string Street { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the City.
/// </summary>
public string City { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the ZipCode.
/// </summary>
@@ -26,14 +26,17 @@ public class WorkHistory
/// Gets or sets the CompanyName.
/// </summary>
public string CompanyName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the Title.
/// </summary>
public string Title { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the DurationYears.
/// </summary>
public int DurationYears { get; set; }
/// <summary>
/// Gets or sets the Tags.
/// </summary>
@@ -46,22 +49,27 @@ public class Person
/// Gets or sets the Id.
/// </summary>
public ObjectId Id { get; set; }
/// <summary>
/// Gets or sets the FirstName.
/// </summary>
public string FirstName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the LastName.
/// </summary>
public string LastName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the Age.
/// </summary>
public int Age { get; set; }
/// <summary>
/// Gets or sets the Bio.
/// </summary>
public string? Bio { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the CreatedAt.
/// </summary>
@@ -72,10 +80,12 @@ public class Person
/// Gets or sets the Balance.
/// </summary>
public decimal Balance { get; set; }
/// <summary>
/// Gets or sets the HomeAddress.
/// </summary>
public Address HomeAddress { get; set; } = new();
/// <summary>
/// Gets or sets the EmploymentHistory.
/// </summary>

59
tests/CBDD.Tests.Benchmark/Fixtures/PersonMapper.cs
View File

@@ -1,7 +1,5 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using System.Buffers;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
@@ -11,15 +9,21 @@ public class PersonMapper : ObjectIdMapperBase<Person>
public override string CollectionName => "people";
/// <inheritdoc />
public override ObjectId GetId(Person entity) => entity.Id;
public override ObjectId GetId(Person entity)
{
return entity.Id;
}
/// <inheritdoc />
public override void SetId(Person entity, ObjectId id) => entity.Id = id;
public override void SetId(Person entity, ObjectId id)
{
entity.Id = id;
}
/// <inheritdoc />
public override int Serialize(Person entity, BsonSpanWriter writer)
{
var sizePos = writer.BeginDocument();
int sizePos = writer.BeginDocument();
writer.WriteObjectId("_id", entity.Id);
writer.WriteString("firstname", entity.FirstName);
@@ -36,34 +40,32 @@ public class PersonMapper : ObjectIdMapperBase<Person>
writer.WriteDouble("balance", (double)entity.Balance);
// Nested Object: Address
var addrPos = writer.BeginDocument("homeaddress");
int addrPos = writer.BeginDocument("homeaddress");
writer.WriteString("street", entity.HomeAddress.Street);
writer.WriteString("city", entity.HomeAddress.City);
writer.WriteString("zipcode", entity.HomeAddress.ZipCode);
writer.EndDocument(addrPos);
// Collection: EmploymentHistory
var histPos = writer.BeginArray("employmenthistory");
for (int i = 0; i < entity.EmploymentHistory.Count; i++)
int histPos = writer.BeginArray("employmenthistory");
for (var i = 0; i < entity.EmploymentHistory.Count; i++)
{
var item = entity.EmploymentHistory[i];
// Array elements are keys "0", "1", "2"...
var itemPos = writer.BeginDocument(i.ToString());
int itemPos = writer.BeginDocument(i.ToString());
writer.WriteString("companyname", item.CompanyName);
writer.WriteString("title", item.Title);
writer.WriteInt32("durationyears", item.DurationYears);
// Nested Collection: Tags
var tagsPos = writer.BeginArray("tags");
for (int j = 0; j < item.Tags.Count; j++)
{
writer.WriteString(j.ToString(), item.Tags[j]);
}
int tagsPos = writer.BeginArray("tags");
for (var j = 0; j < item.Tags.Count; j++) writer.WriteString(j.ToString(), item.Tags[j]);
writer.EndArray(tagsPos);
writer.EndDocument(itemPos);
}
writer.EndArray(histPos);
writer.EndDocument(sizePos);
@@ -84,7 +86,7 @@ public class PersonMapper : ObjectIdMapperBase<Person>
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
switch (name)
{
@@ -105,7 +107,7 @@ public class PersonMapper : ObjectIdMapperBase<Person>
{
var addrType = reader.ReadBsonType();
if (addrType == BsonType.EndOfDocument) break;
var addrName = reader.ReadElementHeader();
string addrName = reader.ReadElementHeader();
// We assume strict schema for benchmark speed, but should handle skipping
if (addrName == "street") person.HomeAddress.Street = reader.ReadString();
@@ -113,6 +115,7 @@ public class PersonMapper : ObjectIdMapperBase<Person>
else if (addrName == "zipcode") person.HomeAddress.ZipCode = reader.ReadString();
else reader.SkipValue(addrType);
}
break;
case "employmenthistory":
@@ -130,11 +133,20 @@ public class PersonMapper : ObjectIdMapperBase<Person>
{
var itemType = reader.ReadBsonType();
if (itemType == BsonType.EndOfDocument) break;
var itemName = reader.ReadElementHeader();
string itemName = reader.ReadElementHeader();
if (itemName == "companyname") workItem.CompanyName = reader.ReadString();
else if (itemName == "title") workItem.Title = reader.ReadString();
else if (itemName == "durationyears") workItem.DurationYears = reader.ReadInt32();
if (itemName == "companyname")
{
workItem.CompanyName = reader.ReadString();
}
else if (itemName == "title")
{
workItem.Title = reader.ReadString();
}
else if (itemName == "durationyears")
{
workItem.DurationYears = reader.ReadInt32();
}
else if (itemName == "tags")
{
reader.ReadDocumentSize(); // Enter Tags Array
@@ -149,10 +161,15 @@ public class PersonMapper : ObjectIdMapperBase<Person>
reader.SkipValue(tagType);
}
}
else reader.SkipValue(itemType);
else
{
reader.SkipValue(itemType);
}
}
person.EmploymentHistory.Add(workItem);
}
break;
default:

35
tests/CBDD.Tests.Benchmark/Infrastructure/BenchmarkTransactionHolder.cs
View File

@@ -1,4 +1,3 @@
using ZB.MOM.WW.CBDD.Core;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
@@ -11,7 +10,7 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
private ITransaction? _currentTransaction;
/// <summary>
/// Initializes a new instance of the <see cref="BenchmarkTransactionHolder"/> class.
/// Initializes a new instance of the <see cref="BenchmarkTransactionHolder" /> class.
/// </summary>
/// <param name="storage">The storage engine used to create transactions.</param>
public BenchmarkTransactionHolder(StorageEngine storage)
@@ -19,6 +18,14 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
}
/// <summary>
/// Disposes this holder and rolls back any outstanding transaction.
/// </summary>
public void Dispose()
{
RollbackAndReset();
}
/// <summary>
/// Gets the current active transaction or starts a new one.
/// </summary>
@@ -28,9 +35,7 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
lock (_sync)
{
if (_currentTransaction == null || _currentTransaction.State != TransactionState.Active)
{
_currentTransaction = _storage.BeginTransaction();
}
return _currentTransaction;
}
@@ -52,16 +57,11 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
{
lock (_sync)
{
if (_currentTransaction == null)
{
return;
}
if (_currentTransaction == null) return;
if (_currentTransaction.State == TransactionState.Active ||
_currentTransaction.State == TransactionState.Preparing)
{
_currentTransaction.Commit();
}
_currentTransaction.Dispose();
_currentTransaction = null;
@@ -75,27 +75,14 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
{
lock (_sync)
{
if (_currentTransaction == null)
{
return;
}
if (_currentTransaction == null) return;
if (_currentTransaction.State == TransactionState.Active ||
_currentTransaction.State == TransactionState.Preparing)
{
_currentTransaction.Rollback();
}
_currentTransaction.Dispose();
_currentTransaction = null;
}
}
/// <summary>
/// Disposes this holder and rolls back any outstanding transaction.
/// </summary>
public void Dispose()
{
RollbackAndReset();
}
}

7
tests/CBDD.Tests.Benchmark/Infrastructure/Logging.cs
View File

@@ -1,5 +1,6 @@
using Microsoft.Extensions.Logging;
using Serilog;
using ILogger = Microsoft.Extensions.Logging.ILogger;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
@@ -16,8 +17,8 @@ internal static class Logging
/// Creates a logger for the specified category type.
/// </summary>
/// <typeparam name="T">The logger category type.</typeparam>
/// <returns>A logger for <typeparamref name="T"/>.</returns>
public static Microsoft.Extensions.Logging.ILogger CreateLogger<T>()
/// <returns>A logger for <typeparamref name="T" />.</returns>
public static ILogger CreateLogger<T>()
{
return LoggerFactory.CreateLogger<T>();
}
@@ -32,7 +33,7 @@ internal static class Logging
return Microsoft.Extensions.Logging.LoggerFactory.Create(builder =>
{
builder.ClearProviders();
builder.AddSerilog(serilogLogger, dispose: true);
builder.AddSerilog(serilogLogger, true);
});
}
}

10
tests/CBDD.Tests.Benchmark/Infrastructure/Program.cs
View File

@@ -3,17 +3,17 @@ using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Exporters;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;
using Microsoft.Extensions.Logging;
using Perfolizer.Horology;
using Serilog.Context;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
class Program
internal class Program
{
static void Main(string[] args)
private static void Main(string[] args)
{
var logger = Logging.CreateLogger<Program>();
var mode = args.Length > 0 ? args[0].Trim().ToLowerInvariant() : string.Empty;
string mode = args.Length > 0 ? args[0].Trim().ToLowerInvariant() : string.Empty;
if (mode == "manual")
{
@@ -84,6 +84,6 @@ class Program
.AddExporter(HtmlExporter.Default)
.WithSummaryStyle(SummaryStyle.Default
.WithRatioStyle(RatioStyle.Trend)
.WithTimeUnit(Perfolizer.Horology.TimeUnit.Microsecond));
.WithTimeUnit(TimeUnit.Microsecond));
}
}

38
tests/CBDD.Tests.Benchmark/Insert/InsertBenchmarks.cs
View File

@@ -1,18 +1,13 @@
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using Microsoft.Extensions.Logging;
using Serilog.Context;
using System.IO;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
[InProcess]
[MemoryDiagnoser]
[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]
@@ -23,15 +18,15 @@ public class InsertBenchmarks
private const int BatchSize = 1000;
private static readonly ILogger Logger = Logging.CreateLogger<InsertBenchmarks>();
private Person[] _batchData = Array.Empty<Person>();
private DocumentCollection<Person>? _collection;
private string _docDbPath = "";
private string _docDbWalPath = "";
private Person? _singlePerson;
private StorageEngine? _storage = null;
private BenchmarkTransactionHolder? _transactionHolder = null;
private DocumentCollection<Person>? _collection = null;
private Person[] _batchData = Array.Empty<Person>();
private Person? _singlePerson = null;
private StorageEngine? _storage;
private BenchmarkTransactionHolder? _transactionHolder;
/// <summary>
/// Tests setup.
@@ -39,17 +34,14 @@ public class InsertBenchmarks
[GlobalSetup]
public void Setup()
{
var temp = AppContext.BaseDirectory;
string temp = AppContext.BaseDirectory;
var id = Guid.NewGuid().ToString("N");
_docDbPath = Path.Combine(temp, $"bench_docdb_{id}.db");
_docDbWalPath = Path.ChangeExtension(_docDbPath, ".wal");
_singlePerson = CreatePerson(0);
_batchData = new Person[BatchSize];
for (int i = 0; i < BatchSize; i++)
{
_batchData[i] = CreatePerson(i);
}
for (var i = 0; i < BatchSize; i++) _batchData[i] = CreatePerson(i);
}
private Person CreatePerson(int i)
@@ -59,7 +51,7 @@ public class InsertBenchmarks
Id = ObjectId.NewObjectId(),
FirstName = $"First_{i}",
LastName = $"Last_{i}",
Age = 20 + (i % 50),
Age = 20 + i % 50,
Bio = null, // Removed large payload to focus on structure
CreatedAt = DateTime.UtcNow,
Balance = 1000.50m * (i + 1),
@@ -72,8 +64,7 @@ public class InsertBenchmarks
};
// Add 10 work history items to stress structure traversal
for (int j = 0; j < 10; j++)
{
for (var j = 0; j < 10; j++)
p.EmploymentHistory.Add(new WorkHistory
{
CompanyName = $"TechCorp_{i}_{j}",
@@ -81,7 +72,6 @@ public class InsertBenchmarks
DurationYears = j,
Tags = new List<string> { "C#", "BSON", "Performance", "Database", "Complex" }
});
}
return p;
}
@@ -111,7 +101,7 @@ public class InsertBenchmarks
_storage?.Dispose();
_storage = null;
System.Threading.Thread.Sleep(100);
Thread.Sleep(100);
if (File.Exists(_docDbPath)) File.Delete(_docDbPath);
if (File.Exists(_docDbWalPath)) File.Delete(_docDbWalPath);

23
tests/CBDD.Tests.Benchmark/Read/ReadBenchmarks.cs
View File

@@ -1,12 +1,8 @@
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using System.IO;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
@@ -19,16 +15,16 @@ namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
public class ReadBenchmarks
{
private const int DocCount = 1000;
private DocumentCollection<Person> _collection = null!;
private string _docDbPath = null!;
private string _docDbWalPath = null!;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private DocumentCollection<Person> _collection = null!;
private ObjectId[] _ids = null!;
private StorageEngine _storage = null!;
private ObjectId _targetId;
private BenchmarkTransactionHolder _transactionHolder = null!;
/// <summary>
/// Tests setup.
@@ -36,7 +32,7 @@ public class ReadBenchmarks
[GlobalSetup]
public void Setup()
{
var temp = AppContext.BaseDirectory;
string temp = AppContext.BaseDirectory;
var id = Guid.NewGuid().ToString("N");
_docDbPath = Path.Combine(temp, $"bench_read_docdb_{id}.db");
_docDbWalPath = Path.ChangeExtension(_docDbPath, ".wal");
@@ -49,11 +45,12 @@ public class ReadBenchmarks
_collection = new DocumentCollection<Person>(_storage, _transactionHolder, new PersonMapper());
_ids = new ObjectId[DocCount];
for (int i = 0; i < DocCount; i++)
for (var i = 0; i < DocCount; i++)
{
var p = CreatePerson(i);
_ids[i] = _collection.Insert(p);
}
_transactionHolder.CommitAndReset();
_targetId = _ids[DocCount / 2];
@@ -79,7 +76,7 @@ public class ReadBenchmarks
Id = ObjectId.NewObjectId(),
FirstName = $"First_{i}",
LastName = $"Last_{i}",
Age = 20 + (i % 50),
Age = 20 + i % 50,
Bio = null,
CreatedAt = DateTime.UtcNow,
Balance = 1000.50m * (i + 1),
@@ -92,8 +89,7 @@ public class ReadBenchmarks
};
// Add 10 work history items
for (int j = 0; j < 10; j++)
{
for (var j = 0; j < 10; j++)
p.EmploymentHistory.Add(new WorkHistory
{
CompanyName = $"TechCorp_{i}_{j}",
@@ -101,7 +97,6 @@ public class ReadBenchmarks
DurationYears = j,
Tags = new List<string> { "C#", "BSON", "Performance", "Database", "Complex" }
});
}
return p;
}

55
tests/CBDD.Tests.Benchmark/Serialization/SerializationBenchmarks.cs
View File

@@ -1,7 +1,8 @@
using System.Collections.Concurrent;
using System.Text.Json;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using ZB.MOM.WW.CBDD.Bson;
using System.Text.Json;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
@@ -13,32 +14,37 @@ namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
public class SerializationBenchmarks
{
private const int BatchSize = 10000;
private Person _person = null!;
private List<Person> _people = null!;
private PersonMapper _mapper = new PersonMapper();
private static readonly ConcurrentDictionary<string, ushort> _keyMap = new(StringComparer.OrdinalIgnoreCase);
private static readonly ConcurrentDictionary<ushort, string> _keys = new();
private readonly List<byte[]> _bsonDataList = new();
private readonly List<byte[]> _jsonDataList = new();
private readonly PersonMapper _mapper = new();
private byte[] _bsonData = Array.Empty<byte>();
private byte[] _jsonData = Array.Empty<byte>();
private List<byte[]> _bsonDataList = new();
private List<byte[]> _jsonDataList = new();
private List<Person> _people = null!;
private Person _person = null!;
private byte[] _serializeBuffer = Array.Empty<byte>();
private static readonly System.Collections.Concurrent.ConcurrentDictionary<string, ushort> _keyMap = new(StringComparer.OrdinalIgnoreCase);
private static readonly System.Collections.Concurrent.ConcurrentDictionary<ushort, string> _keys = new();
static SerializationBenchmarks()
{
ushort id = 1;
string[] initialKeys = { "_id", "firstname", "lastname", "age", "bio", "createdat", "balance", "homeaddress", "street", "city", "zipcode", "employmenthistory", "companyname", "title", "durationyears", "tags" };
foreach (var key in initialKeys)
string[] initialKeys =
{
"_id", "firstname", "lastname", "age", "bio", "createdat", "balance", "homeaddress", "street", "city",
"zipcode", "employmenthistory", "companyname", "title", "durationyears", "tags"
};
foreach (string key in initialKeys)
{
_keyMap[key] = id;
_keys[id] = key;
id++;
}
// Add some indices for arrays
for (int i = 0; i < 100; i++)
for (var i = 0; i < 100; i++)
{
var s = i.ToString();
_keyMap[s] = id;
@@ -55,10 +61,7 @@ public class SerializationBenchmarks
{
_person = CreatePerson(0);
_people = new List<Person>(BatchSize);
for (int i = 0; i < BatchSize; i++)
{
_people.Add(CreatePerson(i));
}
for (var i = 0; i < BatchSize; i++) _people.Add(CreatePerson(i));
// Pre-allocate buffer for BSON serialization
_serializeBuffer = new byte[8192];
@@ -66,7 +69,7 @@ public class SerializationBenchmarks
var writer = new BsonSpanWriter(_serializeBuffer, _keyMap);
// Single item data
var len = _mapper.Serialize(_person, writer);
int len = _mapper.Serialize(_person, writer);
_bsonData = _serializeBuffer.AsSpan(0, len).ToArray();
_jsonData = JsonSerializer.SerializeToUtf8Bytes(_person);
@@ -98,8 +101,7 @@ public class SerializationBenchmarks
}
};
for (int j = 0; j < 10; j++)
{
for (var j = 0; j < 10; j++)
p.EmploymentHistory.Add(new WorkHistory
{
CompanyName = $"TechCorp_{i}_{j}",
@@ -107,7 +109,6 @@ public class SerializationBenchmarks
DurationYears = j,
Tags = new List<string> { "C#", "BSON", "Performance", "Database", "Complex" }
});
}
return p;
}
@@ -174,10 +175,7 @@ public class SerializationBenchmarks
[BenchmarkCategory("Batch")]
public void Serialize_List_Json()
{
foreach (var p in _people)
{
JsonSerializer.SerializeToUtf8Bytes(p);
}
foreach (var p in _people) JsonSerializer.SerializeToUtf8Bytes(p);
}
/// <summary>
@@ -187,7 +185,7 @@ public class SerializationBenchmarks
[BenchmarkCategory("Batch")]
public void Deserialize_List_Bson()
{
foreach (var data in _bsonDataList)
foreach (byte[] data in _bsonDataList)
{
var reader = new BsonSpanReader(data, _keys);
_mapper.Deserialize(reader);
@@ -201,9 +199,6 @@ public class SerializationBenchmarks
[BenchmarkCategory("Batch")]
public void Deserialize_List_Json()
{
foreach (var data in _jsonDataList)
{
JsonSerializer.Deserialize<Person>(data);
}
foreach (byte[] data in _jsonDataList) JsonSerializer.Deserialize<Person>(data);
}
}

99
tests/CBDD.Tests.Benchmark/Storage/DatabaseSizeBenchmark.cs
View File

@@ -1,4 +1,5 @@
using System.Diagnostics;
using System.IO.Compression;
using Microsoft.Extensions.Logging;
using Serilog.Context;
using ZB.MOM.WW.CBDD.Bson;
@@ -10,45 +11,45 @@ namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
internal static class DatabaseSizeBenchmark
{
private const int BatchSize = 50_000;
private const int ProgressInterval = 1_000_000;
private static readonly int[] TargetCounts = [10_000, 1_000_000, 10_000_000];
private static readonly CompressionOptions CompressedBrotliFast = new()
{
EnableCompression = true,
MinSizeBytes = 256,
MinSavingsPercent = 0,
Codec = CompressionCodec.Brotli,
Level = System.IO.Compression.CompressionLevel.Fastest
Level = CompressionLevel.Fastest
};
private static readonly Scenario[] Scenarios =
[
// Separate compression set (no compaction)
new(
Set: "compression",
Name: "CompressionOnly-Uncompressed",
CompressionOptions: CompressionOptions.Default,
RunCompaction: false),
"compression",
"CompressionOnly-Uncompressed",
CompressionOptions.Default,
false),
new(
Set: "compression",
Name: "CompressionOnly-Compressed-BrotliFast",
CompressionOptions: CompressedBrotliFast,
RunCompaction: false),
"compression",
"CompressionOnly-Compressed-BrotliFast",
CompressedBrotliFast,
false),
// Separate compaction set (compaction enabled)
new(
Set: "compaction",
Name: "Compaction-Uncompressed",
CompressionOptions: CompressionOptions.Default,
RunCompaction: true),
"compaction",
"Compaction-Uncompressed",
CompressionOptions.Default,
true),
new(
Set: "compaction",
Name: "Compaction-Compressed-BrotliFast",
CompressionOptions: CompressedBrotliFast,
RunCompaction: true)
"compaction",
"Compaction-Compressed-BrotliFast",
CompressedBrotliFast,
true)
];
private const int BatchSize = 50_000;
private const int ProgressInterval = 1_000_000;
/// <summary>
/// Tests run.
/// </summary>
@@ -62,12 +63,12 @@ internal static class DatabaseSizeBenchmark
logger.LogInformation("Scenarios: {Scenarios}", string.Join(", ", Scenarios.Select(x => $"{x.Set}:{x.Name}")));
logger.LogInformation("Batch size: {BatchSize:N0}", BatchSize);
foreach (var targetCount in TargetCounts)
{
foreach (int targetCount in TargetCounts)
foreach (var scenario in Scenarios)
{
var dbPath = Path.Combine(Path.GetTempPath(), $"cbdd_size_{scenario.Name}_{targetCount}_{Guid.NewGuid():N}.db");
var walPath = Path.ChangeExtension(dbPath, ".wal");
string dbPath = Path.Combine(Path.GetTempPath(),
$"cbdd_size_{scenario.Name}_{targetCount}_{Guid.NewGuid():N}.db");
string walPath = Path.ChangeExtension(dbPath, ".wal");
using var _ = LogContext.PushProperty("TargetCount", targetCount);
using var __ = LogContext.PushProperty("Scenario", scenario.Name);
using var ___ = LogContext.PushProperty("ScenarioSet", scenario.Set);
@@ -97,38 +98,31 @@ internal static class DatabaseSizeBenchmark
var inserted = 0;
while (inserted < targetCount)
{
var currentBatchSize = Math.Min(BatchSize, targetCount - inserted);
int currentBatchSize = Math.Min(BatchSize, targetCount - inserted);
var documents = new SizeBenchmarkDocument[currentBatchSize];
var baseValue = inserted;
int baseValue = inserted;
for (var i = 0; i < currentBatchSize; i++)
{
documents[i] = CreateDocument(baseValue + i);
}
for (var i = 0; i < currentBatchSize; i++) documents[i] = CreateDocument(baseValue + i);
collection.InsertBulk(documents);
transactionHolder.CommitAndReset();
inserted += currentBatchSize;
if (inserted == targetCount || inserted % ProgressInterval == 0)
{
logger.LogInformation("Inserted {Inserted:N0}/{TargetCount:N0}", inserted, targetCount);
}
}
insertStopwatch.Stop();
preCompactDbBytes = File.Exists(dbPath) ? new FileInfo(dbPath).Length : 0;
preCompactWalBytes = File.Exists(walPath) ? new FileInfo(walPath).Length : 0;
if (scenario.RunCompaction)
{
compactionStats = storage.Compact(new CompactionOptions
{
EnableTailTruncation = true,
DefragmentSlottedPages = true,
NormalizeFreeList = true
});
}
postCompactDbBytes = File.Exists(dbPath) ? new FileInfo(dbPath).Length : 0;
postCompactWalBytes = File.Exists(walPath) ? new FileInfo(walPath).Length : 0;
@@ -165,14 +159,12 @@ internal static class DatabaseSizeBenchmark
TryDelete(dbPath);
TryDelete(walPath);
}
}
logger.LogInformation("=== Size Benchmark Summary ===");
foreach (var result in results
.OrderBy(x => x.Set)
.ThenBy(x => x.TargetCount)
.ThenBy(x => x.Scenario))
{
logger.LogInformation(
"{Set,-11} | {Scenario,-38} | {Count,12:N0} docs | insert={Elapsed,12} | pre={Pre,12} | post={Post,12} | shrink={Shrink,12} | compact={CompactBytes,12} | ratio={Ratio}",
result.Set,
@@ -184,7 +176,6 @@ internal static class DatabaseSizeBenchmark
FormatBytes(result.ShrinkBytes),
FormatBytes(result.CompactionStats.ReclaimedFileBytes),
result.CompressionRatioText);
}
WriteSummaryCsv(results, logger);
}
@@ -201,10 +192,7 @@ internal static class DatabaseSizeBenchmark
private static void TryDelete(string path)
{
if (File.Exists(path))
{
File.Delete(path);
}
if (File.Exists(path)) File.Delete(path);
}
private static string FormatBytes(long bytes)
@@ -224,9 +212,9 @@ internal static class DatabaseSizeBenchmark
private static void WriteSummaryCsv(IEnumerable<SizeResult> results, ILogger logger)
{
var outputDirectory = Path.Combine(Directory.GetCurrentDirectory(), "BenchmarkDotNet.Artifacts", "results");
string outputDirectory = Path.Combine(Directory.GetCurrentDirectory(), "BenchmarkDotNet.Artifacts", "results");
Directory.CreateDirectory(outputDirectory);
var outputPath = Path.Combine(outputDirectory, "DatabaseSizeBenchmark-results.csv");
string outputPath = Path.Combine(outputDirectory, "DatabaseSizeBenchmark-results.csv");
var lines = new List<string>
{
@@ -234,7 +222,6 @@ internal static class DatabaseSizeBenchmark
};
foreach (var result in results.OrderBy(x => x.Set).ThenBy(x => x.TargetCount).ThenBy(x => x.Scenario))
{
lines.Add(string.Join(",",
result.Set,
result.Scenario,
@@ -246,7 +233,6 @@ internal static class DatabaseSizeBenchmark
result.ShrinkBytes.ToString(),
result.CompactionStats.ReclaimedFileBytes.ToString(),
result.CompressionRatioText));
}
File.WriteAllLines(outputPath, lines);
logger.LogInformation("Database size summary CSV written to {OutputPath}", outputPath);
@@ -271,10 +257,12 @@ internal static class DatabaseSizeBenchmark
/// Gets or sets the pre compact total bytes.
/// </summary>
public long PreCompactTotalBytes => PreCompactDbBytes + PreCompactWalBytes;
/// <summary>
/// Gets or sets the post compact total bytes.
/// </summary>
public long PostCompactTotalBytes => PostCompactDbBytes + PostCompactWalBytes;
/// <summary>
/// Gets or sets the shrink bytes.
/// </summary>
@@ -295,10 +283,12 @@ internal static class DatabaseSizeBenchmark
/// Gets or sets the id.
/// </summary>
public ObjectId Id { get; set; }
/// <summary>
/// Gets or sets the value.
/// </summary>
public int Value { get; set; }
/// <summary>
/// Gets or sets the name.
/// </summary>
@@ -311,15 +301,21 @@ internal static class DatabaseSizeBenchmark
public override string CollectionName => "size_documents";
/// <inheritdoc />
public override ObjectId GetId(SizeBenchmarkDocument entity) => entity.Id;
public override ObjectId GetId(SizeBenchmarkDocument entity)
{
return entity.Id;
}
/// <inheritdoc />
public override void SetId(SizeBenchmarkDocument entity, ObjectId id) => entity.Id = id;
public override void SetId(SizeBenchmarkDocument entity, ObjectId id)
{
entity.Id = id;
}
/// <inheritdoc />
public override int Serialize(SizeBenchmarkDocument entity, BsonSpanWriter writer)
{
var sizePos = writer.BeginDocument();
int sizePos = writer.BeginDocument();
writer.WriteObjectId("_id", entity.Id);
writer.WriteInt32("value", entity.Value);
writer.WriteString("name", entity.Name);
@@ -336,12 +332,9 @@ internal static class DatabaseSizeBenchmark
while (reader.Remaining > 0)
{
var bsonType = reader.ReadBsonType();
if (bsonType == BsonType.EndOfDocument)
{
break;
}
if (bsonType == BsonType.EndOfDocument) break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
switch (name)
{
case "_id":

17
tests/CBDD.Tests.Benchmark/Workloads/ManualBenchmark.cs
View File

@@ -1,5 +1,4 @@
using System.Diagnostics;
using System.IO;
using System.Text;
using Microsoft.Extensions.Logging;
using Serilog.Context;
@@ -8,7 +7,7 @@ namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
public class ManualBenchmark
{
private static StringBuilder _log = new();
private static readonly StringBuilder _log = new();
private static void Log(ILogger logger, string message = "")
{
@@ -60,10 +59,7 @@ public class ManualBenchmark
try
{
var sw = Stopwatch.StartNew();
for (int i = 0; i < 1000; i++)
{
readBench.DocumentDb_FindById();
}
for (var i = 0; i < 1000; i++) readBench.DocumentDb_FindById();
sw.Stop();
readByIdMs = sw.ElapsedMilliseconds;
Log(logger, $" CBDD FindById x1000: {readByIdMs} ms ({(double)readByIdMs / 1000:F3} ms/op)");
@@ -101,13 +97,10 @@ public class ManualBenchmark
Log(logger, $"FindById x1000: {readByIdMs} ms");
Log(logger, $"Single Insert: {singleInsertMs} ms");
var artifactsDir = Path.Combine(AppContext.BaseDirectory, "BenchmarkDotNet.Artifacts", "results");
if (!Directory.Exists(artifactsDir))
{
Directory.CreateDirectory(artifactsDir);
}
string artifactsDir = Path.Combine(AppContext.BaseDirectory, "BenchmarkDotNet.Artifacts", "results");
if (!Directory.Exists(artifactsDir)) Directory.CreateDirectory(artifactsDir);
var filePath = Path.Combine(artifactsDir, "manual_report.txt");
string filePath = Path.Combine(artifactsDir, "manual_report.txt");
File.WriteAllText(filePath, _log.ToString());
logger.LogInformation("Report saved to: {FilePath}", filePath);
}

Some files were not shown because too many files have changed in this diff Show More