dohertj2/CBDD
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reformat / cleanup
All checks were successful
NuGet Publish / build-and-pack (push) Successful in 46s
Details
NuGet Publish / publish-to-gitea (push) Successful in 56s
Details

Browse Source
This commit is contained in:
Joseph Doherty
2026-02-21 08:10:36 -05:00
parent 4c6aaa5a3f
commit a70d8befae
176 changed files with 50555 additions and 49587 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
CBDD.slnx
View File

@@ -1,12 +1,12 @@
<Solution>
<Folder Name="/src/">
<Project Path="src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj" />
<Project Path="src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj" />
<Project Path="src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj" />
<Project Path="src/CBDD/ZB.MOM.WW.CBDD.csproj" />
</Folder>
<Folder Name="/tests/">
<Project Path="tests/CBDD.Tests.Benchmark/ZB.MOM.WW.CBDD.Tests.Benchmark.csproj" />
<Project Path="tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj" />
</Folder>
<Solution>
<Folder Name="/src/">
<Project Path="src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj"/>
<Project Path="src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj"/>
<Project Path="src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj"/>
<Project Path="src/CBDD/ZB.MOM.WW.CBDD.csproj"/>
</Folder>
<Folder Name="/tests/">
<Project Path="tests/CBDD.Tests.Benchmark/ZB.MOM.WW.CBDD.Tests.Benchmark.csproj"/>
<Project Path="tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj"/>
</Folder>
</Solution>

34
README.md
View File

@@ -1,16 +1,21 @@
# CBDD
CBDD is an embedded, document-oriented database engine for .NET 10. It targets internal platform teams that need predictable ACID behavior, low-latency local persistence, and typed access patterns without running an external database server.
CBDD is an embedded, document-oriented database engine for .NET 10. It targets internal platform teams that need
predictable ACID behavior, low-latency local persistence, and typed access patterns without running an external database
server.
## Purpose And Business Context
CBDD provides a local data layer for services and tools that need transactional durability, deterministic startup, and high-throughput reads/writes. The primary business outcome is reducing operational overhead for workloads that do not require a networked database cluster.
CBDD provides a local data layer for services and tools that need transactional durability, deterministic startup, and
high-throughput reads/writes. The primary business outcome is reducing operational overhead for workloads that do not
require a networked database cluster.
## Ownership And Support
- Owning team: CBDD maintainers (repository owner: `@dohertj2`)
- Primary support path: open a Gitea issue in this repository with labels `incident` or `bug`
- Escalation path: follow [`docs/runbook.md`](docs/runbook.md) and page the release maintainer listed in the active release PR
- Escalation path: follow [`docs/runbook.md`](docs/runbook.md) and page the release maintainer listed in the active
release PR
## Architecture Overview
@@ -22,6 +27,7 @@ CBDD has four primary layers:
4. Source-generated mapping (`src/CBDD.SourceGenerators`)
Detailed architecture material:
- [`docs/architecture.md`](docs/architecture.md)
- [`RFC.md`](RFC.md)
- [`C-BSON.md`](C-BSON.md)
@@ -36,34 +42,44 @@ Detailed architecture material:
## Setup And Local Run
1. Clone the repository.
```bash
git clone https://gitea.dohertylan.com/dohertj2/CBDD.git
cd CBDD
```
Expected outcome: local repository checkout with `CBDD.slnx` present.
2. Restore dependencies.
```bash
dotnet restore
```
Expected outcome: restore completes without package errors.
3. Build the solution.
```bash
dotnet build CBDD.slnx -c Release
```
Expected outcome: solution builds without compiler errors.
4. Run tests.
```bash
dotnet test CBDD.slnx -c Release
```
Expected outcome: all tests pass.
5. Run the full repository fitness check.
```bash
bash scripts/fitness-check.sh
```
Expected outcome: format, build, tests, coverage threshold, and package checks complete.
## Configuration And Secrets
@@ -135,9 +151,12 @@ if (!result.Executed)
Common issues and remediation:
- Build/test environment failures: [`docs/troubleshooting.md#build-and-test-failures`](docs/troubleshooting.md#build-and-test-failures)
- Data-file recovery procedures: [`docs/troubleshooting.md#data-file-and-recovery-issues`](docs/troubleshooting.md#data-file-and-recovery-issues)
- Query/index behavior verification: [`docs/troubleshooting.md#query-and-index-issues`](docs/troubleshooting.md#query-and-index-issues)
- Build/test environment failures: [
`docs/troubleshooting.md#build-and-test-failures`](docs/troubleshooting.md#build-and-test-failures)
- Data-file recovery procedures: [
`docs/troubleshooting.md#data-file-and-recovery-issues`](docs/troubleshooting.md#data-file-and-recovery-issues)
- Query/index behavior verification: [
`docs/troubleshooting.md#query-and-index-issues`](docs/troubleshooting.md#query-and-index-issues)
## Change Governance
@@ -150,4 +169,5 @@ Common issues and remediation:
- Documentation home: [`docs/README.md`](docs/README.md)
- Major feature inventory: [`docs/features/README.md`](docs/features/README.md)
- Architecture decisions: [`docs/adr/0001-storage-engine-and-source-generation.md`](docs/adr/0001-storage-engine-and-source-generation.md)
- Architecture decisions: [
`docs/adr/0001-storage-engine-and-source-generation.md`](docs/adr/0001-storage-engine-and-source-generation.md)

261
src/CBDD.Bson/Document/BsonDocument.cs
View File

@@ -1,60 +1,64 @@
using System;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// Represents an in-memory BSON document with lazy parsing.
/// Uses Memory&lt;byte&gt; to store raw BSON data for zero-copy operations.
/// </summary>
public sealed class BsonDocument
{
using System.Collections.Concurrent;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// Represents an in-memory BSON document with lazy parsing.
/// Uses Memory&lt;byte&gt; to store raw BSON data for zero-copy operations.
/// </summary>
public sealed class BsonDocument
{
private readonly ConcurrentDictionary<ushort, string>? _keys;
private readonly Memory<byte> _rawData;
private readonly System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? _keys;
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocument"/> class from raw BSON memory.
/// Initializes a new instance of the <see cref="BsonDocument" /> class from raw BSON memory.
/// </summary>
/// <param name="rawBsonData">The raw BSON data.</param>
/// <param name="keys">The optional key dictionary.</param>
public BsonDocument(Memory<byte> rawBsonData, System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? keys = null)
public BsonDocument(Memory<byte> rawBsonData, ConcurrentDictionary<ushort, string>? keys = null)
{
_rawData = rawBsonData;
_keys = keys;
}
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocument"/> class from raw BSON bytes.
/// Initializes a new instance of the <see cref="BsonDocument" /> class from raw BSON bytes.
/// </summary>
/// <param name="rawBsonData">The raw BSON data.</param>
/// <param name="keys">The optional key dictionary.</param>
public BsonDocument(byte[] rawBsonData, System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? keys = null)
public BsonDocument(byte[] rawBsonData, ConcurrentDictionary<ushort, string>? keys = null)
{
_rawData = rawBsonData;
_keys = keys;
}
/// <summary>
/// Gets the raw BSON bytes
/// </summary>
public ReadOnlySpan<byte> RawData => _rawData.Span;
/// <summary>
/// Gets the document size in bytes
/// </summary>
public int Size => BitConverter.ToInt32(_rawData.Span[..4]);
/// <summary>
/// Creates a reader for this document
/// </summary>
public BsonSpanReader GetReader() => new BsonSpanReader(_rawData.Span, _keys ?? new System.Collections.Concurrent.ConcurrentDictionary<ushort, string>());
}
/// <summary>
/// Tries to get a field value by name.
/// Returns false if field not found.
/// Gets the raw BSON bytes
/// </summary>
public ReadOnlySpan<byte> RawData => _rawData.Span;
/// <summary>
/// Gets the document size in bytes
/// </summary>
public int Size => BitConverter.ToInt32(_rawData.Span[..4]);
/// <summary>
/// Creates a reader for this document
/// </summary>
public BsonSpanReader GetReader()
{
return new BsonSpanReader(_rawData.Span,
_keys ?? new ConcurrentDictionary<ushort, string>());
}
/// <summary>
/// Tries to get a field value by name.
/// Returns false if field not found.
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise <see langword="null"/>.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <param name="value">When this method returns, contains the field value if found; otherwise <see langword="null" />.</param>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetString(string fieldName, out string? value)
{
value = null;
@@ -66,30 +70,30 @@ public sealed class BsonDocument
fieldName = fieldName.ToLowerInvariant();
while (reader.Remaining > 1)
{
var type = reader.ReadBsonType();
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
var type = reader.ReadBsonType();
if (type == BsonType.EndOfDocument)
break;
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.String)
{
value = reader.ReadString();
return true;
}
reader.SkipValue(type);
}
return false;
}
if (name == fieldName && type == BsonType.String)
{
value = reader.ReadString();
return true;
}
reader.SkipValue(type);
}
return false;
}
/// <summary>
/// Tries to get an Int32 field value by name.
/// Tries to get an Int32 field value by name.
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise zero.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetInt32(string fieldName, out int value)
{
value = 0;
@@ -101,30 +105,30 @@ public sealed class BsonDocument
fieldName = fieldName.ToLowerInvariant();
while (reader.Remaining > 1)
{
var type = reader.ReadBsonType();
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
var type = reader.ReadBsonType();
if (type == BsonType.EndOfDocument)
break;
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.Int32)
{
value = reader.ReadInt32();
return true;
}
reader.SkipValue(type);
}
return false;
}
if (name == fieldName && type == BsonType.Int32)
{
value = reader.ReadInt32();
return true;
}
reader.SkipValue(type);
}
return false;
}
/// <summary>
/// Tries to get an ObjectId field value by name.
/// Tries to get an ObjectId field value by name.
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise default.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetObjectId(string fieldName, out ObjectId value)
{
value = default;
@@ -136,52 +140,53 @@ public sealed class BsonDocument
fieldName = fieldName.ToLowerInvariant();
while (reader.Remaining > 1)
{
var type = reader.ReadBsonType();
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
var type = reader.ReadBsonType();
if (type == BsonType.EndOfDocument)
break;
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.ObjectId)
{
value = reader.ReadObjectId();
return true;
}
reader.SkipValue(type);
}
return false;
}
if (name == fieldName && type == BsonType.ObjectId)
{
value = reader.ReadObjectId();
return true;
}
reader.SkipValue(type);
}
return false;
}
/// <summary>
/// Creates a new BsonDocument from field values using a builder pattern
/// Creates a new BsonDocument from field values using a builder pattern
/// </summary>
/// <param name="keyMap">The key map used for field name encoding.</param>
/// <param name="buildAction">The action that populates the builder.</param>
/// <returns>The created BSON document.</returns>
public static BsonDocument Create(System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap, Action<BsonDocumentBuilder> buildAction)
public static BsonDocument Create(ConcurrentDictionary<string, ushort> keyMap,
Action<BsonDocumentBuilder> buildAction)
{
var builder = new BsonDocumentBuilder(keyMap);
buildAction(builder);
return builder.Build();
}
}
/// <summary>
/// Builder for creating BSON documents
/// </summary>
return builder.Build();
}
}
/// <summary>
/// Builder for creating BSON documents
/// </summary>
public sealed class BsonDocumentBuilder
{
private byte[] _buffer = new byte[1024]; // Start with 1KB
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, ushort> _keyMap;
private readonly ConcurrentDictionary<string, ushort> _keyMap;
private byte[] _buffer = new byte[1024]; // Start with 1KB
private int _position;
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocumentBuilder"/> class.
/// Initializes a new instance of the <see cref="BsonDocumentBuilder" /> class.
/// </summary>
/// <param name="keyMap">The key map used for field name encoding.</param>
public BsonDocumentBuilder(System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap)
public BsonDocumentBuilder(ConcurrentDictionary<string, ushort> keyMap)
{
_keyMap = keyMap;
var writer = new BsonSpanWriter(_buffer, _keyMap);
@@ -189,7 +194,7 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Adds a string field to the document.
/// Adds a string field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -204,7 +209,7 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Adds an Int32 field to the document.
/// Adds an Int32 field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -213,13 +218,13 @@ public sealed class BsonDocumentBuilder
{
EnsureCapacity(64);
var writer = new BsonSpanWriter(_buffer.AsSpan(_position..), _keyMap);
writer.WriteInt32(name, value);
writer.WriteInt32(name, value);
_position += writer.Position;
return this;
}
/// <summary>
/// Adds an Int64 field to the document.
/// Adds an Int64 field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -228,13 +233,13 @@ public sealed class BsonDocumentBuilder
{
EnsureCapacity(64);
var writer = new BsonSpanWriter(_buffer.AsSpan(_position..), _keyMap);
writer.WriteInt64(name, value);
writer.WriteInt64(name, value);
_position += writer.Position;
return this;
}
/// <summary>
/// Adds a Boolean field to the document.
/// Adds a Boolean field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -243,13 +248,13 @@ public sealed class BsonDocumentBuilder
{
EnsureCapacity(64);
var writer = new BsonSpanWriter(_buffer.AsSpan(_position..), _keyMap);
writer.WriteBoolean(name, value);
writer.WriteBoolean(name, value);
_position += writer.Position;
return this;
}
/// <summary>
/// Adds an ObjectId field to the document.
/// Adds an ObjectId field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -258,19 +263,19 @@ public sealed class BsonDocumentBuilder
{
EnsureCapacity(64);
var writer = new BsonSpanWriter(_buffer.AsSpan(_position..), _keyMap);
writer.WriteObjectId(name, value);
writer.WriteObjectId(name, value);
_position += writer.Position;
return this;
}
/// <summary>
/// Builds a BSON document from the accumulated fields.
/// Builds a BSON document from the accumulated fields.
/// </summary>
/// <returns>The constructed BSON document.</returns>
public BsonDocument Build()
{
// Layout: [int32 size][field bytes...][0x00 terminator]
var totalSize = _position + 5;
int totalSize = _position + 5;
var finalBuffer = new byte[totalSize];
BitConverter.TryWriteBytes(finalBuffer.AsSpan(0, 4), totalSize);
@@ -279,14 +284,14 @@ public sealed class BsonDocumentBuilder
return new BsonDocument(finalBuffer);
}
private void EnsureCapacity(int additional)
{
if (_position + additional > _buffer.Length)
{
var newBuffer = new byte[_buffer.Length * 2];
_buffer.CopyTo(newBuffer, 0);
_buffer = newBuffer;
}
}
}
private void EnsureCapacity(int additional)
{
if (_position + additional > _buffer.Length)
{
var newBuffer = new byte[_buffer.Length * 2];
_buffer.CopyTo(newBuffer, 0);
_buffer = newBuffer;
}
}
}

4
src/CBDD.Bson/Document/BsonType.cs
View File

@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// BSON type codes as defined in BSON spec
/// BSON type codes as defined in BSON spec
/// </summary>
public enum BsonType : byte
{
@@ -27,4 +27,4 @@ public enum BsonType : byte
Decimal128 = 0x13,
MinKey = 0xFF,
MaxKey = 0x7F
}
}

297
src/CBDD.Bson/IO/BsonBufferWriter.cs
View File

@@ -1,262 +1,263 @@
using System;
using System.Buffers;
using System.Buffers.Binary;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// BSON writer that serializes to an IBufferWriter, enabling streaming serialization
/// without fixed buffer size limits.
/// </summary>
using System.Buffers;
using System.Buffers.Binary;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// BSON writer that serializes to an IBufferWriter, enabling streaming serialization
/// without fixed buffer size limits.
/// </summary>
public ref struct BsonBufferWriter
{
private IBufferWriter<byte> _writer;
private int _totalBytesWritten;
private readonly IBufferWriter<byte> _writer;
/// <summary>
/// Initializes a new instance of the <see cref="BsonBufferWriter"/> struct.
/// Initializes a new instance of the <see cref="BsonBufferWriter" /> struct.
/// </summary>
/// <param name="writer">The buffer writer to write BSON bytes to.</param>
public BsonBufferWriter(IBufferWriter<byte> writer)
{
_writer = writer;
_totalBytesWritten = 0;
Position = 0;
}
/// <summary>
/// Gets the current write position in bytes.
/// Gets the current write position in bytes.
/// </summary>
public int Position => _totalBytesWritten;
public int Position { get; private set; }
private void WriteBytes(ReadOnlySpan<byte> data)
{
var destination = _writer.GetSpan(data.Length);
data.CopyTo(destination);
_writer.Advance(data.Length);
_totalBytesWritten += data.Length;
private void WriteBytes(ReadOnlySpan<byte> data)
{
var destination = _writer.GetSpan(data.Length);
data.CopyTo(destination);
_writer.Advance(data.Length);
Position += data.Length;
}
private void WriteByte(byte value)
{
var span = _writer.GetSpan(1);
span[0] = value;
_writer.Advance(1);
_totalBytesWritten++;
private void WriteByte(byte value)
{
var span = _writer.GetSpan(1);
span[0] = value;
_writer.Advance(1);
Position++;
}
/// <summary>
/// Writes a BSON date-time field.
/// Writes a BSON date-time field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date-time value.</param>
public void WriteDateTime(string name, DateTime value)
{
WriteByte((byte)BsonType.DateTime);
WriteCString(name);
// BSON DateTime: milliseconds since Unix epoch (UTC)
var unixEpoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var milliseconds = (long)(value.ToUniversalTime() - unixEpoch).TotalMilliseconds;
WriteInt64Internal(milliseconds);
{
WriteByte((byte)BsonType.DateTime);
WriteCString(name);
// BSON DateTime: milliseconds since Unix epoch (UTC)
var unixEpoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var milliseconds = (long)(value.ToUniversalTime() - unixEpoch).TotalMilliseconds;
WriteInt64Internal(milliseconds);
}
/// <summary>
/// Begins writing a BSON document.
/// Begins writing a BSON document.
/// </summary>
/// <returns>The position where the document size placeholder was written.</returns>
public int BeginDocument()
{
// Write placeholder for size (4 bytes)
var sizePosition = _totalBytesWritten;
var span = _writer.GetSpan(4);
// Initialize with default value (will be patched later)
span[0] = 0; span[1] = 0; span[2] = 0; span[3] = 0;
_writer.Advance(4);
_totalBytesWritten += 4;
return sizePosition;
{
// Write placeholder for size (4 bytes)
int sizePosition = Position;
var span = _writer.GetSpan(4);
// Initialize with default value (will be patched later)
span[0] = 0;
span[1] = 0;
span[2] = 0;
span[3] = 0;
_writer.Advance(4);
Position += 4;
return sizePosition;
}
/// <summary>
/// Ends the current BSON document by writing the document terminator.
/// Ends the current BSON document by writing the document terminator.
/// </summary>
/// <param name="sizePosition">The position of the size placeholder for this document.</param>
public void EndDocument(int sizePosition)
{
// Write document terminator
{
// Write document terminator
WriteByte(0);
// Note: Size patching must be done by caller after accessing WrittenSpan
// from ArrayBufferWriter (or equivalent)
}
}
/// <summary>
/// Begins writing a nested BSON document field.
/// Begins writing a nested BSON document field.
/// </summary>
/// <param name="name">The field name.</param>
/// <returns>The position where the nested document size placeholder was written.</returns>
public int BeginDocument(string name)
{
WriteByte((byte)BsonType.Document);
WriteCString(name);
return BeginDocument();
}
{
WriteByte((byte)BsonType.Document);
WriteCString(name);
return BeginDocument();
}
/// <summary>
/// Begins writing a BSON array field.
/// Begins writing a BSON array field.
/// </summary>
/// <param name="name">The field name.</param>
/// <returns>The position where the array document size placeholder was written.</returns>
public int BeginArray(string name)
{
WriteByte((byte)BsonType.Array);
WriteCString(name);
return BeginDocument();
}
{
WriteByte((byte)BsonType.Array);
WriteCString(name);
return BeginDocument();
}
/// <summary>
/// Ends the current BSON array.
/// Ends the current BSON array.
/// </summary>
/// <param name="sizePosition">The position of the size placeholder for this array.</param>
public void EndArray(int sizePosition)
{
EndDocument(sizePosition);
{
EndDocument(sizePosition);
}
// Private helper methods
private void WriteInt32Internal(int value)
{
var span = _writer.GetSpan(4);
BinaryPrimitives.WriteInt32LittleEndian(span, value);
_writer.Advance(4);
_totalBytesWritten += 4;
private void WriteInt32Internal(int value)
{
var span = _writer.GetSpan(4);
BinaryPrimitives.WriteInt32LittleEndian(span, value);
_writer.Advance(4);
Position += 4;
}
private void WriteInt64Internal(long value)
{
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteInt64LittleEndian(span, value);
_writer.Advance(8);
_totalBytesWritten += 8;
private void WriteInt64Internal(long value)
{
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteInt64LittleEndian(span, value);
_writer.Advance(8);
Position += 8;
}
/// <summary>
/// Writes a BSON ObjectId field.
/// Writes a BSON ObjectId field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The ObjectId value.</param>
public void WriteObjectId(string name, ObjectId value)
{
WriteByte((byte)BsonType.ObjectId);
WriteCString(name);
WriteBytes(value.ToByteArray());
}
{
WriteByte((byte)BsonType.ObjectId);
WriteCString(name);
WriteBytes(value.ToByteArray());
}
/// <summary>
/// Writes a BSON string field.
/// Writes a BSON string field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The string value.</param>
public void WriteString(string name, string value)
{
WriteByte((byte)BsonType.String);
WriteCString(name);
WriteStringValue(value);
{
WriteByte((byte)BsonType.String);
WriteCString(name);
WriteStringValue(value);
}
/// <summary>
/// Writes a BSON boolean field.
/// Writes a BSON boolean field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The boolean value.</param>
public void WriteBoolean(string name, bool value)
{
WriteByte((byte)BsonType.Boolean);
WriteCString(name);
WriteByte((byte)(value ? 1 : 0));
{
WriteByte((byte)BsonType.Boolean);
WriteCString(name);
WriteByte((byte)(value ? 1 : 0));
}
/// <summary>
/// Writes a BSON null field.
/// Writes a BSON null field.
/// </summary>
/// <param name="name">The field name.</param>
public void WriteNull(string name)
{
WriteByte((byte)BsonType.Null);
WriteCString(name);
}
private void WriteStringValue(string value)
{
// String: length (int32) + UTF8 bytes + null terminator
var bytes = Encoding.UTF8.GetBytes(value);
WriteInt32Internal(bytes.Length + 1); // +1 for null terminator
WriteBytes(bytes);
WriteByte(0);
{
WriteByte((byte)BsonType.Null);
WriteCString(name);
}
private void WriteDoubleInternal(double value)
{
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteDoubleLittleEndian(span, value);
_writer.Advance(8);
_totalBytesWritten += 8;
private void WriteStringValue(string value)
{
// String: length (int32) + UTF8 bytes + null terminator
byte[] bytes = Encoding.UTF8.GetBytes(value);
WriteInt32Internal(bytes.Length + 1); // +1 for null terminator
WriteBytes(bytes);
WriteByte(0);
}
private void WriteDoubleInternal(double value)
{
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteDoubleLittleEndian(span, value);
_writer.Advance(8);
Position += 8;
}
/// <summary>
/// Writes a BSON binary field.
/// Writes a BSON binary field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="data">The binary data.</param>
public void WriteBinary(string name, ReadOnlySpan<byte> data)
{
WriteByte((byte)BsonType.Binary);
WriteCString(name);
WriteInt32Internal(data.Length);
WriteByte(0); // Binary subtype: Generic
WriteBytes(data);
{
WriteByte((byte)BsonType.Binary);
WriteCString(name);
WriteInt32Internal(data.Length);
WriteByte(0); // Binary subtype: Generic
WriteBytes(data);
}
/// <summary>
/// Writes a BSON 64-bit integer field.
/// Writes a BSON 64-bit integer field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 64-bit integer value.</param>
public void WriteInt64(string name, long value)
{
WriteByte((byte)BsonType.Int64);
WriteCString(name);
WriteInt64Internal(value);
{
WriteByte((byte)BsonType.Int64);
WriteCString(name);
WriteInt64Internal(value);
}
/// <summary>
/// Writes a BSON double field.
/// Writes a BSON double field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The double value.</param>
public void WriteDouble(string name, double value)
{
WriteByte((byte)BsonType.Double);
WriteCString(name);
WriteDoubleInternal(value);
{
WriteByte((byte)BsonType.Double);
WriteCString(name);
WriteDoubleInternal(value);
}
private void WriteCString(string value)
{
byte[] bytes = Encoding.UTF8.GetBytes(value);
WriteBytes(bytes);
WriteByte(0); // Null terminator
}
private void WriteCString(string value)
{
var bytes = Encoding.UTF8.GetBytes(value);
WriteBytes(bytes);
WriteByte(0); // Null terminator
}
/// <summary>
/// Writes a BSON 32-bit integer field.
/// Writes a BSON 32-bit integer field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 32-bit integer value.</param>
public void WriteInt32(string name, int value)
{
WriteByte((byte)BsonType.Int32);
WriteCString(name);
WriteInt32Internal(value);
}
}
{
WriteByte((byte)BsonType.Int32);
WriteCString(name);
WriteInt32Internal(value);
}
}

370
src/CBDD.Bson/IO/BsonSpanReader.cs
View File

@@ -1,344 +1,343 @@
using System;
using System.Buffers.Binary;
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// Zero-allocation BSON reader using ReadOnlySpan&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// Zero-allocation BSON reader using ReadOnlySpan&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// </summary>
public ref struct BsonSpanReader
{
private ReadOnlySpan<byte> _buffer;
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<ushort, string> _keys;
/// <summary>
/// Initializes a new instance of the <see cref="BsonSpanReader"/> struct.
/// </summary>
/// <param name="buffer">The BSON buffer to read.</param>
/// <param name="keys">The reverse key dictionary used for compressed element headers.</param>
public BsonSpanReader(ReadOnlySpan<byte> buffer, System.Collections.Concurrent.ConcurrentDictionary<ushort, string> keys)
{
_buffer = buffer;
_position = 0;
_keys = keys;
}
/// <summary>
/// Gets the current read position in the buffer.
/// </summary>
public int Position => _position;
/// <summary>
/// Gets the number of unread bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - _position;
private readonly ConcurrentDictionary<ushort, string> _keys;
/// <summary>
/// Reads the document size (first 4 bytes of a BSON document)
/// Initializes a new instance of the <see cref="BsonSpanReader" /> struct.
/// </summary>
/// <param name="buffer">The BSON buffer to read.</param>
/// <param name="keys">The reverse key dictionary used for compressed element headers.</param>
public BsonSpanReader(ReadOnlySpan<byte> buffer, ConcurrentDictionary<ushort, string> keys)
{
_buffer = buffer;
Position = 0;
_keys = keys;
}
/// <summary>
/// Gets the current read position in the buffer.
/// </summary>
public int Position { get; private set; }
/// <summary>
/// Gets the number of unread bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - Position;
/// <summary>
/// Reads the document size (first 4 bytes of a BSON document)
/// </summary>
public int ReadDocumentSize()
{
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to read document size");
var size = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int size = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
return size;
}
/// <summary>
/// Reads a BSON element type
/// Reads a BSON element type
/// </summary>
public BsonType ReadBsonType()
{
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read BSON type");
var type = (BsonType)_buffer[_position];
_position++;
var type = (BsonType)_buffer[Position];
Position++;
return type;
}
/// <summary>
/// Reads a C-style null-terminated string (e-name in BSON spec)
/// Reads a C-style null-terminated string (e-name in BSON spec)
/// </summary>
public string ReadCString()
{
var start = _position;
while (_position < _buffer.Length && _buffer[_position] != 0)
_position++;
int start = Position;
while (Position < _buffer.Length && _buffer[Position] != 0)
Position++;
if (_position >= _buffer.Length)
if (Position >= _buffer.Length)
throw new InvalidOperationException("Unterminated C-string");
var nameBytes = _buffer.Slice(start, _position - start);
_position++; // Skip null terminator
var nameBytes = _buffer.Slice(start, Position - start);
Position++; // Skip null terminator
return Encoding.UTF8.GetString(nameBytes);
}
/// <summary>
/// Reads a C-string into a destination span. Returns the number of bytes written.
/// </summary>
/// <param name="destination">The destination character span.</param>
public int ReadCString(Span<char> destination)
/// <summary>
/// Reads a C-string into a destination span. Returns the number of bytes written.
/// </summary>
/// <param name="destination">The destination character span.</param>
public int ReadCString(Span<char> destination)
{
var start = _position;
while (_position < _buffer.Length && _buffer[_position] != 0)
_position++;
int start = Position;
while (Position < _buffer.Length && _buffer[Position] != 0)
Position++;
if (_position >= _buffer.Length)
if (Position >= _buffer.Length)
throw new InvalidOperationException("Unterminated C-string");
var nameBytes = _buffer.Slice(start, _position - start);
_position++; // Skip null terminator
var nameBytes = _buffer.Slice(start, Position - start);
Position++; // Skip null terminator
return Encoding.UTF8.GetChars(nameBytes, destination);
}
/// <summary>
/// Reads a BSON string (4-byte length + UTF-8 bytes + null terminator)
/// Reads a BSON string (4-byte length + UTF-8 bytes + null terminator)
/// </summary>
public string ReadString()
{
var length = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int length = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
if (length < 1)
throw new InvalidOperationException("Invalid string length");
var stringBytes = _buffer.Slice(_position, length - 1); // Exclude null terminator
_position += length;
var stringBytes = _buffer.Slice(Position, length - 1); // Exclude null terminator
Position += length;
return Encoding.UTF8.GetString(stringBytes);
}
/// <summary>
/// Reads a 32-bit integer.
/// </summary>
public int ReadInt32()
/// <summary>
/// Reads a 32-bit integer.
/// </summary>
public int ReadInt32()
{
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to read Int32");
var value = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
return value;
}
/// <summary>
/// Reads a 64-bit integer.
/// </summary>
public long ReadInt64()
{
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Int64");
var value = BinaryPrimitives.ReadInt64LittleEndian(_buffer.Slice(_position, 8));
_position += 8;
return value;
}
/// <summary>
/// Reads a double-precision floating point value.
/// </summary>
public double ReadDouble()
{
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Double");
var value = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
int value = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
return value;
}
/// <summary>
/// Reads spatial coordinates from a BSON array [X, Y].
/// Returns a (double, double) tuple.
/// Reads a 64-bit integer.
/// </summary>
public long ReadInt64()
{
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Int64");
long value = BinaryPrimitives.ReadInt64LittleEndian(_buffer.Slice(Position, 8));
Position += 8;
return value;
}
/// <summary>
/// Reads a double-precision floating point value.
/// </summary>
public double ReadDouble()
{
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Double");
double value = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
return value;
}
/// <summary>
/// Reads spatial coordinates from a BSON array [X, Y].
/// Returns a (double, double) tuple.
/// </summary>
public (double, double) ReadCoordinates()
{
// Skip array size (4 bytes)
_position += 4;
Position += 4;
// Skip element 0 header: Type(1) + Name("0\0") (3 bytes)
_position += 3;
var x = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
Position += 3;
double x = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
// Skip element 1 header: Type(1) + Name("1\0") (3 bytes)
_position += 3;
var y = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
Position += 3;
double y = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
// Skip end of array marker (1 byte)
_position++;
Position++;
return (x, y);
}
/// <summary>
/// Reads a Decimal128 value.
/// </summary>
public decimal ReadDecimal128()
/// <summary>
/// Reads a Decimal128 value.
/// </summary>
public decimal ReadDecimal128()
{
if (Remaining < 16)
throw new InvalidOperationException("Not enough bytes to read Decimal128");
var bits = new int[4];
bits[0] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
bits[1] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 4, 4));
bits[2] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 8, 4));
bits[3] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 12, 4));
_position += 16;
bits[0] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
bits[1] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 4, 4));
bits[2] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 8, 4));
bits[3] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 12, 4));
Position += 16;
return new decimal(bits);
}
/// <summary>
/// Reads a boolean value.
/// </summary>
public bool ReadBoolean()
/// <summary>
/// Reads a boolean value.
/// </summary>
public bool ReadBoolean()
{
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read Boolean");
var value = _buffer[_position] != 0;
_position++;
bool value = _buffer[Position] != 0;
Position++;
return value;
}
/// <summary>
/// Reads a BSON DateTime (UTC milliseconds since Unix epoch)
/// Reads a BSON DateTime (UTC milliseconds since Unix epoch)
/// </summary>
public DateTime ReadDateTime()
{
var milliseconds = ReadInt64();
long milliseconds = ReadInt64();
return DateTimeOffset.FromUnixTimeMilliseconds(milliseconds).UtcDateTime;
}
/// <summary>
/// Reads a BSON DateTime as DateTimeOffset (UTC milliseconds since Unix epoch)
/// Reads a BSON DateTime as DateTimeOffset (UTC milliseconds since Unix epoch)
/// </summary>
public DateTimeOffset ReadDateTimeOffset()
{
var milliseconds = ReadInt64();
long milliseconds = ReadInt64();
return DateTimeOffset.FromUnixTimeMilliseconds(milliseconds);
}
/// <summary>
/// Reads a TimeSpan from BSON Int64 (ticks)
/// Reads a TimeSpan from BSON Int64 (ticks)
/// </summary>
public TimeSpan ReadTimeSpan()
{
var ticks = ReadInt64();
long ticks = ReadInt64();
return TimeSpan.FromTicks(ticks);
}
/// <summary>
/// Reads a DateOnly from BSON Int32 (day number)
/// Reads a DateOnly from BSON Int32 (day number)
/// </summary>
public DateOnly ReadDateOnly()
{
var dayNumber = ReadInt32();
int dayNumber = ReadInt32();
return DateOnly.FromDayNumber(dayNumber);
}
/// <summary>
/// Reads a TimeOnly from BSON Int64 (ticks)
/// Reads a TimeOnly from BSON Int64 (ticks)
/// </summary>
public TimeOnly ReadTimeOnly()
{
var ticks = ReadInt64();
long ticks = ReadInt64();
return new TimeOnly(ticks);
}
/// <summary>
/// Reads a GUID value.
/// </summary>
public Guid ReadGuid()
/// <summary>
/// Reads a GUID value.
/// </summary>
public Guid ReadGuid()
{
return Guid.Parse(ReadString());
}
/// <summary>
/// Reads a BSON ObjectId (12 bytes)
/// Reads a BSON ObjectId (12 bytes)
/// </summary>
public ObjectId ReadObjectId()
{
if (Remaining < 12)
throw new InvalidOperationException("Not enough bytes to read ObjectId");
var oidBytes = _buffer.Slice(_position, 12);
_position += 12;
var oidBytes = _buffer.Slice(Position, 12);
Position += 12;
return new ObjectId(oidBytes);
}
/// <summary>
/// Reads binary data (subtype + length + bytes)
/// </summary>
/// <param name="subtype">When this method returns, contains the BSON binary subtype.</param>
public ReadOnlySpan<byte> ReadBinary(out byte subtype)
/// <summary>
/// Reads binary data (subtype + length + bytes)
/// </summary>
/// <param name="subtype">When this method returns, contains the BSON binary subtype.</param>
public ReadOnlySpan<byte> ReadBinary(out byte subtype)
{
var length = ReadInt32();
int length = ReadInt32();
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read binary subtype");
subtype = _buffer[_position];
_position++;
throw new InvalidOperationException("Not enough bytes to read binary subtype");
subtype = _buffer[Position];
Position++;
if (Remaining < length)
throw new InvalidOperationException("Not enough bytes to read binary data");
var data = _buffer.Slice(_position, length);
_position += length;
var data = _buffer.Slice(Position, length);
Position += length;
return data;
}
/// <summary>
/// Skips the current value based on type
/// </summary>
/// <param name="type">The BSON type of the value to skip.</param>
public void SkipValue(BsonType type)
/// <summary>
/// Skips the current value based on type
/// </summary>
/// <param name="type">The BSON type of the value to skip.</param>
public void SkipValue(BsonType type)
{
switch (type)
{
case BsonType.Double:
_position += 8;
Position += 8;
break;
case BsonType.String:
var stringLength = ReadInt32();
_position += stringLength;
int stringLength = ReadInt32();
Position += stringLength;
break;
case BsonType.Document:
case BsonType.Array:
var docLength = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += docLength;
int docLength = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += docLength;
break;
case BsonType.Binary:
var binaryLength = ReadInt32();
_position += 1 + binaryLength; // subtype + data
int binaryLength = ReadInt32();
Position += 1 + binaryLength; // subtype + data
break;
case BsonType.ObjectId:
_position += 12;
Position += 12;
break;
case BsonType.Boolean:
_position += 1;
Position += 1;
break;
case BsonType.DateTime:
case BsonType.Int64:
case BsonType.Timestamp:
_position += 8;
Position += 8;
break;
case BsonType.Decimal128:
_position += 16;
Position += 16;
break;
case BsonType.Int32:
_position += 4;
Position += 4;
break;
case BsonType.Null:
// No data
@@ -348,49 +347,50 @@ public ref struct BsonSpanReader
}
}
/// <summary>
/// Reads a single byte.
/// </summary>
public byte ReadByte()
/// <summary>
/// Reads a single byte.
/// </summary>
public byte ReadByte()
{
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read byte");
var value = _buffer[_position];
_position++;
byte value = _buffer[Position];
Position++;
return value;
}
/// <summary>
/// Peeks a 32-bit integer at the current position without advancing.
/// </summary>
public int PeekInt32()
/// <summary>
/// Peeks a 32-bit integer at the current position without advancing.
/// </summary>
public int PeekInt32()
{
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to peek Int32");
return BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
return BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
}
/// <summary>
/// Reads an element header key identifier and resolves it to a key name.
/// </summary>
public string ReadElementHeader()
/// <summary>
/// Reads an element header key identifier and resolves it to a key name.
/// </summary>
public string ReadElementHeader()
{
if (Remaining < 2)
throw new InvalidOperationException("Not enough bytes to read BSON element key ID");
var id = BinaryPrimitives.ReadUInt16LittleEndian(_buffer.Slice(_position, 2));
_position += 2;
ushort id = BinaryPrimitives.ReadUInt16LittleEndian(_buffer.Slice(Position, 2));
Position += 2;
if (!_keys.TryGetValue(id, out var key))
{
if (!_keys.TryGetValue(id, out string? key))
throw new InvalidOperationException($"BSON Key ID {id} not found in reverse key dictionary.");
}
return key;
}
/// <summary>
/// Returns a span containing all unread bytes.
/// </summary>
public ReadOnlySpan<byte> RemainingBytes() => _buffer[_position..];
}
/// <summary>
/// Returns a span containing all unread bytes.
/// </summary>
public ReadOnlySpan<byte> RemainingBytes()
{
return _buffer[Position..];
}
}

466
src/CBDD.Bson/IO/BsonSpanWriter.cs
View File

@@ -1,382 +1,380 @@
using System;
using System.Buffers.Binary;
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// Zero-allocation BSON writer using Span&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// Zero-allocation BSON writer using Span&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// </summary>
public ref struct BsonSpanWriter
{
private Span<byte> _buffer;
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, ushort> _keyMap;
private readonly ConcurrentDictionary<string, ushort> _keyMap;
/// <summary>
/// Initializes a new instance of the <see cref="BsonSpanWriter"/> struct.
/// </summary>
/// <param name="buffer">The destination buffer to write BSON bytes into.</param>
/// <param name="keyMap">The cached key-name to key-id mapping.</param>
public BsonSpanWriter(Span<byte> buffer, System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap)
/// <summary>
/// Initializes a new instance of the <see cref="BsonSpanWriter" /> struct.
/// </summary>
/// <param name="buffer">The destination buffer to write BSON bytes into.</param>
/// <param name="keyMap">The cached key-name to key-id mapping.</param>
public BsonSpanWriter(Span<byte> buffer, ConcurrentDictionary<string, ushort> keyMap)
{
_buffer = buffer;
_keyMap = keyMap;
_position = 0;
Position = 0;
}
/// <summary>
/// Gets the current write position in the buffer.
/// </summary>
public int Position => _position;
/// <summary>
/// Gets the number of bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - _position;
/// <summary>
/// Gets the current write position in the buffer.
/// </summary>
public int Position { get; private set; }
/// <summary>
/// Writes document size placeholder and returns the position to patch later
/// Gets the number of bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - Position;
/// <summary>
/// Writes document size placeholder and returns the position to patch later
/// </summary>
public int WriteDocumentSizePlaceholder()
{
var sizePosition = _position;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), 0);
_position += 4;
int sizePosition = Position;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), 0);
Position += 4;
return sizePosition;
}
/// <summary>
/// Patches the document size at the given position
/// </summary>
/// <param name="sizePosition">The position where the size placeholder was written.</param>
public void PatchDocumentSize(int sizePosition)
/// <summary>
/// Patches the document size at the given position
/// </summary>
/// <param name="sizePosition">The position where the size placeholder was written.</param>
public void PatchDocumentSize(int sizePosition)
{
var size = _position - sizePosition;
int size = Position - sizePosition;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(sizePosition, 4), size);
}
/// <summary>
/// Writes a BSON element header (type + name)
/// </summary>
/// <param name="type">The BSON element type.</param>
/// <param name="name">The field name.</param>
public void WriteElementHeader(BsonType type, string name)
/// <summary>
/// Writes a BSON element header (type + name)
/// </summary>
/// <param name="type">The BSON element type.</param>
/// <param name="name">The field name.</param>
public void WriteElementHeader(BsonType type, string name)
{
_buffer[_position] = (byte)type;
_position++;
_buffer[Position] = (byte)type;
Position++;
if (!_keyMap.TryGetValue(name, out var id))
{
throw new InvalidOperationException($"BSON Key '{name}' not found in dictionary cache. Ensure all keys are registered before serialization.");
}
if (!_keyMap.TryGetValue(name, out ushort id))
throw new InvalidOperationException(
$"BSON Key '{name}' not found in dictionary cache. Ensure all keys are registered before serialization.");
BinaryPrimitives.WriteUInt16LittleEndian(_buffer.Slice(_position, 2), id);
_position += 2;
BinaryPrimitives.WriteUInt16LittleEndian(_buffer.Slice(Position, 2), id);
Position += 2;
}
/// <summary>
/// Writes a C-style null-terminated string
/// Writes a C-style null-terminated string
/// </summary>
private void WriteCString(string value)
{
var bytesWritten = Encoding.UTF8.GetBytes(value, _buffer[_position..]);
_position += bytesWritten;
_buffer[_position] = 0; // Null terminator
_position++;
int bytesWritten = Encoding.UTF8.GetBytes(value, _buffer[Position..]);
Position += bytesWritten;
_buffer[Position] = 0; // Null terminator
Position++;
}
/// <summary>
/// Writes end-of-document marker
/// Writes end-of-document marker
/// </summary>
public void WriteEndOfDocument()
{
_buffer[_position] = 0;
_position++;
_buffer[Position] = 0;
Position++;
}
/// <summary>
/// Writes a BSON string element
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The string value.</param>
public void WriteString(string name, string value)
/// <summary>
/// Writes a BSON string element
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The string value.</param>
public void WriteString(string name, string value)
{
WriteElementHeader(BsonType.String, name);
var valueBytes = Encoding.UTF8.GetByteCount(value);
var stringLength = valueBytes + 1; // Include null terminator
int valueBytes = Encoding.UTF8.GetByteCount(value);
int stringLength = valueBytes + 1; // Include null terminator
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), stringLength);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), stringLength);
Position += 4;
Encoding.UTF8.GetBytes(value, _buffer[_position..]);
_position += valueBytes;
Encoding.UTF8.GetBytes(value, _buffer[Position..]);
Position += valueBytes;
_buffer[_position] = 0; // Null terminator
_position++;
_buffer[Position] = 0; // Null terminator
Position++;
}
/// <summary>
/// Writes a BSON int32 element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 32-bit integer value.</param>
public void WriteInt32(string name, int value)
/// <summary>
/// Writes a BSON int32 element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 32-bit integer value.</param>
public void WriteInt32(string name, int value)
{
WriteElementHeader(BsonType.Int32, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), value);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), value);
Position += 4;
}
/// <summary>
/// Writes a BSON int64 element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 64-bit integer value.</param>
public void WriteInt64(string name, long value)
/// <summary>
/// Writes a BSON int64 element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 64-bit integer value.</param>
public void WriteInt64(string name, long value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value);
Position += 8;
}
/// <summary>
/// Writes a BSON double element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The double-precision value.</param>
public void WriteDouble(string name, double value)
/// <summary>
/// Writes a BSON double element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The double-precision value.</param>
public void WriteDouble(string name, double value)
{
WriteElementHeader(BsonType.Double, name);
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), value);
_position += 8;
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), value);
Position += 8;
}
/// <summary>
/// Writes spatial coordinates as a BSON array [X, Y].
/// Optimized for (double, double) tuples.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="coordinates">The coordinate tuple as (X, Y).</param>
public void WriteCoordinates(string name, (double, double) coordinates)
/// <summary>
/// Writes spatial coordinates as a BSON array [X, Y].
/// Optimized for (double, double) tuples.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="coordinates">The coordinate tuple as (X, Y).</param>
public void WriteCoordinates(string name, (double, double) coordinates)
{
WriteElementHeader(BsonType.Array, name);
var startPos = _position;
_position += 4; // Placeholder for array size
WriteElementHeader(BsonType.Array, name);
int startPos = Position;
Position += 4; // Placeholder for array size
// Element 0: X
_buffer[_position++] = (byte)BsonType.Double;
_buffer[_position++] = 0x30; // '0'
_buffer[_position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), coordinates.Item1);
_position += 8;
_buffer[Position++] = (byte)BsonType.Double;
_buffer[Position++] = 0x30; // '0'
_buffer[Position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), coordinates.Item1);
Position += 8;
// Element 1: Y
_buffer[_position++] = (byte)BsonType.Double;
_buffer[_position++] = 0x31; // '1'
_buffer[_position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), coordinates.Item2);
_position += 8;
_buffer[Position++] = (byte)BsonType.Double;
_buffer[Position++] = 0x31; // '1'
_buffer[Position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), coordinates.Item2);
Position += 8;
_buffer[_position++] = 0x00; // End of array marker
_buffer[Position++] = 0x00; // End of array marker
// Patch array size
var size = _position - startPos;
int size = Position - startPos;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(startPos, 4), size);
}
/// <summary>
/// Writes a BSON Decimal128 element from a <see cref="decimal"/> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The decimal value.</param>
public void WriteDecimal128(string name, decimal value)
/// <summary>
/// Writes a BSON Decimal128 element from a <see cref="decimal" /> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The decimal value.</param>
public void WriteDecimal128(string name, decimal value)
{
WriteElementHeader(BsonType.Decimal128, name);
// Note: usage of C# decimal bits for round-trip fidelity within ZB.MOM.WW.CBDD.
// This makes it compatible with CBDD Reader but strictly speaking not standard IEEE 754-2008 Decimal128.
var bits = decimal.GetBits(value);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), bits[0]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 4, 4), bits[1]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 8, 4), bits[2]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 12, 4), bits[3]);
_position += 16;
int[] bits = decimal.GetBits(value);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), bits[0]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 4, 4), bits[1]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 8, 4), bits[2]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 12, 4), bits[3]);
Position += 16;
}
/// <summary>
/// Writes a BSON boolean element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The boolean value.</param>
public void WriteBoolean(string name, bool value)
/// <summary>
/// Writes a BSON boolean element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The boolean value.</param>
public void WriteBoolean(string name, bool value)
{
WriteElementHeader(BsonType.Boolean, name);
_buffer[_position] = (byte)(value ? 1 : 0);
_position++;
_buffer[Position] = (byte)(value ? 1 : 0);
Position++;
}
/// <summary>
/// Writes a BSON UTC datetime element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date and time value.</param>
public void WriteDateTime(string name, DateTime value)
/// <summary>
/// Writes a BSON UTC datetime element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date and time value.</param>
public void WriteDateTime(string name, DateTime value)
{
WriteElementHeader(BsonType.DateTime, name);
var milliseconds = new DateTimeOffset(value.ToUniversalTime()).ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), milliseconds);
_position += 8;
long milliseconds = new DateTimeOffset(value.ToUniversalTime()).ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), milliseconds);
Position += 8;
}
/// <summary>
/// Writes a BSON UTC datetime element from a <see cref="DateTimeOffset"/> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date and time offset value.</param>
public void WriteDateTimeOffset(string name, DateTimeOffset value)
/// <summary>
/// Writes a BSON UTC datetime element from a <see cref="DateTimeOffset" /> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date and time offset value.</param>
public void WriteDateTimeOffset(string name, DateTimeOffset value)
{
WriteElementHeader(BsonType.DateTime, name);
var milliseconds = value.ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), milliseconds);
_position += 8;
long milliseconds = value.ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), milliseconds);
Position += 8;
}
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeSpan"/>.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time span value.</param>
public void WriteTimeSpan(string name, TimeSpan value)
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeSpan" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time span value.</param>
public void WriteTimeSpan(string name, TimeSpan value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value.Ticks);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value.Ticks);
Position += 8;
}
/// <summary>
/// Writes a BSON int32 element containing the <see cref="DateOnly.DayNumber"/>.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date-only value.</param>
public void WriteDateOnly(string name, DateOnly value)
/// <summary>
/// Writes a BSON int32 element containing the <see cref="DateOnly.DayNumber" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date-only value.</param>
public void WriteDateOnly(string name, DateOnly value)
{
WriteElementHeader(BsonType.Int32, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), value.DayNumber);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), value.DayNumber);
Position += 4;
}
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeOnly"/>.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time-only value.</param>
public void WriteTimeOnly(string name, TimeOnly value)
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeOnly" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time-only value.</param>
public void WriteTimeOnly(string name, TimeOnly value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value.Ticks);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value.Ticks);
Position += 8;
}
/// <summary>
/// Writes a GUID as a BSON string element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The GUID value.</param>
public void WriteGuid(string name, Guid value)
/// <summary>
/// Writes a GUID as a BSON string element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The GUID value.</param>
public void WriteGuid(string name, Guid value)
{
WriteString(name, value.ToString());
}
/// <summary>
/// Writes a BSON ObjectId element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The ObjectId value.</param>
public void WriteObjectId(string name, ObjectId value)
/// <summary>
/// Writes a BSON ObjectId element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The ObjectId value.</param>
public void WriteObjectId(string name, ObjectId value)
{
WriteElementHeader(BsonType.ObjectId, name);
value.WriteTo(_buffer.Slice(_position, 12));
_position += 12;
value.WriteTo(_buffer.Slice(Position, 12));
Position += 12;
}
/// <summary>
/// Writes a BSON null element.
/// </summary>
/// <param name="name">The field name.</param>
public void WriteNull(string name)
/// <summary>
/// Writes a BSON null element.
/// </summary>
/// <param name="name">The field name.</param>
public void WriteNull(string name)
{
WriteElementHeader(BsonType.Null, name);
// No value to write for null
}
/// <summary>
/// Writes binary data
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="data">The binary payload.</param>
/// <param name="subtype">The BSON binary subtype.</param>
public void WriteBinary(string name, ReadOnlySpan<byte> data, byte subtype = 0)
/// <summary>
/// Writes binary data
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="data">The binary payload.</param>
/// <param name="subtype">The BSON binary subtype.</param>
public void WriteBinary(string name, ReadOnlySpan<byte> data, byte subtype = 0)
{
WriteElementHeader(BsonType.Binary, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), data.Length);
_position += 4;
_buffer[_position] = subtype;
_position++;
data.CopyTo(_buffer[_position..]);
_position += data.Length;
WriteElementHeader(BsonType.Binary, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), data.Length);
Position += 4;
_buffer[Position] = subtype;
Position++;
data.CopyTo(_buffer[Position..]);
Position += data.Length;
}
/// <summary>
/// Begins writing a subdocument and returns the size position to patch later
/// </summary>
/// <param name="name">The field name for the subdocument.</param>
public int BeginDocument(string name)
/// <summary>
/// Begins writing a subdocument and returns the size position to patch later
/// </summary>
/// <param name="name">The field name for the subdocument.</param>
public int BeginDocument(string name)
{
WriteElementHeader(BsonType.Document, name);
return WriteDocumentSizePlaceholder();
}
/// <summary>
/// Begins writing the root document and returns the size position to patch later
/// Begins writing the root document and returns the size position to patch later
/// </summary>
public int BeginDocument()
{
return WriteDocumentSizePlaceholder();
}
/// <summary>
/// Ends the current document
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder"/>.</param>
public void EndDocument(int sizePosition)
/// <summary>
/// Ends the current document
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder" />.</param>
public void EndDocument(int sizePosition)
{
WriteEndOfDocument();
PatchDocumentSize(sizePosition);
}
/// <summary>
/// Begins writing a BSON array and returns the size position to patch later
/// </summary>
/// <param name="name">The field name for the array.</param>
public int BeginArray(string name)
/// <summary>
/// Begins writing a BSON array and returns the size position to patch later
/// </summary>
/// <param name="name">The field name for the array.</param>
public int BeginArray(string name)
{
WriteElementHeader(BsonType.Array, name);
return WriteDocumentSizePlaceholder();
}
/// <summary>
/// Ends the current BSON array
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder"/>.</param>
public void EndArray(int sizePosition)
/// <summary>
/// Ends the current BSON array
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder" />.</param>
public void EndArray(int sizePosition)
{
WriteEndOfDocument();
PatchDocumentSize(sizePosition);
}
}
}

19
src/CBDD.Bson/Metadata/Attributes.cs
View File

@@ -1,14 +1,11 @@
using System;
namespace ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Bson
[AttributeUsage(AttributeTargets.Property)]
public class BsonIdAttribute : Attribute
{
[AttributeUsage(AttributeTargets.Property)]
public class BsonIdAttribute : Attribute
{
}
[AttributeUsage(AttributeTargets.Property)]
public class BsonIgnoreAttribute : Attribute
{
}
}
[AttributeUsage(AttributeTargets.Property)]
public class BsonIgnoreAttribute : Attribute
{
}

115
src/CBDD.Bson/Schema/BsonField.cs
View File

@@ -1,59 +1,56 @@
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public partial class BsonField
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public class BsonField
{
/// <summary>
/// Gets the field name.
/// Gets the field name.
/// </summary>
public required string Name { get; init; }
/// <summary>
/// Gets the field BSON type.
/// Gets the field BSON type.
/// </summary>
public BsonType Type { get; init; }
/// <summary>
/// Gets a value indicating whether the field is nullable.
/// Gets a value indicating whether the field is nullable.
/// </summary>
public bool IsNullable { get; init; }
/// <summary>
/// Gets the nested schema when this field is a document.
/// Gets the nested schema when this field is a document.
/// </summary>
public BsonSchema? NestedSchema { get; init; }
/// <summary>
/// Gets the array item type when this field is an array.
/// Gets the array item type when this field is an array.
/// </summary>
public BsonType? ArrayItemType { get; init; }
/// <summary>
/// Writes this field definition to BSON.
/// Writes this field definition to BSON.
/// </summary>
/// <param name="writer">The BSON writer.</param>
public void ToBson(ref BsonSpanWriter writer)
{
var size = writer.BeginDocument();
writer.WriteString("n", Name);
writer.WriteInt32("t", (int)Type);
int size = writer.BeginDocument();
writer.WriteString("n", Name);
writer.WriteInt32("t", (int)Type);
writer.WriteBoolean("b", IsNullable);
if (NestedSchema != null)
{
writer.WriteElementHeader(BsonType.Document, "s");
NestedSchema.ToBson(ref writer);
if (NestedSchema != null)
{
writer.WriteElementHeader(BsonType.Document, "s");
NestedSchema.ToBson(ref writer);
}
if (ArrayItemType != null)
{
writer.WriteInt32("a", (int)ArrayItemType.Value);
}
if (ArrayItemType != null) writer.WriteInt32("a", (int)ArrayItemType.Value);
writer.EndDocument(size);
}
/// <summary>
/// Reads a field definition from BSON.
/// Reads a field definition from BSON.
/// </summary>
/// <param name="reader">The BSON reader.</param>
/// <returns>The deserialized field.</returns>
@@ -61,59 +58,59 @@ public partial class BsonField
{
reader.ReadInt32(); // Read doc size
string name = "";
BsonType type = BsonType.Null;
bool isNullable = false;
BsonSchema? nestedSchema = null;
BsonType? arrayItemType = null;
while (reader.Remaining > 1)
{
var btype = reader.ReadBsonType();
var name = "";
var type = BsonType.Null;
var isNullable = false;
BsonSchema? nestedSchema = null;
BsonType? arrayItemType = null;
while (reader.Remaining > 1)
{
var btype = reader.ReadBsonType();
if (btype == BsonType.EndOfDocument) break;
var key = reader.ReadElementHeader();
switch (key)
{
case "n": name = reader.ReadString(); break;
case "t": type = (BsonType)reader.ReadInt32(); break;
case "b": isNullable = reader.ReadBoolean(); break;
case "s": nestedSchema = BsonSchema.FromBson(ref reader); break;
case "a": arrayItemType = (BsonType)reader.ReadInt32(); break;
default: reader.SkipValue(btype); break;
}
string key = reader.ReadElementHeader();
switch (key)
{
case "n": name = reader.ReadString(); break;
case "t": type = (BsonType)reader.ReadInt32(); break;
case "b": isNullable = reader.ReadBoolean(); break;
case "s": nestedSchema = BsonSchema.FromBson(ref reader); break;
case "a": arrayItemType = (BsonType)reader.ReadInt32(); break;
default: reader.SkipValue(btype); break;
}
}
return new BsonField
{
Name = name,
Type = type,
IsNullable = isNullable,
NestedSchema = nestedSchema,
return new BsonField
{
Name = name,
Type = type,
IsNullable = isNullable,
NestedSchema = nestedSchema,
ArrayItemType = arrayItemType
};
}
/// <summary>
/// Computes a hash representing the field definition.
/// Computes a hash representing the field definition.
/// </summary>
/// <returns>The computed hash value.</returns>
public long GetHash()
{
var hash = new HashCode();
hash.Add(Name);
hash.Add((int)Type);
hash.Add(IsNullable);
hash.Add(ArrayItemType);
hash.Add(Name);
hash.Add((int)Type);
hash.Add(IsNullable);
hash.Add(ArrayItemType);
if (NestedSchema != null) hash.Add(NestedSchema.GetHash());
return hash.ToHashCode();
}
/// <summary>
/// Determines whether this field is equal to another field.
/// Determines whether this field is equal to another field.
/// </summary>
/// <param name="other">The other field.</param>
/// <returns><see langword="true"/> if the fields are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the fields are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(BsonField? other)
{
if (other == null) return false;
@@ -121,8 +118,14 @@ public partial class BsonField
}
/// <inheritdoc />
public override bool Equals(object? obj) => Equals(obj as BsonField);
public override bool Equals(object? obj)
{
return Equals(obj as BsonField);
}
/// <inheritdoc />
public override int GetHashCode() => (int)GetHash();
}
public override int GetHashCode()
{
return (int)GetHash();
}
}

127
src/CBDD.Bson/Schema/BsonSchema.cs
View File

@@ -1,45 +1,46 @@
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public partial class BsonSchema
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public class BsonSchema
{
/// <summary>
/// Gets or sets the schema title.
/// Gets or sets the schema title.
/// </summary>
public string? Title { get; set; }
/// <summary>
/// Gets or sets the schema version.
/// Gets or sets the schema version.
/// </summary>
public int? Version { get; set; }
/// <summary>
/// Gets the schema fields.
/// Gets the schema fields.
/// </summary>
public List<BsonField> Fields { get; } = new();
/// <summary>
/// Serializes this schema instance to BSON.
/// Serializes this schema instance to BSON.
/// </summary>
/// <param name="writer">The BSON writer to write into.</param>
public void ToBson(ref BsonSpanWriter writer)
{
var size = writer.BeginDocument();
if (Title != null) writer.WriteString("t", Title);
int size = writer.BeginDocument();
if (Title != null) writer.WriteString("t", Title);
if (Version != null) writer.WriteInt32("_v", Version.Value);
var fieldsSize = writer.BeginArray("f");
for (int i = 0; i < Fields.Count; i++)
{
writer.WriteElementHeader(BsonType.Document, i.ToString());
Fields[i].ToBson(ref writer);
}
int fieldsSize = writer.BeginArray("f");
for (var i = 0; i < Fields.Count; i++)
{
writer.WriteElementHeader(BsonType.Document, i.ToString());
Fields[i].ToBson(ref writer);
}
writer.EndArray(fieldsSize);
writer.EndDocument(size);
}
/// <summary>
/// Deserializes a schema instance from BSON.
/// Deserializes a schema instance from BSON.
/// </summary>
/// <param name="reader">The BSON reader to read from.</param>
/// <returns>The deserialized schema.</returns>
@@ -47,55 +48,53 @@ public partial class BsonSchema
{
reader.ReadInt32(); // Read doc size
var schema = new BsonSchema();
while (reader.Remaining > 1)
{
var btype = reader.ReadBsonType();
var schema = new BsonSchema();
while (reader.Remaining > 1)
{
var btype = reader.ReadBsonType();
if (btype == BsonType.EndOfDocument) break;
var key = reader.ReadElementHeader();
switch (key)
{
case "t": schema.Title = reader.ReadString(); break;
case "_v": schema.Version = reader.ReadInt32(); break;
case "f":
reader.ReadInt32(); // array size
while (reader.Remaining > 1)
{
var itemType = reader.ReadBsonType();
if (itemType == BsonType.EndOfDocument) break;
reader.ReadElementHeader(); // index
schema.Fields.Add(BsonField.FromBson(ref reader));
}
break;
default: reader.SkipValue(btype); break;
}
string key = reader.ReadElementHeader();
switch (key)
{
case "t": schema.Title = reader.ReadString(); break;
case "_v": schema.Version = reader.ReadInt32(); break;
case "f":
reader.ReadInt32(); // array size
while (reader.Remaining > 1)
{
var itemType = reader.ReadBsonType();
if (itemType == BsonType.EndOfDocument) break;
reader.ReadElementHeader(); // index
schema.Fields.Add(BsonField.FromBson(ref reader));
}
break;
default: reader.SkipValue(btype); break;
}
}
return schema;
}
/// <summary>
/// Computes a hash value for this schema based on its contents.
/// Computes a hash value for this schema based on its contents.
/// </summary>
/// <returns>The computed hash value.</returns>
public long GetHash()
{
var hash = new HashCode();
hash.Add(Title);
foreach (var field in Fields)
{
hash.Add(field.GetHash());
}
var hash = new HashCode();
hash.Add(Title);
foreach (var field in Fields) hash.Add(field.GetHash());
return hash.ToHashCode();
}
/// <summary>
/// Determines whether this schema is equal to another schema.
/// Determines whether this schema is equal to another schema.
/// </summary>
/// <param name="other">The schema to compare with.</param>
/// <returns><see langword="true"/> when schemas are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when schemas are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(BsonSchema? other)
{
if (other == null) return false;
@@ -103,27 +102,29 @@ public partial class BsonSchema
}
/// <inheritdoc />
public override bool Equals(object? obj) => Equals(obj as BsonSchema);
public override bool Equals(object? obj)
{
return Equals(obj as BsonSchema);
}
/// <inheritdoc />
public override int GetHashCode() => (int)GetHash();
public override int GetHashCode()
{
return (int)GetHash();
}
/// <summary>
/// Enumerates all field keys in this schema, including nested schema keys.
/// Enumerates all field keys in this schema, including nested schema keys.
/// </summary>
/// <returns>An enumerable of field keys.</returns>
public IEnumerable<string> GetAllKeys()
{
foreach (var field in Fields)
{
yield return field.Name;
if (field.NestedSchema != null)
{
foreach (var nestedKey in field.NestedSchema.GetAllKeys())
{
yield return nestedKey;
}
}
}
}
}
foreach (var field in Fields)
{
yield return field.Name;
if (field.NestedSchema != null)
foreach (string nestedKey in field.NestedSchema.GetAllKeys())
yield return nestedKey;
}
}
}

104
src/CBDD.Bson/Types/ObjectId.cs
View File

@@ -1,11 +1,10 @@
using System;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// 12-byte ObjectId compatible with MongoDB ObjectId.
/// Implemented as readonly struct for zero allocation.
/// 12-byte ObjectId compatible with MongoDB ObjectId.
/// Implemented as readonly struct for zero allocation.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 12)]
public readonly struct ObjectId : IEquatable<ObjectId>
@@ -14,20 +13,20 @@ public readonly struct ObjectId : IEquatable<ObjectId>
[FieldOffset(4)] private readonly long _randomAndCounter;
/// <summary>
/// Empty ObjectId (all zeros)
/// Empty ObjectId (all zeros)
/// </summary>
public static readonly ObjectId Empty = new ObjectId(0, 0);
public static readonly ObjectId Empty = new(0, 0);
/// <summary>
/// Maximum ObjectId (all 0xFF bytes) - useful for range queries
/// Maximum ObjectId (all 0xFF bytes) - useful for range queries
/// </summary>
public static readonly ObjectId MaxValue = new ObjectId(int.MaxValue, long.MaxValue);
public static readonly ObjectId MaxValue = new(int.MaxValue, long.MaxValue);
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId"/> struct from raw bytes.
/// </summary>
/// <param name="bytes">The 12-byte ObjectId value.</param>
public ObjectId(ReadOnlySpan<byte> bytes)
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId" /> struct from raw bytes.
/// </summary>
/// <param name="bytes">The 12-byte ObjectId value.</param>
public ObjectId(ReadOnlySpan<byte> bytes)
{
if (bytes.Length != 12)
throw new ArgumentException("ObjectId must be exactly 12 bytes", nameof(bytes));
@@ -36,32 +35,32 @@ public readonly struct ObjectId : IEquatable<ObjectId>
_randomAndCounter = BitConverter.ToInt64(bytes[4..12]);
}
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId"/> struct from its components.
/// </summary>
/// <param name="timestamp">The Unix timestamp portion.</param>
/// <param name="randomAndCounter">The random and counter portion.</param>
public ObjectId(int timestamp, long randomAndCounter)
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId" /> struct from its components.
/// </summary>
/// <param name="timestamp">The Unix timestamp portion.</param>
/// <param name="randomAndCounter">The random and counter portion.</param>
public ObjectId(int timestamp, long randomAndCounter)
{
_timestamp = timestamp;
_randomAndCounter = randomAndCounter;
}
/// <summary>
/// Creates a new ObjectId with current timestamp
/// Creates a new ObjectId with current timestamp
/// </summary>
public static ObjectId NewObjectId()
{
var timestamp = (int)DateTimeOffset.UtcNow.ToUnixTimeSeconds();
var random = Random.Shared.NextInt64();
long random = Random.Shared.NextInt64();
return new ObjectId(timestamp, random);
}
/// <summary>
/// Writes the ObjectId to the destination span (must be 12 bytes)
/// </summary>
/// <param name="destination">The destination span to write into.</param>
public void WriteTo(Span<byte> destination)
/// <summary>
/// Writes the ObjectId to the destination span (must be 12 bytes)
/// </summary>
/// <param name="destination">The destination span to write into.</param>
public void WriteTo(Span<byte> destination)
{
if (destination.Length < 12)
throw new ArgumentException("Destination must be at least 12 bytes", nameof(destination));
@@ -71,7 +70,7 @@ public readonly struct ObjectId : IEquatable<ObjectId>
}
/// <summary>
/// Converts ObjectId to byte array
/// Converts ObjectId to byte array
/// </summary>
public byte[] ToByteArray()
{
@@ -81,32 +80,47 @@ public readonly struct ObjectId : IEquatable<ObjectId>
}
/// <summary>
/// Gets timestamp portion as UTC DateTime
/// Gets timestamp portion as UTC DateTime
/// </summary>
public DateTime Timestamp => DateTimeOffset.FromUnixTimeSeconds(_timestamp).UtcDateTime;
/// <summary>
/// Determines whether this instance and another <see cref="ObjectId"/> have the same value.
/// </summary>
/// <param name="other">The object to compare with this instance.</param>
/// <returns><see langword="true"/> if the values are equal; otherwise, <see langword="false"/>.</returns>
public bool Equals(ObjectId other) =>
_timestamp == other._timestamp && _randomAndCounter == other._randomAndCounter;
/// <inheritdoc />
public override bool Equals(object? obj) => obj is ObjectId other && Equals(other);
/// <inheritdoc />
public override int GetHashCode() => HashCode.Combine(_timestamp, _randomAndCounter);
/// <summary>
/// Determines whether this instance and another <see cref="ObjectId" /> have the same value.
/// </summary>
/// <param name="other">The object to compare with this instance.</param>
/// <returns><see langword="true" /> if the values are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(ObjectId other)
{
return _timestamp == other._timestamp && _randomAndCounter == other._randomAndCounter;
}
public static bool operator ==(ObjectId left, ObjectId right) => left.Equals(right);
public static bool operator !=(ObjectId left, ObjectId right) => !left.Equals(right);
/// <inheritdoc />
public override bool Equals(object? obj)
{
return obj is ObjectId other && Equals(other);
}
/// <inheritdoc />
public override string ToString()
/// <inheritdoc />
public override int GetHashCode()
{
return HashCode.Combine(_timestamp, _randomAndCounter);
}
public static bool operator ==(ObjectId left, ObjectId right)
{
return left.Equals(right);
}
public static bool operator !=(ObjectId left, ObjectId right)
{
return !left.Equals(right);
}
/// <inheritdoc />
public override string ToString()
{
Span<byte> bytes = stackalloc byte[12];
WriteTo(bytes);
return Convert.ToHexString(bytes).ToLowerInvariant();
}
}
}

46
src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj
View File

@@ -1,28 +1,28 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Bson</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Bson</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PackageId>ZB.MOM.WW.CBDD.Bson</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>BSON Serialization Library for High-Performance Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Bson</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Bson</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<PackageId>ZB.MOM.WW.CBDD.Bson</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>BSON Serialization Library for High-Performance Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

156
src/CBDD.Core/CDC/ChangeStreamDispatcher.cs
View File

@@ -1,23 +1,21 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Channels;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.CDC;
internal sealed class ChangeStreamDispatcher : IDisposable
{
internal sealed class ChangeStreamDispatcher : IDisposable
{
private readonly Channel<InternalChangeEvent> _channel;
private readonly ConcurrentDictionary<string, ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>> _subscriptions = new();
private readonly ConcurrentDictionary<string, int> _payloadWatcherCounts = new();
private readonly CancellationTokenSource _cts = new();
private readonly ConcurrentDictionary<string, int> _payloadWatcherCounts = new();
/// <summary>
/// Initializes a new change stream dispatcher.
/// </summary>
public ChangeStreamDispatcher()
private readonly ConcurrentDictionary<string, ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>>
_subscriptions = new();
/// <summary>
/// Initializes a new change stream dispatcher.
/// </summary>
public ChangeStreamDispatcher()
{
_channel = Channel.CreateUnbounded<InternalChangeEvent>(new UnboundedChannelOptions
{
@@ -28,50 +26,57 @@ internal sealed class ChangeStreamDispatcher : IDisposable
Task.Run(ProcessEventsAsync);
}
/// <summary>
/// Publishes a change event to subscribers.
/// </summary>
/// <param name="change">The change event to publish.</param>
public void Publish(InternalChangeEvent change)
/// <summary>
/// Releases dispatcher resources.
/// </summary>
public void Dispose()
{
_cts.Cancel();
_cts.Dispose();
}
/// <summary>
/// Publishes a change event to subscribers.
/// </summary>
/// <param name="change">The change event to publish.</param>
public void Publish(InternalChangeEvent change)
{
_channel.Writer.TryWrite(change);
}
/// <summary>
/// Determines whether a collection has subscribers that require payloads.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true"/> if payload watchers exist; otherwise, <see langword="false"/>.</returns>
public bool HasPayloadWatchers(string collectionName)
/// <summary>
/// Determines whether a collection has subscribers that require payloads.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true" /> if payload watchers exist; otherwise, <see langword="false" />.</returns>
public bool HasPayloadWatchers(string collectionName)
{
return _payloadWatcherCounts.TryGetValue(collectionName, out var count) && count > 0;
return _payloadWatcherCounts.TryGetValue(collectionName, out int count) && count > 0;
}
/// <summary>
/// Determines whether a collection has any subscribers.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true"/> if subscribers exist; otherwise, <see langword="false"/>.</returns>
public bool HasAnyWatchers(string collectionName)
/// <summary>
/// Determines whether a collection has any subscribers.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true" /> if subscribers exist; otherwise, <see langword="false" />.</returns>
public bool HasAnyWatchers(string collectionName)
{
return _subscriptions.TryGetValue(collectionName, out var subs) && !subs.IsEmpty;
}
/// <summary>
/// Subscribes a channel writer to collection change events.
/// </summary>
/// <param name="collectionName">The collection name to subscribe to.</param>
/// <param name="capturePayload">Whether this subscriber requires event payloads.</param>
/// <param name="writer">The destination channel writer.</param>
/// <returns>An <see cref="IDisposable"/> that removes the subscription when disposed.</returns>
public IDisposable Subscribe(string collectionName, bool capturePayload, ChannelWriter<InternalChangeEvent> writer)
/// <summary>
/// Subscribes a channel writer to collection change events.
/// </summary>
/// <param name="collectionName">The collection name to subscribe to.</param>
/// <param name="capturePayload">Whether this subscriber requires event payloads.</param>
/// <param name="writer">The destination channel writer.</param>
/// <returns>An <see cref="IDisposable" /> that removes the subscription when disposed.</returns>
public IDisposable Subscribe(string collectionName, bool capturePayload, ChannelWriter<InternalChangeEvent> writer)
{
if (capturePayload)
{
_payloadWatcherCounts.AddOrUpdate(collectionName, 1, (_, count) => count + 1);
}
if (capturePayload) _payloadWatcherCounts.AddOrUpdate(collectionName, 1, (_, count) => count + 1);
var collectionSubs = _subscriptions.GetOrAdd(collectionName, _ => new ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>());
var collectionSubs = _subscriptions.GetOrAdd(collectionName,
_ => new ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>());
collectionSubs.TryAdd(writer, 0);
return new Subscription(() => Unsubscribe(collectionName, capturePayload, writer));
@@ -79,15 +84,9 @@ internal sealed class ChangeStreamDispatcher : IDisposable
private void Unsubscribe(string collectionName, bool capturePayload, ChannelWriter<InternalChangeEvent> writer)
{
if (_subscriptions.TryGetValue(collectionName, out var collectionSubs))
{
collectionSubs.TryRemove(writer, out _);
}
if (_subscriptions.TryGetValue(collectionName, out var collectionSubs)) collectionSubs.TryRemove(writer, out _);
if (capturePayload)
{
_payloadWatcherCounts.AddOrUpdate(collectionName, 0, (_, count) => Math.Max(0, count - 1));
}
if (capturePayload) _payloadWatcherCounts.AddOrUpdate(collectionName, 0, (_, count) => Math.Max(0, count - 1));
}
private async Task ProcessEventsAsync()
@@ -96,60 +95,45 @@ internal sealed class ChangeStreamDispatcher : IDisposable
{
var reader = _channel.Reader;
while (await reader.WaitToReadAsync(_cts.Token))
{
while (reader.TryRead(out var @event))
{
if (_subscriptions.TryGetValue(@event.CollectionName, out var collectionSubs))
{
foreach (var writer in collectionSubs.Keys)
{
// Optimized fan-out: non-blocking TryWrite.
// If a subscriber channel is full (unlikely with Unbounded),
// we skip or drop. Usually, subscribers will also use Unbounded.
writer.TryWrite(@event);
}
}
}
}
while (reader.TryRead(out var @event))
if (_subscriptions.TryGetValue(@event.CollectionName, out var collectionSubs))
foreach (var writer in collectionSubs.Keys)
// Optimized fan-out: non-blocking TryWrite.
// If a subscriber channel is full (unlikely with Unbounded),
// we skip or drop. Usually, subscribers will also use Unbounded.
writer.TryWrite(@event);
}
catch (OperationCanceledException)
{
}
catch (OperationCanceledException) { }
catch (Exception)
{
// Internal error logging could go here
}
}
/// <summary>
/// Releases dispatcher resources.
/// </summary>
public void Dispose()
{
_cts.Cancel();
_cts.Dispose();
}
private sealed class Subscription : IDisposable
{
private readonly Action _onDispose;
private bool _disposed;
/// <summary>
/// Initializes a new subscription token.
/// </summary>
/// <param name="onDispose">Callback executed when the subscription is disposed.</param>
public Subscription(Action onDispose)
/// <summary>
/// Initializes a new subscription token.
/// </summary>
/// <param name="onDispose">Callback executed when the subscription is disposed.</param>
public Subscription(Action onDispose)
{
_onDispose = onDispose;
}
/// <summary>
/// Disposes the subscription and unregisters the subscriber.
/// </summary>
public void Dispose()
/// <summary>
/// Disposes the subscription and unregisters the subscriber.
/// </summary>
public void Dispose()
{
if (_disposed) return;
_onDispose();
_disposed = true;
}
}
}
}

67
src/CBDD.Core/CDC/ChangeStreamEvent.cs
View File

@@ -1,76 +1,75 @@
using System;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.CDC;
/// <summary>
/// A generic, immutable struct representing a data change in a collection.
/// </summary>
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.CDC;
/// <summary>
/// A generic, immutable struct representing a data change in a collection.
/// </summary>
public readonly struct ChangeStreamEvent<TId, T> where T : class
{
/// <summary>
/// Gets the UTC timestamp when the change was recorded.
/// Gets the UTC timestamp when the change was recorded.
/// </summary>
public long Timestamp { get; init; }
/// <summary>
/// Gets the transaction identifier that produced the change.
/// Gets the transaction identifier that produced the change.
/// </summary>
public ulong TransactionId { get; init; }
/// <summary>
/// Gets the collection name where the change occurred.
/// Gets the collection name where the change occurred.
/// </summary>
public string CollectionName { get; init; }
/// <summary>
/// Gets the operation type associated with the change.
/// Gets the operation type associated with the change.
/// </summary>
public OperationType Type { get; init; }
/// <summary>
/// Gets the changed document identifier.
/// Gets the changed document identifier.
/// </summary>
public TId DocumentId { get; init; }
/// <summary>
/// The deserialized entity. Null if capturePayload was false during Watch().
/// </summary>
public T? Entity { get; init; }
}
/// <summary>
/// Low-level event structure used internally to transport changes before deserialization.
/// </summary>
/// <summary>
/// The deserialized entity. Null if capturePayload was false during Watch().
/// </summary>
public T? Entity { get; init; }
}
/// <summary>
/// Low-level event structure used internally to transport changes before deserialization.
/// </summary>
internal readonly struct InternalChangeEvent
{
/// <summary>
/// Gets the UTC timestamp when the change was recorded.
/// Gets the UTC timestamp when the change was recorded.
/// </summary>
public long Timestamp { get; init; }
/// <summary>
/// Gets the transaction identifier that produced the change.
/// Gets the transaction identifier that produced the change.
/// </summary>
public ulong TransactionId { get; init; }
/// <summary>
/// Gets the collection name where the change occurred.
/// Gets the collection name where the change occurred.
/// </summary>
public string CollectionName { get; init; }
/// <summary>
/// Gets the operation type associated with the change.
/// Gets the operation type associated with the change.
/// </summary>
public OperationType Type { get; init; }
/// <summary>
/// Raw BSON of the Document ID.
/// </summary>
/// <summary>
/// Raw BSON of the Document ID.
/// </summary>
public ReadOnlyMemory<byte> IdBytes { get; init; }
/// <summary>
/// Raw BSON of the Entity. Null if payload not captured.
/// </summary>
public ReadOnlyMemory<byte>? PayloadBytes { get; init; }
}
/// <summary>
/// Raw BSON of the Entity. Null if payload not captured.
/// </summary>
public ReadOnlyMemory<byte>? PayloadBytes { get; init; }
}

166
src/CBDD.Core/CDC/ChangeStreamObservable.cs
View File

@@ -1,49 +1,45 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading.Channels;
using System.Threading.Tasks;
using System.Threading;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.CDC;
internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamEvent<TId, T>> where T : class
{
private readonly ChangeStreamDispatcher _dispatcher;
private readonly string _collectionName;
internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamEvent<TId, T>> where T : class
{
private readonly bool _capturePayload;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly string _collectionName;
private readonly ChangeStreamDispatcher _dispatcher;
private readonly ConcurrentDictionary<ushort, string> _keyReverseMap;
private readonly IDocumentMapper<TId, T> _mapper;
/// <summary>
/// Initializes a new observable wrapper for collection change events.
/// </summary>
/// <param name="dispatcher">The dispatcher producing internal change events.</param>
/// <param name="collectionName">The collection to subscribe to.</param>
/// <param name="capturePayload">Whether full entity payloads should be included.</param>
/// <param name="mapper">The document mapper used for ID and payload deserialization.</param>
/// <param name="keyReverseMap">The key reverse map used by BSON readers.</param>
public ChangeStreamObservable(
ChangeStreamDispatcher dispatcher,
string collectionName,
bool capturePayload,
IDocumentMapper<TId, T> mapper,
/// <summary>
/// Initializes a new observable wrapper for collection change events.
/// </summary>
/// <param name="dispatcher">The dispatcher producing internal change events.</param>
/// <param name="collectionName">The collection to subscribe to.</param>
/// <param name="capturePayload">Whether full entity payloads should be included.</param>
/// <param name="mapper">The document mapper used for ID and payload deserialization.</param>
/// <param name="keyReverseMap">The key reverse map used by BSON readers.</param>
public ChangeStreamObservable(
ChangeStreamDispatcher dispatcher,
string collectionName,
bool capturePayload,
IDocumentMapper<TId, T> mapper,
ConcurrentDictionary<ushort, string> keyReverseMap)
{
_dispatcher = dispatcher;
_collectionName = collectionName;
_capturePayload = capturePayload;
_mapper = mapper;
_keyReverseMap = keyReverseMap;
}
/// <inheritdoc />
public IDisposable Subscribe(IObserver<ChangeStreamEvent<TId, T>> observer)
{
if (observer == null) throw new ArgumentNullException(nameof(observer));
_mapper = mapper;
_keyReverseMap = keyReverseMap;
}
/// <inheritdoc />
public IDisposable Subscribe(IObserver<ChangeStreamEvent<TId, T>> observer)
{
if (observer == null) throw new ArgumentNullException(nameof(observer));
var cts = new CancellationTokenSource();
var channel = Channel.CreateUnbounded<InternalChangeEvent>(new UnboundedChannelOptions
@@ -60,46 +56,43 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
return new CompositeDisposable(dispatcherSubscription, cts, channel.Writer, bridgeTask);
}
private async Task BridgeChannelToObserverAsync(ChannelReader<InternalChangeEvent> reader, IObserver<ChangeStreamEvent<TId, T>> observer, CancellationToken ct)
private async Task BridgeChannelToObserverAsync(ChannelReader<InternalChangeEvent> reader,
IObserver<ChangeStreamEvent<TId, T>> observer, CancellationToken ct)
{
try
{
while (await reader.WaitToReadAsync(ct))
{
while (reader.TryRead(out var internalEvent))
while (reader.TryRead(out var internalEvent))
try
{
try
{
// Deserializza ID
var eventId = _mapper.FromIndexKey(new IndexKey(internalEvent.IdBytes.ToArray()));
// Deserializza Payload (se presente)
T? entity = default;
if (internalEvent.PayloadBytes.HasValue)
{
entity = _mapper.Deserialize(new BsonSpanReader(internalEvent.PayloadBytes.Value.Span, _keyReverseMap));
}
// Deserializza ID
var eventId = _mapper.FromIndexKey(new IndexKey(internalEvent.IdBytes.ToArray()));
var externalEvent = new ChangeStreamEvent<TId, T>
{
Timestamp = internalEvent.Timestamp,
TransactionId = internalEvent.TransactionId,
CollectionName = internalEvent.CollectionName,
Type = internalEvent.Type,
DocumentId = eventId,
Entity = entity
};
// Deserializza Payload (se presente)
T? entity = default;
if (internalEvent.PayloadBytes.HasValue)
entity = _mapper.Deserialize(new BsonSpanReader(internalEvent.PayloadBytes.Value.Span,
_keyReverseMap));
observer.OnNext(externalEvent);
}
catch (Exception ex)
var externalEvent = new ChangeStreamEvent<TId, T>
{
// In case of deserialization error, we notify and continue if possible
// Or we can stop the observer.
observer.OnError(ex);
}
Timestamp = internalEvent.Timestamp,
TransactionId = internalEvent.TransactionId,
CollectionName = internalEvent.CollectionName,
Type = internalEvent.Type,
DocumentId = eventId,
Entity = entity
};
observer.OnNext(externalEvent);
}
}
catch (Exception ex)
{
// In case of deserialization error, we notify and continue if possible
// Or we can stop the observer.
observer.OnError(ex);
}
observer.OnCompleted();
}
catch (OperationCanceledException)
@@ -112,33 +105,34 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
}
}
private sealed class CompositeDisposable : IDisposable
{
private readonly IDisposable _dispatcherSubscription;
private readonly CancellationTokenSource _cts;
private readonly ChannelWriter<InternalChangeEvent> _writer;
private sealed class CompositeDisposable : IDisposable
{
private readonly Task _bridgeTask;
private readonly CancellationTokenSource _cts;
private readonly IDisposable _dispatcherSubscription;
private readonly ChannelWriter<InternalChangeEvent> _writer;
private bool _disposed;
/// <summary>
/// Initializes a new disposable wrapper for change stream resources.
/// </summary>
/// <param name="dispatcherSubscription">The dispatcher subscription handle.</param>
/// <param name="cts">The cancellation source controlling the bridge task.</param>
/// <param name="writer">The channel writer for internal change events.</param>
/// <param name="bridgeTask">The running bridge task.</param>
public CompositeDisposable(IDisposable dispatcherSubscription, CancellationTokenSource cts, ChannelWriter<InternalChangeEvent> writer, Task bridgeTask)
{
_dispatcherSubscription = dispatcherSubscription;
_cts = cts;
_writer = writer;
_bridgeTask = bridgeTask;
}
/// <inheritdoc />
public void Dispose()
{
if (_disposed) return;
/// <summary>
/// Initializes a new disposable wrapper for change stream resources.
/// </summary>
/// <param name="dispatcherSubscription">The dispatcher subscription handle.</param>
/// <param name="cts">The cancellation source controlling the bridge task.</param>
/// <param name="writer">The channel writer for internal change events.</param>
/// <param name="bridgeTask">The running bridge task.</param>
public CompositeDisposable(IDisposable dispatcherSubscription, CancellationTokenSource cts,
ChannelWriter<InternalChangeEvent> writer, Task bridgeTask)
{
_dispatcherSubscription = dispatcherSubscription;
_cts = cts;
_writer = writer;
_bridgeTask = bridgeTask;
}
/// <inheritdoc />
public void Dispose()
{
if (_disposed) return;
_disposed = true;
_dispatcherSubscription.Dispose();
@@ -147,4 +141,4 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
_cts.Dispose();
}
}
}
}

26
src/CBDD.Core/CDC/CollectionCdcPublisher.cs
View File

@@ -1,26 +1,25 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.CDC;
/// <summary>
/// Handles CDC watch/notify behavior for a single collection.
/// Extracted from DocumentCollection to keep storage/query concerns separated from event plumbing.
/// Handles CDC watch/notify behavior for a single collection.
/// Extracted from DocumentCollection to keep storage/query concerns separated from event plumbing.
/// </summary>
/// <typeparam name="TId">Document identifier type.</typeparam>
/// <typeparam name="T">Document type.</typeparam>
internal sealed class CollectionCdcPublisher<TId, T> where T : class
{
private readonly ITransactionHolder _transactionHolder;
private readonly string _collectionName;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly ChangeStreamDispatcher? _dispatcher;
private readonly ConcurrentDictionary<ushort, string> _keyReverseMap;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly ITransactionHolder _transactionHolder;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionCdcPublisher"/> class.
/// Initializes a new instance of the <see cref="CollectionCdcPublisher" /> class.
/// </summary>
/// <param name="transactionHolder">The transaction holder.</param>
/// <param name="collectionName">The collection name.</param>
@@ -42,7 +41,7 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
}
/// <summary>
/// Executes Watch.
/// Executes Watch.
/// </summary>
/// <param name="capturePayload">Whether to include payload data.</param>
public IObservable<ChangeStreamEvent<TId, T>> Watch(bool capturePayload = false)
@@ -59,7 +58,7 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
}
/// <summary>
/// Executes Notify.
/// Executes Notify.
/// </summary>
/// <param name="type">The operation type.</param>
/// <param name="id">The document identifier.</param>
@@ -74,15 +73,11 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
return;
ReadOnlyMemory<byte>? payload = null;
if (!docData.IsEmpty && _dispatcher.HasPayloadWatchers(_collectionName))
{
payload = docData.ToArray();
}
if (!docData.IsEmpty && _dispatcher.HasPayloadWatchers(_collectionName)) payload = docData.ToArray();
var idBytes = _mapper.ToIndexKey(id).Data.ToArray();
byte[] idBytes = _mapper.ToIndexKey(id).Data.ToArray();
if (transaction is Transaction t)
{
t.AddChange(new InternalChangeEvent
{
Timestamp = DateTime.UtcNow.Ticks,
@@ -92,6 +87,5 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
IdBytes = idBytes,
PayloadBytes = payload
});
}
}
}
}

129
src/CBDD.Core/Collections/BaseMappers.cs
View File

@@ -1,25 +1,21 @@
using System;
using System.Buffers;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using System.Linq;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson.Schema;
namespace ZB.MOM.WW.CBDD.Core.Collections;
/// <summary>
/// Base class for custom mappers that provides bidirectional IndexKey mapping for standard types.
/// </summary>
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Bson.Schema;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Collections;
/// <summary>
/// Base class for custom mappers that provides bidirectional IndexKey mapping for standard types.
/// </summary>
public abstract class DocumentMapperBase<TId, T> : IDocumentMapper<TId, T> where T : class
{
/// <summary>
/// Gets the target collection name for the mapped entity type.
/// Gets the target collection name for the mapped entity type.
/// </summary>
public abstract string CollectionName { get; }
/// <summary>
/// Serializes an entity instance into BSON.
/// Serializes an entity instance into BSON.
/// </summary>
/// <param name="entity">The entity to serialize.</param>
/// <param name="writer">The BSON writer to write into.</param>
@@ -27,96 +23,129 @@ public abstract class DocumentMapperBase<TId, T> : IDocumentMapper<TId, T> where
public abstract int Serialize(T entity, BsonSpanWriter writer);
/// <summary>
/// Deserializes an entity instance from BSON.
/// Deserializes an entity instance from BSON.
/// </summary>
/// <param name="reader">The BSON reader to read from.</param>
/// <returns>The deserialized entity.</returns>
public abstract T Deserialize(BsonSpanReader reader);
/// <summary>
/// Gets the identifier value from an entity.
/// Gets the identifier value from an entity.
/// </summary>
/// <param name="entity">The entity to read the identifier from.</param>
/// <returns>The identifier value.</returns>
public abstract TId GetId(T entity);
/// <summary>
/// Sets the identifier value on an entity.
/// Sets the identifier value on an entity.
/// </summary>
/// <param name="entity">The entity to update.</param>
/// <param name="id">The identifier value to assign.</param>
public abstract void SetId(T entity, TId id);
/// <summary>
/// Converts a typed identifier value into an index key.
/// Converts a typed identifier value into an index key.
/// </summary>
/// <param name="id">The identifier value.</param>
/// <returns>The index key representation of the identifier.</returns>
public virtual IndexKey ToIndexKey(TId id) => IndexKey.Create(id);
public virtual IndexKey ToIndexKey(TId id)
{
return IndexKey.Create(id);
}
/// <summary>
/// Converts an index key back into a typed identifier value.
/// Converts an index key back into a typed identifier value.
/// </summary>
/// <param name="key">The index key to convert.</param>
/// <returns>The typed identifier value.</returns>
public virtual TId FromIndexKey(IndexKey key) => key.As<TId>();
public virtual TId FromIndexKey(IndexKey key)
{
return key.As<TId>();
}
/// <summary>
/// Gets all mapped field keys used by this mapper.
/// Gets all mapped field keys used by this mapper.
/// </summary>
public virtual IEnumerable<string> UsedKeys => GetSchema().GetAllKeys();
/// <summary>
/// Builds the BSON schema for the mapped entity type.
/// Builds the BSON schema for the mapped entity type.
/// </summary>
/// <returns>The generated BSON schema.</returns>
public virtual BsonSchema GetSchema() => BsonSchemaGenerator.FromType<T>();
public virtual BsonSchema GetSchema()
{
return BsonSchemaGenerator.FromType<T>();
}
}
/// <summary>
/// Base class for mappers using ObjectId as primary key.
/// </summary>
/// <summary>
/// Base class for mappers using ObjectId as primary key.
/// </summary>
public abstract class ObjectIdMapperBase<T> : DocumentMapperBase<ObjectId, T>, IDocumentMapper<T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(ObjectId id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(ObjectId id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override ObjectId FromIndexKey(IndexKey key) => key.As<ObjectId>();
public override ObjectId FromIndexKey(IndexKey key)
{
return key.As<ObjectId>();
}
}
/// <summary>
/// Base class for mappers using Int32 as primary key.
/// </summary>
/// <summary>
/// Base class for mappers using Int32 as primary key.
/// </summary>
public abstract class Int32MapperBase<T> : DocumentMapperBase<int, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(int id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(int id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override int FromIndexKey(IndexKey key) => key.As<int>();
public override int FromIndexKey(IndexKey key)
{
return key.As<int>();
}
}
/// <summary>
/// Base class for mappers using String as primary key.
/// </summary>
/// <summary>
/// Base class for mappers using String as primary key.
/// </summary>
public abstract class StringMapperBase<T> : DocumentMapperBase<string, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(string id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(string id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override string FromIndexKey(IndexKey key) => key.As<string>();
public override string FromIndexKey(IndexKey key)
{
return key.As<string>();
}
}
/// <summary>
/// Base class for mappers using Guid as primary key.
/// </summary>
/// <summary>
/// Base class for mappers using Guid as primary key.
/// </summary>
public abstract class GuidMapperBase<T> : DocumentMapperBase<Guid, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(Guid id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(Guid id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override Guid FromIndexKey(IndexKey key) => key.As<Guid>();
}
public override Guid FromIndexKey(IndexKey key)
{
return key.As<Guid>();
}
}

79
src/CBDD.Core/Collections/BsonSchemaGenerator.cs
View File

@@ -1,36 +1,33 @@
using System.Reflection;
using System.Linq;
using System.Collections;
using System.Collections.Generic;
using System.Collections.Concurrent;
using System.Reflection;
using ZB.MOM.WW.CBDD.Bson;
using System;
using ZB.MOM.WW.CBDD.Bson.Schema;
namespace ZB.MOM.WW.CBDD.Core.Collections;
public static class BsonSchemaGenerator
{
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// </summary>
/// <typeparam name="T">The CLR type to inspect.</typeparam>
/// <returns>The generated BSON schema.</returns>
public static BsonSchema FromType<T>()
{
return FromType(typeof(T));
}
public static class BsonSchemaGenerator
{
private static readonly ConcurrentDictionary<Type, BsonSchema> _cache = new();
private static readonly ConcurrentDictionary<Type, BsonSchema> _cache = new();
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// </summary>
/// <param name="type">The CLR type to inspect.</param>
/// <returns>The generated BSON schema.</returns>
public static BsonSchema FromType(Type type)
{
return _cache.GetOrAdd(type, GenerateSchema);
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// </summary>
/// <typeparam name="T">The CLR type to inspect.</typeparam>
/// <returns>The generated BSON schema.</returns>
public static BsonSchema FromType<T>()
{
return FromType(typeof(T));
}
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// </summary>
/// <param name="type">The CLR type to inspect.</param>
/// <returns>The generated BSON schema.</returns>
public static BsonSchema FromType(Type type)
{
return _cache.GetOrAdd(type, GenerateSchema);
}
private static BsonSchema GenerateSchema(Type type)
@@ -47,10 +44,7 @@ public static class BsonSchemaGenerator
AddField(schema, prop.Name, prop.PropertyType);
}
foreach (var field in fields)
{
AddField(schema, field.Name, field.FieldType);
}
foreach (var field in fields) AddField(schema, field.Name, field.FieldType);
return schema;
}
@@ -60,10 +54,7 @@ public static class BsonSchemaGenerator
name = name.ToLowerInvariant();
// Convention: id -> _id for root document
if (name.Equals("id", StringComparison.OrdinalIgnoreCase))
{
name = "_id";
}
if (name.Equals("id", StringComparison.OrdinalIgnoreCase)) name = "_id";
var (bsonType, nestedSchema, itemType) = GetBsonType(type);
@@ -97,20 +88,18 @@ public static class BsonSchemaGenerator
if (type != typeof(string) && typeof(IEnumerable).IsAssignableFrom(type))
{
var itemType = GetCollectionItemType(type);
var (itemBsonType, itemNested, _) = GetBsonType(itemType);
// For arrays, if item is Document, we use NestedSchema to describe the item
var (itemBsonType, itemNested, _) = GetBsonType(itemType);
// For arrays, if item is Document, we use NestedSchema to describe the item
return (BsonType.Array, itemNested, itemBsonType);
}
// Nested Objects / Structs
// If it's not a string, not a primitive, and not an array/list, treat as Document
if (type != typeof(string) && !type.IsPrimitive && !type.IsEnum)
{
// Avoid infinite recursion?
// Simple approach: generating nested schema
return (BsonType.Document, FromType(type), null);
}
return (BsonType.Undefined, null, null);
}
@@ -122,17 +111,15 @@ public static class BsonSchemaGenerator
private static Type GetCollectionItemType(Type type)
{
if (type.IsArray) return type.GetElementType()!;
// If type itself is IEnumerable<T>
if (type.IsArray) return type.GetElementType()!;
// If type itself is IEnumerable<T>
if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof(IEnumerable<>))
{
return type.GetGenericArguments()[0];
}
var enumerableType = type.GetInterfaces()
.FirstOrDefault(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IEnumerable<>));
.FirstOrDefault(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IEnumerable<>));
return enumerableType?.GetGenericArguments()[0] ?? typeof(object);
}
}
}

28
src/CBDD.Core/Collections/DocumentCollection.Scan.cs
View File

@@ -8,8 +8,8 @@ namespace ZB.MOM.WW.CBDD.Core.Collections;
public partial class DocumentCollection<TId, T> where T : class
{
/// <summary>
/// Scans the entire collection using a raw BSON predicate.
/// This avoids deserializing documents that don't match the criteria.
/// Scans the entire collection using a raw BSON predicate.
/// This avoids deserializing documents that don't match the criteria.
/// </summary>
/// <param name="predicate">Function to evaluate raw BSON data</param>
/// <returns>Matching documents</returns>
@@ -18,8 +18,8 @@ public partial class DocumentCollection<TId, T> where T : class
if (predicate == null) throw new ArgumentNullException(nameof(predicate));
var transaction = _transactionHolder.GetCurrentTransactionOrStart();
var txnId = transaction.TransactionId;
var pageCount = _storage.PageCount;
ulong txnId = transaction.TransactionId;
uint pageCount = _storage.PageCount;
var buffer = new byte[_storage.PageSize];
var pageResults = new List<T>();
@@ -28,16 +28,13 @@ public partial class DocumentCollection<TId, T> where T : class
pageResults.Clear();
ScanPage(pageId, txnId, buffer, predicate, pageResults);
foreach (var doc in pageResults)
{
yield return doc;
}
foreach (var doc in pageResults) yield return doc;
}
}
/// <summary>
/// Scans the collection in parallel using multiple threads.
/// Useful for large collections on multi-core machines.
/// Scans the collection in parallel using multiple threads.
/// Useful for large collections on multi-core machines.
/// </summary>
/// <param name="predicate">Function to evaluate raw BSON data</param>
/// <param name="degreeOfParallelism">Number of threads to use (default: -1 = ProcessorCount)</param>
@@ -46,7 +43,7 @@ public partial class DocumentCollection<TId, T> where T : class
if (predicate == null) throw new ArgumentNullException(nameof(predicate));
var transaction = _transactionHolder.GetCurrentTransactionOrStart();
var txnId = transaction.TransactionId;
ulong txnId = transaction.TransactionId;
var pageCount = (int)_storage.PageCount;
if (degreeOfParallelism <= 0)
@@ -61,15 +58,14 @@ public partial class DocumentCollection<TId, T> where T : class
var localResults = new List<T>();
for (int i = range.Item1; i < range.Item2; i++)
{
ScanPage((uint)i, txnId, localBuffer, predicate, localResults);
}
return localResults;
});
}
private void ScanPage(uint pageId, ulong txnId, byte[] buffer, Func<BsonSpanReader, bool> predicate, List<T> results)
private void ScanPage(uint pageId, ulong txnId, byte[] buffer, Func<BsonSpanReader, bool> predicate,
List<T> results)
{
_storage.ReadPage(pageId, txnId, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
@@ -80,7 +76,7 @@ public partial class DocumentCollection<TId, T> where T : class
var slots = MemoryMarshal.Cast<byte, SlotEntry>(
buffer.AsSpan(SlottedPageHeader.Size, header.SlotCount * SlotEntry.Size));
for (int i = 0; i < header.SlotCount; i++)
for (var i = 0; i < header.SlotCount; i++)
{
var slot = slots[i];
@@ -98,4 +94,4 @@ public partial class DocumentCollection<TId, T> where T : class
}
}
}
}
}

1946
src/CBDD.Core/Collections/DocumentCollection.cs
View File

File diff suppressed because it is too large Load Diff

63
src/CBDD.Core/Collections/IDocumentMapper.cs
View File

@@ -1,42 +1,39 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using System;
using System.Buffers;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson.Schema;
namespace ZB.MOM.WW.CBDD.Core.Collections;
/// <summary>
/// Non-generic interface for common mapper operations.
/// </summary>
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Bson.Schema;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Collections;
/// <summary>
/// Non-generic interface for common mapper operations.
/// </summary>
public interface IDocumentMapper
{
/// <summary>
/// Gets the collection name handled by this mapper.
/// Gets the collection name handled by this mapper.
/// </summary>
string CollectionName { get; }
/// <summary>
/// Gets the set of document keys used during mapping.
/// Gets the set of document keys used during mapping.
/// </summary>
IEnumerable<string> UsedKeys { get; }
/// <summary>
/// Gets the BSON schema for the mapped document.
/// Gets the BSON schema for the mapped document.
/// </summary>
/// <returns>The BSON schema.</returns>
BsonSchema GetSchema();
}
/// <summary>
/// Interface for mapping between entities and BSON using zero-allocation serialization.
/// Handles bidirectional mapping between TId and IndexKey.
/// </summary>
/// <summary>
/// Interface for mapping between entities and BSON using zero-allocation serialization.
/// Handles bidirectional mapping between TId and IndexKey.
/// </summary>
public interface IDocumentMapper<TId, T> : IDocumentMapper where T : class
{
/// <summary>
/// Serializes an entity to BSON.
/// Serializes an entity to BSON.
/// </summary>
/// <param name="entity">The entity to serialize.</param>
/// <param name="writer">The BSON writer.</param>
@@ -44,44 +41,44 @@ public interface IDocumentMapper<TId, T> : IDocumentMapper where T : class
int Serialize(T entity, BsonSpanWriter writer);
/// <summary>
/// Deserializes an entity from BSON.
/// Deserializes an entity from BSON.
/// </summary>
/// <param name="reader">The BSON reader.</param>
/// <returns>The deserialized entity.</returns>
T Deserialize(BsonSpanReader reader);
/// <summary>
/// Gets the identifier value from an entity.
/// Gets the identifier value from an entity.
/// </summary>
/// <param name="entity">The entity.</param>
/// <returns>The identifier value.</returns>
TId GetId(T entity);
/// <summary>
/// Sets the identifier value on an entity.
/// Sets the identifier value on an entity.
/// </summary>
/// <param name="entity">The entity.</param>
/// <param name="id">The identifier value.</param>
void SetId(T entity, TId id);
/// <summary>
/// Converts an identifier to an index key.
/// Converts an identifier to an index key.
/// </summary>
/// <param name="id">The identifier value.</param>
/// <returns>The index key representation.</returns>
IndexKey ToIndexKey(TId id);
/// <summary>
/// Converts an index key back to an identifier.
/// Converts an index key back to an identifier.
/// </summary>
/// <param name="key">The index key.</param>
/// <returns>The identifier value.</returns>
TId FromIndexKey(IndexKey key);
}
/// <summary>
/// Legacy interface for compatibility with existing ObjectId-based collections.
/// </summary>
public interface IDocumentMapper<T> : IDocumentMapper<ObjectId, T> where T : class
{
}
/// <summary>
/// Legacy interface for compatibility with existing ObjectId-based collections.
/// </summary>
public interface IDocumentMapper<T> : IDocumentMapper<ObjectId, T> where T : class
{
}

19
src/CBDD.Core/Collections/SchemaVersion.cs
View File

@@ -1,21 +1,19 @@
using System;
namespace ZB.MOM.WW.CBDD.Core.Collections;
namespace ZB.MOM.WW.CBDD.Core.Collections;
public readonly struct SchemaVersion
{
/// <summary>
/// Gets the schema version number.
/// Gets the schema version number.
/// </summary>
public int Version { get; }
/// <summary>
/// Gets the schema hash.
/// Gets the schema hash.
/// </summary>
public long Hash { get; }
/// <summary>
/// Initializes a new instance of the <see cref="SchemaVersion"/> struct.
/// Initializes a new instance of the <see cref="SchemaVersion" /> struct.
/// </summary>
/// <param name="version">The schema version number.</param>
/// <param name="hash">The schema hash.</param>
@@ -26,5 +24,8 @@ public readonly struct SchemaVersion
}
/// <inheritdoc />
public override string ToString() => $"v{Version} (0x{Hash:X16})";
}
public override string ToString()
{
return $"v{Version} (0x{Hash:X16})";
}
}

53
src/CBDD.Core/Compression/CompressedPayloadHeader.cs
View File

@@ -3,34 +3,34 @@ using System.Buffers.Binary;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Fixed header prefix for compressed payload blobs.
/// Fixed header prefix for compressed payload blobs.
/// </summary>
public readonly struct CompressedPayloadHeader
{
public const int Size = 16;
/// <summary>
/// Compression codec used for payload bytes.
/// Compression codec used for payload bytes.
/// </summary>
public CompressionCodec Codec { get; }
/// <summary>
/// Original uncompressed payload length.
/// Original uncompressed payload length.
/// </summary>
public int OriginalLength { get; }
/// <summary>
/// Compressed payload length.
/// Compressed payload length.
/// </summary>
public int CompressedLength { get; }
/// <summary>
/// CRC32 checksum of compressed payload bytes.
/// CRC32 checksum of compressed payload bytes.
/// </summary>
public uint Checksum { get; }
/// <summary>
/// Initializes a new instance of the <see cref="CompressedPayloadHeader"/> class.
/// Initializes a new instance of the <see cref="CompressedPayloadHeader" /> class.
/// </summary>
/// <param name="codec">Compression codec used for payload bytes.</param>
/// <param name="originalLength">Original uncompressed payload length.</param>
@@ -50,19 +50,20 @@ public readonly struct CompressedPayloadHeader
}
/// <summary>
/// Create.
/// Create.
/// </summary>
/// <param name="codec">Compression codec used for payload bytes.</param>
/// <param name="originalLength">Original uncompressed payload length.</param>
/// <param name="compressedPayload">Compressed payload bytes.</param>
public static CompressedPayloadHeader Create(CompressionCodec codec, int originalLength, ReadOnlySpan<byte> compressedPayload)
public static CompressedPayloadHeader Create(CompressionCodec codec, int originalLength,
ReadOnlySpan<byte> compressedPayload)
{
var checksum = ComputeChecksum(compressedPayload);
uint checksum = ComputeChecksum(compressedPayload);
return new CompressedPayloadHeader(codec, originalLength, compressedPayload.Length, checksum);
}
/// <summary>
/// Write To.
/// Write To.
/// </summary>
/// <param name="destination">Destination span that receives the serialized header.</param>
public void WriteTo(Span<byte> destination)
@@ -80,7 +81,7 @@ public readonly struct CompressedPayloadHeader
}
/// <summary>
/// Read From.
/// Read From.
/// </summary>
/// <param name="source">Source span containing a serialized header.</param>
public static CompressedPayloadHeader ReadFrom(ReadOnlySpan<byte> source)
@@ -89,14 +90,14 @@ public readonly struct CompressedPayloadHeader
throw new ArgumentException($"Source must be at least {Size} bytes.", nameof(source));
var codec = (CompressionCodec)source[0];
var originalLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(4, 4));
var compressedLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(8, 4));
var checksum = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(12, 4));
int originalLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(4, 4));
int compressedLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(8, 4));
uint checksum = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(12, 4));
return new CompressedPayloadHeader(codec, originalLength, compressedLength, checksum);
}
/// <summary>
/// Validate Checksum.
/// Validate Checksum.
/// </summary>
/// <param name="compressedPayload">Compressed payload bytes to validate.</param>
public bool ValidateChecksum(ReadOnlySpan<byte> compressedPayload)
@@ -105,10 +106,13 @@ public readonly struct CompressedPayloadHeader
}
/// <summary>
/// Compute Checksum.
/// Compute Checksum.
/// </summary>
/// <param name="payload">Payload bytes.</param>
public static uint ComputeChecksum(ReadOnlySpan<byte> payload) => Crc32Calculator.Compute(payload);
public static uint ComputeChecksum(ReadOnlySpan<byte> payload)
{
return Crc32Calculator.Compute(payload);
}
private static class Crc32Calculator
{
@@ -116,15 +120,15 @@ public readonly struct CompressedPayloadHeader
private static readonly uint[] Table = CreateTable();
/// <summary>
/// Compute.
/// Compute.
/// </summary>
/// <param name="payload">Payload bytes.</param>
public static uint Compute(ReadOnlySpan<byte> payload)
{
uint crc = 0xFFFFFFFFu;
for (int i = 0; i < payload.Length; i++)
var crc = 0xFFFFFFFFu;
for (var i = 0; i < payload.Length; i++)
{
var index = (crc ^ payload[i]) & 0xFF;
uint index = (crc ^ payload[i]) & 0xFF;
crc = (crc >> 8) ^ Table[index];
}
@@ -137,10 +141,7 @@ public readonly struct CompressedPayloadHeader
for (uint i = 0; i < table.Length; i++)
{
uint value = i;
for (int bit = 0; bit < 8; bit++)
{
value = (value & 1) != 0 ? (value >> 1) ^ Polynomial : value >> 1;
}
for (var bit = 0; bit < 8; bit++) value = (value & 1) != 0 ? (value >> 1) ^ Polynomial : value >> 1;
table[i] = value;
}
@@ -148,4 +149,4 @@ public readonly struct CompressedPayloadHeader
return table;
}
}
}
}

4
src/CBDD.Core/Compression/CompressionCodec.cs
View File

@@ -1,11 +1,11 @@
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Supported payload compression codecs.
/// Supported payload compression codecs.
/// </summary>
public enum CompressionCodec : byte
{
None = 0,
Brotli = 1,
Deflate = 2
}
}

31
src/CBDD.Core/Compression/CompressionOptions.cs
View File

@@ -3,52 +3,52 @@ using System.IO.Compression;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Compression configuration for document payload processing.
/// Compression configuration for document payload processing.
/// </summary>
public sealed class CompressionOptions
{
/// <summary>
/// Default compression options (compression disabled).
/// Default compression options (compression disabled).
/// </summary>
public static CompressionOptions Default { get; } = new();
/// <summary>
/// Enables payload compression for new writes.
/// Enables payload compression for new writes.
/// </summary>
public bool EnableCompression { get; init; } = false;
/// <summary>
/// Minimum payload size (bytes) required before compression is attempted.
/// Minimum payload size (bytes) required before compression is attempted.
/// </summary>
public int MinSizeBytes { get; init; } = 1024;
/// <summary>
/// Minimum percentage of size reduction required to keep compressed output.
/// Minimum percentage of size reduction required to keep compressed output.
/// </summary>
public int MinSavingsPercent { get; init; } = 10;
/// <summary>
/// Preferred default codec for new writes.
/// Preferred default codec for new writes.
/// </summary>
public CompressionCodec Codec { get; init; } = CompressionCodec.Brotli;
/// <summary>
/// Compression level passed to codec implementations.
/// Compression level passed to codec implementations.
/// </summary>
public CompressionLevel Level { get; init; } = CompressionLevel.Fastest;
/// <summary>
/// Maximum allowed decompressed payload size.
/// Maximum allowed decompressed payload size.
/// </summary>
public int MaxDecompressedSizeBytes { get; init; } = 16 * 1024 * 1024;
/// <summary>
/// Optional maximum input size allowed for compression attempts.
/// Optional maximum input size allowed for compression attempts.
/// </summary>
public int? MaxCompressionInputBytes { get; init; }
/// <summary>
/// Normalizes and validates compression options.
/// Normalizes and validates compression options.
/// </summary>
/// <param name="options">Optional user-provided options.</param>
internal static CompressionOptions Normalize(CompressionOptions? options)
@@ -59,17 +59,20 @@ public sealed class CompressionOptions
throw new ArgumentOutOfRangeException(nameof(MinSizeBytes), "MinSizeBytes must be non-negative.");
if (candidate.MinSavingsPercent is < 0 or > 100)
throw new ArgumentOutOfRangeException(nameof(MinSavingsPercent), "MinSavingsPercent must be between 0 and 100.");
throw new ArgumentOutOfRangeException(nameof(MinSavingsPercent),
"MinSavingsPercent must be between 0 and 100.");
if (!Enum.IsDefined(candidate.Codec))
throw new ArgumentOutOfRangeException(nameof(Codec), $"Unsupported codec: {candidate.Codec}.");
if (candidate.MaxDecompressedSizeBytes <= 0)
throw new ArgumentOutOfRangeException(nameof(MaxDecompressedSizeBytes), "MaxDecompressedSizeBytes must be greater than 0.");
throw new ArgumentOutOfRangeException(nameof(MaxDecompressedSizeBytes),
"MaxDecompressedSizeBytes must be greater than 0.");
if (candidate.MaxCompressionInputBytes is <= 0)
throw new ArgumentOutOfRangeException(nameof(MaxCompressionInputBytes), "MaxCompressionInputBytes must be greater than 0 when provided.");
throw new ArgumentOutOfRangeException(nameof(MaxCompressionInputBytes),
"MaxCompressionInputBytes must be greater than 0 when provided.");
return candidate;
}
}
}

266
src/CBDD.Core/Compression/CompressionService.cs
View File

@@ -5,14 +5,14 @@ using System.IO.Compression;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Compression codec registry and utility service.
/// Compression codec registry and utility service.
/// </summary>
public sealed class CompressionService
{
private readonly ConcurrentDictionary<CompressionCodec, ICompressionCodec> _codecs = new();
/// <summary>
/// Initializes a new instance of the <see cref="CompressionService"/> class.
/// Initializes a new instance of the <see cref="CompressionService" /> class.
/// </summary>
/// <param name="additionalCodecs">Optional additional codecs to register.</param>
public CompressionService(IEnumerable<ICompressionCodec>? additionalCodecs = null)
@@ -24,14 +24,11 @@ public sealed class CompressionService
if (additionalCodecs == null)
return;
foreach (var codec in additionalCodecs)
{
RegisterCodec(codec);
}
foreach (var codec in additionalCodecs) RegisterCodec(codec);
}
/// <summary>
/// Registers or replaces a compression codec implementation.
/// Registers or replaces a compression codec implementation.
/// </summary>
/// <param name="codec">The codec implementation to register.</param>
public void RegisterCodec(ICompressionCodec codec)
@@ -41,18 +38,21 @@ public sealed class CompressionService
}
/// <summary>
/// Attempts to resolve a registered codec implementation.
/// Attempts to resolve a registered codec implementation.
/// </summary>
/// <param name="codec">The codec identifier to resolve.</param>
/// <param name="compressionCodec">When this method returns, contains the resolved codec when found.</param>
/// <returns><see langword="true"/> when a codec is registered for <paramref name="codec"/>; otherwise, <see langword="false"/>.</returns>
/// <returns>
/// <see langword="true" /> when a codec is registered for <paramref name="codec" />; otherwise,
/// <see langword="false" />.
/// </returns>
public bool TryGetCodec(CompressionCodec codec, out ICompressionCodec compressionCodec)
{
return _codecs.TryGetValue(codec, out compressionCodec!);
}
/// <summary>
/// Gets a registered codec implementation.
/// Gets a registered codec implementation.
/// </summary>
/// <param name="codec">The codec identifier to resolve.</param>
/// <returns>The registered codec implementation.</returns>
@@ -65,7 +65,7 @@ public sealed class CompressionService
}
/// <summary>
/// Compresses payload bytes using the selected codec and level.
/// Compresses payload bytes using the selected codec and level.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="codec">The codec to use.</param>
@@ -77,131 +77,40 @@ public sealed class CompressionService
}
/// <summary>
/// Decompresses payload bytes using the selected codec.
/// Decompresses payload bytes using the selected codec.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="codec">The codec to use.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, CompressionCodec codec, int expectedLength, int maxDecompressedSizeBytes)
public byte[] Decompress(ReadOnlySpan<byte> input, CompressionCodec codec, int expectedLength,
int maxDecompressedSizeBytes)
{
return GetCodec(codec).Decompress(input, expectedLength, maxDecompressedSizeBytes);
}
/// <summary>
/// Compresses and then decompresses payload bytes using the selected codec.
/// Compresses and then decompresses payload bytes using the selected codec.
/// </summary>
/// <param name="input">The payload bytes to roundtrip.</param>
/// <param name="codec">The codec to use.</param>
/// <param name="level">The compression level.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes after roundtrip.</returns>
public byte[] Roundtrip(ReadOnlySpan<byte> input, CompressionCodec codec, CompressionLevel level, int maxDecompressedSizeBytes)
public byte[] Roundtrip(ReadOnlySpan<byte> input, CompressionCodec codec, CompressionLevel level,
int maxDecompressedSizeBytes)
{
var compressed = Compress(input, codec, level);
byte[] compressed = Compress(input, codec, level);
return Decompress(compressed, codec, input.Length, maxDecompressedSizeBytes);
}
private sealed class NoneCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.None;
/// <summary>
/// Returns a copy of the input payload without compression.
/// </summary>
/// <param name="input">The payload bytes to copy.</param>
/// <param name="level">The requested compression level.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level) => input.ToArray();
/// <summary>
/// Validates and returns an uncompressed payload copy.
/// </summary>
/// <param name="input">The payload bytes to validate and copy.</param>
/// <param name="expectedLength">The expected payload length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed payload size in bytes.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
if (input.Length > maxDecompressedSizeBytes)
throw new InvalidDataException($"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
if (expectedLength >= 0 && expectedLength != input.Length)
throw new InvalidDataException($"Expected decompressed length {expectedLength}, actual {input.Length}.");
return input.ToArray();
}
}
private sealed class BrotliCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Brotli;
/// <summary>
/// Compresses payload bytes using Brotli.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new BrotliStream(stream, level, leaveOpen: true));
}
/// <summary>
/// Decompresses Brotli-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input, stream => new BrotliStream(stream, CompressionMode.Decompress, leaveOpen: true), expectedLength, maxDecompressedSizeBytes);
}
}
private sealed class DeflateCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Deflate;
/// <summary>
/// Compresses payload bytes using Deflate.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new DeflateStream(stream, level, leaveOpen: true));
}
/// <summary>
/// Decompresses Deflate-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input, stream => new DeflateStream(stream, CompressionMode.Decompress, leaveOpen: true), expectedLength, maxDecompressedSizeBytes);
}
}
private static byte[] CompressWithCodecStream(ReadOnlySpan<byte> input, Func<Stream, Stream> streamFactory)
{
using var output = new MemoryStream(capacity: input.Length);
using var output = new MemoryStream(input.Length);
using (var codecStream = streamFactory(output))
{
codecStream.Write(input);
@@ -220,31 +129,33 @@ public sealed class CompressionService
if (maxDecompressedSizeBytes <= 0)
throw new ArgumentOutOfRangeException(nameof(maxDecompressedSizeBytes));
using var compressed = new MemoryStream(input.ToArray(), writable: false);
using var compressed = new MemoryStream(input.ToArray(), false);
using var codecStream = streamFactory(compressed);
using var output = expectedLength > 0
? new MemoryStream(capacity: expectedLength)
? new MemoryStream(expectedLength)
: new MemoryStream();
var buffer = ArrayPool<byte>.Shared.Rent(8192);
byte[] buffer = ArrayPool<byte>.Shared.Rent(8192);
try
{
int totalWritten = 0;
var totalWritten = 0;
while (true)
{
var bytesRead = codecStream.Read(buffer, 0, buffer.Length);
int bytesRead = codecStream.Read(buffer, 0, buffer.Length);
if (bytesRead <= 0)
break;
totalWritten += bytesRead;
if (totalWritten > maxDecompressedSizeBytes)
throw new InvalidDataException($"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
throw new InvalidDataException(
$"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
output.Write(buffer, 0, bytesRead);
}
if (expectedLength >= 0 && totalWritten != expectedLength)
throw new InvalidDataException($"Expected decompressed length {expectedLength}, actual {totalWritten}.");
throw new InvalidDataException(
$"Expected decompressed length {expectedLength}, actual {totalWritten}.");
return output.ToArray();
}
@@ -253,4 +164,115 @@ public sealed class CompressionService
ArrayPool<byte>.Shared.Return(buffer);
}
}
}
private sealed class NoneCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.None;
/// <summary>
/// Returns a copy of the input payload without compression.
/// </summary>
/// <param name="input">The payload bytes to copy.</param>
/// <param name="level">The requested compression level.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return input.ToArray();
}
/// <summary>
/// Validates and returns an uncompressed payload copy.
/// </summary>
/// <param name="input">The payload bytes to validate and copy.</param>
/// <param name="expectedLength">The expected payload length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed payload size in bytes.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
if (input.Length > maxDecompressedSizeBytes)
throw new InvalidDataException(
$"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
if (expectedLength >= 0 && expectedLength != input.Length)
throw new InvalidDataException(
$"Expected decompressed length {expectedLength}, actual {input.Length}.");
return input.ToArray();
}
}
private sealed class BrotliCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Brotli;
/// <summary>
/// Compresses payload bytes using Brotli.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new BrotliStream(stream, level, true));
}
/// <summary>
/// Decompresses Brotli-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input,
stream => new BrotliStream(stream, CompressionMode.Decompress, true), expectedLength,
maxDecompressedSizeBytes);
}
}
private sealed class DeflateCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Deflate;
/// <summary>
/// Compresses payload bytes using Deflate.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new DeflateStream(stream, level, true));
}
/// <summary>
/// Decompresses Deflate-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input,
stream => new DeflateStream(stream, CompressionMode.Decompress, true), expectedLength,
maxDecompressedSizeBytes);
}
}
}

27
src/CBDD.Core/Compression/CompressionStats.cs
View File

@@ -1,40 +1,47 @@
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Snapshot of aggregated compression and decompression telemetry.
/// Snapshot of aggregated compression and decompression telemetry.
/// </summary>
public readonly struct CompressionStats
{
/// <summary>
/// Gets or sets the CompressedDocumentCount.
/// Gets or sets the CompressedDocumentCount.
/// </summary>
public long CompressedDocumentCount { get; init; }
/// <summary>
/// Gets or sets the BytesBeforeCompression.
/// Gets or sets the BytesBeforeCompression.
/// </summary>
public long BytesBeforeCompression { get; init; }
/// <summary>
/// Gets or sets the BytesAfterCompression.
/// Gets or sets the BytesAfterCompression.
/// </summary>
public long BytesAfterCompression { get; init; }
/// <summary>
/// Gets or sets the CompressionCpuTicks.
/// Gets or sets the CompressionCpuTicks.
/// </summary>
public long CompressionCpuTicks { get; init; }
/// <summary>
/// Gets or sets the DecompressionCpuTicks.
/// Gets or sets the DecompressionCpuTicks.
/// </summary>
public long DecompressionCpuTicks { get; init; }
/// <summary>
/// Gets or sets the CompressionFailureCount.
/// Gets or sets the CompressionFailureCount.
/// </summary>
public long CompressionFailureCount { get; init; }
/// <summary>
/// Gets or sets the ChecksumFailureCount.
/// Gets or sets the ChecksumFailureCount.
/// </summary>
public long ChecksumFailureCount { get; init; }
/// <summary>
/// Gets or sets the SafetyLimitRejectionCount.
/// Gets or sets the SafetyLimitRejectionCount.
/// </summary>
public long SafetyLimitRejectionCount { get; init; }
}
}

129
src/CBDD.Core/Compression/CompressionTelemetry.cs
View File

@@ -1,111 +1,109 @@
using System.Threading;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Thread-safe counters for compression/decompression lifecycle events.
/// Thread-safe counters for compression/decompression lifecycle events.
/// </summary>
public sealed class CompressionTelemetry
{
private long _checksumFailureCount;
private long _compressedDocumentCount;
private long _compressionAttempts;
private long _compressionSuccesses;
private long _compressionCpuTicks;
private long _compressionFailures;
private long _compressionSkippedTooSmall;
private long _compressionSkippedInsufficientSavings;
private long _decompressionAttempts;
private long _decompressionSuccesses;
private long _decompressionFailures;
private long _compressionInputBytes;
private long _compressionOutputBytes;
private long _decompressionOutputBytes;
private long _compressedDocumentCount;
private long _compressionCpuTicks;
private long _compressionSkippedInsufficientSavings;
private long _compressionSkippedTooSmall;
private long _compressionSuccesses;
private long _decompressionAttempts;
private long _decompressionCpuTicks;
private long _checksumFailureCount;
private long _decompressionFailures;
private long _decompressionOutputBytes;
private long _decompressionSuccesses;
private long _safetyLimitRejectionCount;
/// <summary>
/// Gets the number of attempted compression operations.
/// Gets the number of attempted compression operations.
/// </summary>
public long CompressionAttempts => Interlocked.Read(ref _compressionAttempts);
/// <summary>
/// Gets the number of successful compression operations.
/// Gets the number of successful compression operations.
/// </summary>
public long CompressionSuccesses => Interlocked.Read(ref _compressionSuccesses);
/// <summary>
/// Gets the number of failed compression operations.
/// Gets the number of failed compression operations.
/// </summary>
public long CompressionFailures => Interlocked.Read(ref _compressionFailures);
/// <summary>
/// Gets the number of compression attempts skipped because payloads were too small.
/// Gets the number of compression attempts skipped because payloads were too small.
/// </summary>
public long CompressionSkippedTooSmall => Interlocked.Read(ref _compressionSkippedTooSmall);
/// <summary>
/// Gets the number of compression attempts skipped due to insufficient savings.
/// Gets the number of compression attempts skipped due to insufficient savings.
/// </summary>
public long CompressionSkippedInsufficientSavings => Interlocked.Read(ref _compressionSkippedInsufficientSavings);
/// <summary>
/// Gets the number of attempted decompression operations.
/// Gets the number of attempted decompression operations.
/// </summary>
public long DecompressionAttempts => Interlocked.Read(ref _decompressionAttempts);
/// <summary>
/// Gets the number of successful decompression operations.
/// Gets the number of successful decompression operations.
/// </summary>
public long DecompressionSuccesses => Interlocked.Read(ref _decompressionSuccesses);
/// <summary>
/// Gets the number of failed decompression operations.
/// Gets the number of failed decompression operations.
/// </summary>
public long DecompressionFailures => Interlocked.Read(ref _decompressionFailures);
/// <summary>
/// Gets the total input bytes observed by compression attempts.
/// Gets the total input bytes observed by compression attempts.
/// </summary>
public long CompressionInputBytes => Interlocked.Read(ref _compressionInputBytes);
/// <summary>
/// Gets the total output bytes produced by successful compression attempts.
/// Gets the total output bytes produced by successful compression attempts.
/// </summary>
public long CompressionOutputBytes => Interlocked.Read(ref _compressionOutputBytes);
/// <summary>
/// Gets the total output bytes produced by successful decompression attempts.
/// Gets the total output bytes produced by successful decompression attempts.
/// </summary>
public long DecompressionOutputBytes => Interlocked.Read(ref _decompressionOutputBytes);
/// <summary>
/// Gets the number of documents stored in compressed form.
/// Gets the number of documents stored in compressed form.
/// </summary>
public long CompressedDocumentCount => Interlocked.Read(ref _compressedDocumentCount);
/// <summary>
/// Gets the total CPU ticks spent on compression.
/// Gets the total CPU ticks spent on compression.
/// </summary>
public long CompressionCpuTicks => Interlocked.Read(ref _compressionCpuTicks);
/// <summary>
/// Gets the total CPU ticks spent on decompression.
/// Gets the total CPU ticks spent on decompression.
/// </summary>
public long DecompressionCpuTicks => Interlocked.Read(ref _decompressionCpuTicks);
/// <summary>
/// Gets the number of checksum validation failures.
/// Gets the number of checksum validation failures.
/// </summary>
public long ChecksumFailureCount => Interlocked.Read(ref _checksumFailureCount);
/// <summary>
/// Gets the number of decompression safety-limit rejections.
/// Gets the number of decompression safety-limit rejections.
/// </summary>
public long SafetyLimitRejectionCount => Interlocked.Read(ref _safetyLimitRejectionCount);
/// <summary>
/// Records a compression attempt and its input byte size.
/// Records a compression attempt and its input byte size.
/// </summary>
/// <param name="inputBytes">The number of input bytes provided to compression.</param>
public void RecordCompressionAttempt(int inputBytes)
@@ -115,7 +113,7 @@ public sealed class CompressionTelemetry
}
/// <summary>
/// Records a successful compression operation.
/// Records a successful compression operation.
/// </summary>
/// <param name="outputBytes">The number of compressed bytes produced.</param>
public void RecordCompressionSuccess(int outputBytes)
@@ -126,49 +124,73 @@ public sealed class CompressionTelemetry
}
/// <summary>
/// Records a failed compression operation.
/// Records a failed compression operation.
/// </summary>
public void RecordCompressionFailure() => Interlocked.Increment(ref _compressionFailures);
public void RecordCompressionFailure()
{
Interlocked.Increment(ref _compressionFailures);
}
/// <summary>
/// Records that compression was skipped because the payload was too small.
/// Records that compression was skipped because the payload was too small.
/// </summary>
public void RecordCompressionSkippedTooSmall() => Interlocked.Increment(ref _compressionSkippedTooSmall);
public void RecordCompressionSkippedTooSmall()
{
Interlocked.Increment(ref _compressionSkippedTooSmall);
}
/// <summary>
/// Records that compression was skipped due to insufficient expected savings.
/// Records that compression was skipped due to insufficient expected savings.
/// </summary>
public void RecordCompressionSkippedInsufficientSavings() => Interlocked.Increment(ref _compressionSkippedInsufficientSavings);
public void RecordCompressionSkippedInsufficientSavings()
{
Interlocked.Increment(ref _compressionSkippedInsufficientSavings);
}
/// <summary>
/// Records a decompression attempt.
/// Records a decompression attempt.
/// </summary>
public void RecordDecompressionAttempt() => Interlocked.Increment(ref _decompressionAttempts);
public void RecordDecompressionAttempt()
{
Interlocked.Increment(ref _decompressionAttempts);
}
/// <summary>
/// Adds CPU ticks spent performing compression.
/// Adds CPU ticks spent performing compression.
/// </summary>
/// <param name="ticks">The CPU ticks to add.</param>
public void RecordCompressionCpuTicks(long ticks) => Interlocked.Add(ref _compressionCpuTicks, ticks);
public void RecordCompressionCpuTicks(long ticks)
{
Interlocked.Add(ref _compressionCpuTicks, ticks);
}
/// <summary>
/// Adds CPU ticks spent performing decompression.
/// Adds CPU ticks spent performing decompression.
/// </summary>
/// <param name="ticks">The CPU ticks to add.</param>
public void RecordDecompressionCpuTicks(long ticks) => Interlocked.Add(ref _decompressionCpuTicks, ticks);
public void RecordDecompressionCpuTicks(long ticks)
{
Interlocked.Add(ref _decompressionCpuTicks, ticks);
}
/// <summary>
/// Records a checksum validation failure.
/// Records a checksum validation failure.
/// </summary>
public void RecordChecksumFailure() => Interlocked.Increment(ref _checksumFailureCount);
public void RecordChecksumFailure()
{
Interlocked.Increment(ref _checksumFailureCount);
}
/// <summary>
/// Records a decompression rejection due to safety limits.
/// Records a decompression rejection due to safety limits.
/// </summary>
public void RecordSafetyLimitRejection() => Interlocked.Increment(ref _safetyLimitRejectionCount);
public void RecordSafetyLimitRejection()
{
Interlocked.Increment(ref _safetyLimitRejectionCount);
}
/// <summary>
/// Records a successful decompression operation.
/// Records a successful decompression operation.
/// </summary>
/// <param name="outputBytes">The number of decompressed bytes produced.</param>
public void RecordDecompressionSuccess(int outputBytes)
@@ -178,12 +200,15 @@ public sealed class CompressionTelemetry
}
/// <summary>
/// Records a failed decompression operation.
/// Records a failed decompression operation.
/// </summary>
public void RecordDecompressionFailure() => Interlocked.Increment(ref _decompressionFailures);
public void RecordDecompressionFailure()
{
Interlocked.Increment(ref _decompressionFailures);
}
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry.
/// Returns a point-in-time snapshot of compression telemetry.
/// </summary>
/// <returns>The aggregated compression statistics.</returns>
public CompressionStats GetSnapshot()
@@ -200,4 +225,4 @@ public sealed class CompressionTelemetry
SafetyLimitRejectionCount = SafetyLimitRejectionCount
};
}
}
}

10
src/CBDD.Core/Compression/ICompressionCodec.cs
View File

@@ -3,27 +3,27 @@ using System.IO.Compression;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Codec abstraction for payload compression and decompression.
/// Codec abstraction for payload compression and decompression.
/// </summary>
public interface ICompressionCodec
{
/// <summary>
/// Codec identifier.
/// Codec identifier.
/// </summary>
CompressionCodec Codec { get; }
/// <summary>
/// Compresses input bytes.
/// Compresses input bytes.
/// </summary>
/// <param name="input">Input payload bytes to compress.</param>
/// <param name="level">Compression level to apply.</param>
byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level);
/// <summary>
/// Decompresses payload bytes with output bounds validation.
/// Decompresses payload bytes with output bounds validation.
/// </summary>
/// <param name="input">Input payload bytes to decompress.</param>
/// <param name="expectedLength">Expected decompressed length.</param>
/// <param name="maxDecompressedSizeBytes">Maximum allowed decompressed payload size in bytes.</param>
byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes);
}
}

382
src/CBDD.Core/Context/DocumentDbContext.cs
View File

@@ -1,52 +1,39 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Compression;
using ZB.MOM.WW.CBDD.Core.Metadata;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Metadata;
using ZB.MOM.WW.CBDD.Core.Compression;
using System.Threading;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Core;
internal interface ICompactionAwareCollection
{
/// <summary>
/// Refreshes index bindings after compaction.
/// Refreshes index bindings after compaction.
/// </summary>
void RefreshIndexBindingsAfterCompaction();
}
/// <summary>
/// Base class for database contexts.
/// Inherit and add DocumentCollection{T} properties for your entities.
/// Use partial class for Source Generator integration.
/// Base class for database contexts.
/// Inherit and add DocumentCollection{T} properties for your entities.
/// Use partial class for Source Generator integration.
/// </summary>
public abstract partial class DocumentDbContext : IDisposable, ITransactionHolder
public abstract class DocumentDbContext : IDisposable, ITransactionHolder
{
internal readonly ChangeStreamDispatcher _cdc;
private readonly List<ICompactionAwareCollection> _compactionAwareCollections = new();
private readonly IReadOnlyDictionary<Type, object> _model;
private readonly List<IDocumentMapper> _registeredMappers = new();
private readonly IStorageEngine _storage;
internal readonly CDC.ChangeStreamDispatcher _cdc;
private readonly SemaphoreSlim _transactionLock = new(1, 1);
protected bool _disposed;
private readonly SemaphoreSlim _transactionLock = new SemaphoreSlim(1, 1);
/// <summary>
/// Gets the current active transaction, if any.
/// </summary>
public ITransaction? CurrentTransaction
{
get
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return field != null && (field.State == TransactionState.Active) ? field : null;
}
private set;
}
/// <summary>
/// Creates a new database context with default configuration
/// Creates a new database context with default configuration
/// </summary>
/// <param name="databasePath">The database file path.</param>
protected DocumentDbContext(string databasePath)
@@ -55,7 +42,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Creates a new database context with default storage configuration and custom compression settings.
/// Creates a new database context with default storage configuration and custom compression settings.
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="compressionOptions">Compression behavior options.</param>
@@ -65,7 +52,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Creates a new database context with custom configuration
/// Creates a new database context with custom configuration
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="config">The page file configuration.</param>
@@ -75,7 +62,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Creates a new database context with custom storage and compression configuration.
/// Creates a new database context with custom storage and compression configuration.
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="config">The page file configuration.</param>
@@ -91,7 +78,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
throw new ArgumentNullException(nameof(databasePath));
_storage = new StorageEngine(databasePath, config, compressionOptions, maintenanceOptions);
_cdc = new CDC.ChangeStreamDispatcher();
_cdc = new ChangeStreamDispatcher();
_storage.RegisterCdc(_cdc);
// Initialize model before collections
@@ -102,108 +89,41 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Initializes document collections for the context.
/// Gets the current active transaction, if any.
/// </summary>
protected virtual void InitializeCollections()
public ITransaction? CurrentTransaction
{
// Derived classes can override to initialize collections
get
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return field != null && field.State == TransactionState.Active ? field : null;
}
private set;
}
private readonly IReadOnlyDictionary<Type, object> _model;
private readonly List<IDocumentMapper> _registeredMappers = new();
private readonly List<ICompactionAwareCollection> _compactionAwareCollections = new();
/// <summary>
/// Gets the concrete storage engine for advanced scenarios in derived contexts.
/// Gets the concrete storage engine for advanced scenarios in derived contexts.
/// </summary>
protected StorageEngine Engine => (StorageEngine)_storage;
/// <summary>
/// Gets compression options bound to this context's storage engine.
/// Gets compression options bound to this context's storage engine.
/// </summary>
protected CompressionOptions CompressionOptions => _storage.CompressionOptions;
/// <summary>
/// Gets the compression service for codec operations.
/// Gets the compression service for codec operations.
/// </summary>
protected CompressionService CompressionService => _storage.CompressionService;
/// <summary>
/// Gets compression telemetry counters.
/// Gets compression telemetry counters.
/// </summary>
protected CompressionTelemetry CompressionTelemetry => _storage.CompressionTelemetry;
/// <summary>
/// Override to configure the model using Fluent API.
/// </summary>
/// <param name="modelBuilder">The model builder instance.</param>
protected virtual void OnModelCreating(ModelBuilder modelBuilder)
{
}
/// <summary>
/// Helper to create a DocumentCollection instance with custom TId.
/// Used by derived classes in InitializeCollections for typed primary keys.
/// </summary>
/// <typeparam name="TId">The document identifier type.</typeparam>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="mapper">The mapper used for document serialization and key access.</param>
/// <returns>The created document collection.</returns>
protected DocumentCollection<TId, T> CreateCollection<TId, T>(IDocumentMapper<TId, T> mapper)
where T : class
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
string? customName = null;
EntityTypeBuilder<T>? builder = null;
if (_model.TryGetValue(typeof(T), out var builderObj))
{
builder = builderObj as EntityTypeBuilder<T>;
customName = builder?.CollectionName;
}
_registeredMappers.Add(mapper);
var collection = new DocumentCollection<TId, T>(_storage, this, mapper, customName);
if (collection is ICompactionAwareCollection compactionAwareCollection)
{
_compactionAwareCollections.Add(compactionAwareCollection);
}
// Apply configurations from ModelBuilder
if (builder != null)
{
foreach (var indexBuilder in builder.Indexes)
{
collection.ApplyIndexBuilder(indexBuilder);
}
}
_storage.RegisterMappers(_registeredMappers);
return collection;
}
/// <summary>
/// Gets the document collection for the specified entity type using an ObjectId as the key.
/// </summary>
/// <typeparam name="T">The type of entity to retrieve the document collection for. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;ObjectId, T&gt; instance for the specified entity type.</returns>
public DocumentCollection<ObjectId, T> Set<T>() where T : class => Set<ObjectId, T>();
/// <summary>
/// Gets a collection for managing documents of type T, identified by keys of type TId.
/// Override is generated automatically by the Source Generator for partial DbContext classes.
/// </summary>
/// <typeparam name="TId">The type of the unique identifier for documents in the collection.</typeparam>
/// <typeparam name="T">The type of the document to be managed. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;TId, T&gt; instance for performing operations on documents of type T.</returns>
public virtual DocumentCollection<TId, T> Set<TId, T>() where T : class
=> throw new InvalidOperationException($"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.");
/// <summary>
/// Releases resources used by the context.
/// Releases resources used by the context.
/// </summary>
public void Dispose()
{
@@ -220,7 +140,102 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Begins a transaction or returns the current active transaction.
/// Gets the current active transaction or starts a new one.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction GetCurrentTransactionOrStart()
{
return BeginTransaction();
}
/// <summary>
/// Gets the current active transaction or starts a new one asynchronously.
/// </summary>
/// <returns>The active transaction.</returns>
public async Task<ITransaction> GetCurrentTransactionOrStartAsync()
{
return await BeginTransactionAsync();
}
/// <summary>
/// Initializes document collections for the context.
/// </summary>
protected virtual void InitializeCollections()
{
// Derived classes can override to initialize collections
}
/// <summary>
/// Override to configure the model using Fluent API.
/// </summary>
/// <param name="modelBuilder">The model builder instance.</param>
protected virtual void OnModelCreating(ModelBuilder modelBuilder)
{
}
/// <summary>
/// Helper to create a DocumentCollection instance with custom TId.
/// Used by derived classes in InitializeCollections for typed primary keys.
/// </summary>
/// <typeparam name="TId">The document identifier type.</typeparam>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="mapper">The mapper used for document serialization and key access.</param>
/// <returns>The created document collection.</returns>
protected DocumentCollection<TId, T> CreateCollection<TId, T>(IDocumentMapper<TId, T> mapper)
where T : class
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
string? customName = null;
EntityTypeBuilder<T>? builder = null;
if (_model.TryGetValue(typeof(T), out object? builderObj))
{
builder = builderObj as EntityTypeBuilder<T>;
customName = builder?.CollectionName;
}
_registeredMappers.Add(mapper);
var collection = new DocumentCollection<TId, T>(_storage, this, mapper, customName);
if (collection is ICompactionAwareCollection compactionAwareCollection)
_compactionAwareCollections.Add(compactionAwareCollection);
// Apply configurations from ModelBuilder
if (builder != null)
foreach (var indexBuilder in builder.Indexes)
collection.ApplyIndexBuilder(indexBuilder);
_storage.RegisterMappers(_registeredMappers);
return collection;
}
/// <summary>
/// Gets the document collection for the specified entity type using an ObjectId as the key.
/// </summary>
/// <typeparam name="T">The type of entity to retrieve the document collection for. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;ObjectId, T&gt; instance for the specified entity type.</returns>
public DocumentCollection<ObjectId, T> Set<T>() where T : class
{
return Set<ObjectId, T>();
}
/// <summary>
/// Gets a collection for managing documents of type T, identified by keys of type TId.
/// Override is generated automatically by the Source Generator for partial DbContext classes.
/// </summary>
/// <typeparam name="TId">The type of the unique identifier for documents in the collection.</typeparam>
/// <typeparam name="T">The type of the document to be managed. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;TId, T&gt; instance for performing operations on documents of type T.</returns>
public virtual DocumentCollection<TId, T> Set<TId, T>() where T : class
{
throw new InvalidOperationException(
$"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.");
}
/// <summary>
/// Begins a transaction or returns the current active transaction.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction BeginTransaction()
@@ -243,7 +258,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Begins a transaction asynchronously or returns the current active transaction.
/// Begins a transaction asynchronously or returns the current active transaction.
/// </summary>
/// <param name="ct">The cancellation token.</param>
/// <returns>The active transaction.</returns>
@@ -252,7 +267,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
bool lockAcquired = false;
var lockAcquired = false;
try
{
await _transactionLock.WaitAsync(ct);
@@ -271,32 +286,13 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets the current active transaction or starts a new one.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction GetCurrentTransactionOrStart()
{
return BeginTransaction();
}
/// <summary>
/// Gets the current active transaction or starts a new one asynchronously.
/// </summary>
/// <returns>The active transaction.</returns>
public async Task<ITransaction> GetCurrentTransactionOrStartAsync()
{
return await BeginTransactionAsync();
}
/// <summary>
/// Commits the current transaction if one is active.
/// Commits the current transaction if one is active.
/// </summary>
public void SaveChanges()
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
if (CurrentTransaction != null)
{
try
{
CurrentTransaction.Commit();
@@ -305,19 +301,17 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
{
CurrentTransaction = null;
}
}
}
/// <summary>
/// Commits the current transaction asynchronously if one is active.
/// Commits the current transaction asynchronously if one is active.
/// </summary>
/// <param name="ct">The cancellation token.</param>
public async Task SaveChangesAsync(CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
public async Task SaveChangesAsync(CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
if (CurrentTransaction != null)
{
try
{
await CurrentTransaction.CommitAsync(ct);
@@ -325,40 +319,40 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
finally
{
CurrentTransaction = null;
}
}
}
/// <summary>
/// Executes a checkpoint using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <returns>The checkpoint execution result.</returns>
public CheckpointResult Checkpoint(CheckpointMode mode = CheckpointMode.Truncate)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return Engine.Checkpoint(mode);
}
/// <summary>
/// Executes a checkpoint asynchronously using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The checkpoint execution result.</returns>
public Task<CheckpointResult> CheckpointAsync(CheckpointMode mode = CheckpointMode.Truncate, CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return Engine.CheckpointAsync(mode, ct);
}
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
}
}
/// <summary>
/// Executes a checkpoint using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <returns>The checkpoint execution result.</returns>
public CheckpointResult Checkpoint(CheckpointMode mode = CheckpointMode.Truncate)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return Engine.Checkpoint(mode);
}
/// <summary>
/// Executes a checkpoint asynchronously using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The checkpoint execution result.</returns>
public Task<CheckpointResult> CheckpointAsync(CheckpointMode mode = CheckpointMode.Truncate,
CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return Engine.CheckpointAsync(mode, ct);
}
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats()
{
if (_disposed)
@@ -368,7 +362,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// </summary>
/// <param name="options">Compaction execution options.</param>
public CompactionStats Compact(CompactionOptions? options = null)
@@ -382,7 +376,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// </summary>
/// <param name="options">Compaction execution options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
@@ -395,7 +389,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Alias for <see cref="Compact(CompactionOptions?)"/>.
/// Alias for <see cref="Compact(CompactionOptions?)" />.
/// </summary>
/// <param name="options">Compaction execution options.</param>
public CompactionStats Vacuum(CompactionOptions? options = null)
@@ -409,7 +403,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Async alias for <see cref="CompactAsync(CompactionOptions?, CancellationToken)"/>.
/// Async alias for <see cref="CompactAsync(CompactionOptions?, CancellationToken)" />.
/// </summary>
/// <param name="options">Compaction execution options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
@@ -437,14 +431,11 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
private void RefreshCollectionBindingsAfterCompaction()
{
foreach (var collection in _compactionAwareCollections)
{
collection.RefreshIndexBindingsAfterCompaction();
}
foreach (var collection in _compactionAwareCollections) collection.RefreshIndexBindingsAfterCompaction();
}
/// <summary>
/// Gets page usage grouped by page type.
/// Gets page usage grouped by page type.
/// </summary>
public IReadOnlyList<PageTypeUsageEntry> GetPageUsageByPageType()
{
@@ -455,7 +446,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets per-collection page usage diagnostics.
/// Gets per-collection page usage diagnostics.
/// </summary>
public IReadOnlyList<CollectionPageUsageEntry> GetPageUsageByCollection()
{
@@ -466,7 +457,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets per-collection compression ratio diagnostics.
/// Gets per-collection compression ratio diagnostics.
/// </summary>
public IReadOnlyList<CollectionCompressionRatioEntry> GetCompressionRatioByCollection()
{
@@ -477,7 +468,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets free-list summary diagnostics.
/// Gets free-list summary diagnostics.
/// </summary>
public FreeListSummary GetFreeListSummary()
{
@@ -488,7 +479,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets page-level fragmentation diagnostics.
/// Gets page-level fragmentation diagnostics.
/// </summary>
public FragmentationMapReport GetFragmentationMap()
{
@@ -499,7 +490,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs compression migration as dry-run estimation by default.
/// Runs compression migration as dry-run estimation by default.
/// </summary>
/// <param name="options">Compression migration options.</param>
public CompressionMigrationResult MigrateCompression(CompressionMigrationOptions? options = null)
@@ -511,15 +502,16 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs compression migration asynchronously as dry-run estimation by default.
/// Runs compression migration asynchronously as dry-run estimation by default.
/// </summary>
/// <param name="options">Compression migration options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
public Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null, CancellationToken ct = default)
public Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null,
CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return Engine.MigrateCompressionAsync(options, ct);
}
}
}

324
src/CBDD.Core/Indexing/BTreeCursor.cs
View File

@@ -1,131 +1,129 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Bson;
using System.Collections.Generic;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
internal sealed class BTreeCursor : IBTreeCursor
{
private readonly BTreeIndex _index;
private readonly ulong _transactionId;
private readonly IIndexStorage _storage;
// State
private byte[] _pageBuffer;
private uint _currentPageId;
internal sealed class BTreeCursor : IBTreeCursor
{
private readonly List<IndexEntry> _currentEntries;
private readonly BTreeIndex _index;
private readonly IIndexStorage _storage;
private readonly ulong _transactionId;
private int _currentEntryIndex;
private BTreeNodeHeader _currentHeader;
private List<IndexEntry> _currentEntries;
private uint _currentPageId;
private bool _isValid;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeCursor"/> class.
/// </summary>
/// <param name="index">The index to traverse.</param>
/// <param name="storage">The storage engine for page access.</param>
/// <param name="transactionId">The transaction identifier used for reads.</param>
public BTreeCursor(BTreeIndex index, IIndexStorage storage, ulong transactionId)
{
_index = index;
_storage = storage;
// State
private byte[] _pageBuffer;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeCursor" /> class.
/// </summary>
/// <param name="index">The index to traverse.</param>
/// <param name="storage">The storage engine for page access.</param>
/// <param name="transactionId">The transaction identifier used for reads.</param>
public BTreeCursor(BTreeIndex index, IIndexStorage storage, ulong transactionId)
{
_index = index;
_storage = storage;
_transactionId = transactionId;
_pageBuffer = ArrayPool<byte>.Shared.Rent(storage.PageSize);
_currentEntries = new List<IndexEntry>();
_isValid = false;
}
/// <summary>
/// Gets the current index entry at the cursor position.
/// </summary>
public IndexEntry Current
{
get
_isValid = false;
}
/// <summary>
/// Gets the current index entry at the cursor position.
/// </summary>
public IndexEntry Current
{
get
{
if (!_isValid) throw new InvalidOperationException("Cursor is not valid.");
return _currentEntries[_currentEntryIndex];
}
}
/// <summary>
/// Moves the cursor to the first entry in the index.
/// </summary>
/// <returns><see langword="true"/> if an entry is available; otherwise, <see langword="false"/>.</returns>
public bool MoveToFirst()
{
return _currentEntries[_currentEntryIndex];
}
}
/// <summary>
/// Moves the cursor to the first entry in the index.
/// </summary>
/// <returns><see langword="true" /> if an entry is available; otherwise, <see langword="false" />.</returns>
public bool MoveToFirst()
{
// Find left-most leaf
var pageId = _index.RootPageId;
uint pageId = _index.RootPageId;
while (true)
{
LoadPage(pageId);
if (_currentHeader.IsLeaf) break;
// Go to first child (P0)
// Internal node format: [Header] [P0] [Entry1] ...
var dataOffset = 32 + 20;
if (_currentHeader.IsLeaf) break;
// Go to first child (P0)
// Internal node format: [Header] [P0] [Entry1] ...
int dataOffset = 32 + 20;
pageId = BitConverter.ToUInt32(_pageBuffer.AsSpan(dataOffset, 4));
}
return PositionAtStart();
}
/// <summary>
/// Moves the cursor to the last entry in the index.
/// </summary>
/// <returns><see langword="true"/> if an entry is available; otherwise, <see langword="false"/>.</returns>
public bool MoveToLast()
{
return PositionAtStart();
}
/// <summary>
/// Moves the cursor to the last entry in the index.
/// </summary>
/// <returns><see langword="true" /> if an entry is available; otherwise, <see langword="false" />.</returns>
public bool MoveToLast()
{
// Find right-most leaf
var pageId = _index.RootPageId;
uint pageId = _index.RootPageId;
while (true)
{
LoadPage(pageId);
if (_currentHeader.IsLeaf) break;
// Go to last child (last pointer)
// Iterate all entries to find last pointer
// P0 is at 32+20 (4 bytes). Entry 0 starts at 32+20+4.
// Wait, we need the last pointer.
// P0 is at offset.
// Then EncryCount entries: Key + Pointer.
// We want the last pointer.
// Re-read P0 just in case
uint lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(32 + 20, 4));
if (_currentHeader.IsLeaf) break;
var offset = 32 + 20 + 4;
for (int i = 0; i < _currentHeader.EntryCount; i++)
// Go to last child (last pointer)
// Iterate all entries to find last pointer
// P0 is at 32+20 (4 bytes). Entry 0 starts at 32+20+4.
// Wait, we need the last pointer.
// P0 is at offset.
// Then EncryCount entries: Key + Pointer.
// We want the last pointer.
// Re-read P0 just in case
var lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(32 + 20, 4));
int offset = 32 + 20 + 4;
for (var i = 0; i < _currentHeader.EntryCount; i++)
{
var keyLen = BitConverter.ToInt32(_pageBuffer.AsSpan(offset, 4));
offset += 4 + keyLen;
lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(offset, 4));
offset += 4;
}
pageId = lastPointer;
}
return PositionAtEnd();
}
/// <summary>
/// Seeks to the specified key or the next greater key.
/// </summary>
/// <param name="key">The key to seek.</param>
/// <returns>
/// <see langword="true"/> if an exact key match is found; otherwise, <see langword="false"/>.
/// </returns>
public bool Seek(IndexKey key)
{
return PositionAtEnd();
}
/// <summary>
/// Seeks to the specified key or the next greater key.
/// </summary>
/// <param name="key">The key to seek.</param>
/// <returns>
/// <see langword="true" /> if an exact key match is found; otherwise, <see langword="false" />.
/// </returns>
public bool Seek(IndexKey key)
{
// Use Index to find leaf
var leafPageId = _index.FindLeafNode(key, _transactionId);
uint leafPageId = _index.FindLeafNode(key, _transactionId);
LoadPage(leafPageId);
ParseEntries();
// Binary search in entries
var idx = _currentEntries.BinarySearch(new IndexEntry(key, default(DocumentLocation)));
int idx = _currentEntries.BinarySearch(new IndexEntry(key, default(DocumentLocation)));
if (idx >= 0)
{
// Found exact match
@@ -133,51 +131,44 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
// Not found, ~idx is the next larger value
_currentEntryIndex = ~idx;
if (_currentEntryIndex < _currentEntries.Count)
{
// Not found, ~idx is the next larger value
_currentEntryIndex = ~idx;
if (_currentEntryIndex < _currentEntries.Count)
_isValid = true;
return false; // Positioned at next greater
}
// Key is larger than max in this page, move to next page
if (_currentHeader.NextLeafPageId != 0)
{
LoadPage(_currentHeader.NextLeafPageId);
ParseEntries();
_currentEntryIndex = 0;
if (_currentEntries.Count > 0)
{
_isValid = true;
return false; // Positioned at next greater
}
else
{
// Key is larger than max in this page, move to next page
if (_currentHeader.NextLeafPageId != 0)
{
LoadPage(_currentHeader.NextLeafPageId);
ParseEntries();
_currentEntryIndex = 0;
if (_currentEntries.Count > 0)
{
_isValid = true;
return false;
}
}
// End of index
_isValid = false;
return false;
}
}
}
/// <summary>
/// Moves the cursor to the next entry.
/// </summary>
/// <returns><see langword="true"/> if the cursor moved to a valid entry; otherwise, <see langword="false"/>.</returns>
public bool MoveNext()
{
}
}
// End of index
_isValid = false;
return false;
}
/// <summary>
/// Moves the cursor to the next entry.
/// </summary>
/// <returns><see langword="true" /> if the cursor moved to a valid entry; otherwise, <see langword="false" />.</returns>
public bool MoveNext()
{
if (!_isValid) return false;
_currentEntryIndex++;
if (_currentEntryIndex < _currentEntries.Count)
{
return true;
}
if (_currentEntryIndex < _currentEntries.Count) return true;
// Move to next page
if (_currentHeader.NextLeafPageId != 0)
@@ -186,23 +177,20 @@ internal sealed class BTreeCursor : IBTreeCursor
return PositionAtStart();
}
_isValid = false;
return false;
}
/// <summary>
/// Moves the cursor to the previous entry.
/// </summary>
/// <returns><see langword="true"/> if the cursor moved to a valid entry; otherwise, <see langword="false"/>.</returns>
public bool MovePrev()
{
_isValid = false;
return false;
}
/// <summary>
/// Moves the cursor to the previous entry.
/// </summary>
/// <returns><see langword="true" /> if the cursor moved to a valid entry; otherwise, <see langword="false" />.</returns>
public bool MovePrev()
{
if (!_isValid) return false;
_currentEntryIndex--;
if (_currentEntryIndex >= 0)
{
return true;
}
if (_currentEntryIndex >= 0) return true;
// Move to prev page
if (_currentHeader.PrevLeafPageId != 0)
@@ -211,9 +199,21 @@ internal sealed class BTreeCursor : IBTreeCursor
return PositionAtEnd();
}
_isValid = false;
return false;
}
_isValid = false;
return false;
}
/// <summary>
/// Releases cursor resources.
/// </summary>
public void Dispose()
{
if (_pageBuffer != null)
{
ArrayPool<byte>.Shared.Return(_pageBuffer);
_pageBuffer = null!;
}
}
private void LoadPage(uint pageId)
{
@@ -229,9 +229,9 @@ internal sealed class BTreeCursor : IBTreeCursor
// Helper to parse entries from current page buffer
// (Similar to BTreeIndex.ReadLeafEntries)
_currentEntries.Clear();
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
for (int i = 0; i < _currentHeader.EntryCount; i++)
for (var i = 0; i < _currentHeader.EntryCount; i++)
{
// Read Key
var keyLen = BitConverter.ToInt32(_pageBuffer.AsSpan(dataOffset, 4));
@@ -257,12 +257,10 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
// Empty page? Should not happen in helper logic unless root leaf is empty
_isValid = false;
return false;
}
// Empty page? Should not happen in helper logic unless root leaf is empty
_isValid = false;
return false;
}
private bool PositionAtEnd()
@@ -274,22 +272,8 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
_isValid = false;
return false;
}
}
/// <summary>
/// Releases cursor resources.
/// </summary>
public void Dispose()
{
if (_pageBuffer != null)
{
ArrayPool<byte>.Shared.Return(_pageBuffer);
_pageBuffer = null!;
}
_isValid = false;
return false;
}
}
}

783
src/CBDD.Core/Indexing/BTreeIndex.cs
View File

File diff suppressed because it is too large Load Diff

118
src/CBDD.Core/Indexing/BTreeStructures.cs
View File

@@ -1,27 +1,26 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents an entry in an index mapping a key to a document location.
/// Implemented as struct for memory efficiency.
/// </summary>
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Storage;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents an entry in an index mapping a key to a document location.
/// Implemented as struct for memory efficiency.
/// </summary>
public struct IndexEntry : IComparable<IndexEntry>, IComparable
{
/// <summary>
/// Gets or sets the index key.
/// Gets or sets the index key.
/// </summary>
public IndexKey Key { get; set; }
/// <summary>
/// Gets or sets the document location for the key.
/// Gets or sets the document location for the key.
/// </summary>
public DocumentLocation Location { get; set; }
/// <summary>
/// Initializes a new instance of the <see cref="IndexEntry"/> struct.
/// Initializes a new instance of the <see cref="IndexEntry" /> struct.
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
@@ -34,7 +33,7 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
// Backward compatibility: constructor that takes ObjectId (for migration)
// Will be removed once all code is migrated
/// <summary>
/// Initializes a legacy instance of the <see cref="IndexEntry"/> struct for migration scenarios.
/// Initializes a legacy instance of the <see cref="IndexEntry" /> struct for migration scenarios.
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="documentId">The legacy document identifier.</param>
@@ -47,12 +46,12 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
}
/// <summary>
/// Compares this entry to another entry by key.
/// Compares this entry to another entry by key.
/// </summary>
/// <param name="other">The other index entry to compare.</param>
/// <returns>
/// A value less than zero if this instance is less than <paramref name="other"/>,
/// zero if they are equal, or greater than zero if this instance is greater.
/// A value less than zero if this instance is less than <paramref name="other" />,
/// zero if they are equal, or greater than zero if this instance is greater.
/// </returns>
public int CompareTo(IndexEntry other)
{
@@ -60,76 +59,76 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
}
/// <summary>
/// Compares this entry to another object.
/// Compares this entry to another object.
/// </summary>
/// <param name="obj">The object to compare.</param>
/// <returns>
/// A value less than zero if this instance is less than <paramref name="obj"/>,
/// zero if they are equal, or greater than zero if this instance is greater.
/// A value less than zero if this instance is less than <paramref name="obj" />,
/// zero if they are equal, or greater than zero if this instance is greater.
/// </returns>
/// <exception cref="ArgumentException">Thrown when <paramref name="obj"/> is not an <see cref="IndexEntry"/>.</exception>
/// <exception cref="ArgumentException">Thrown when <paramref name="obj" /> is not an <see cref="IndexEntry" />.</exception>
public int CompareTo(object? obj)
{
if (obj is IndexEntry other) return CompareTo(other);
throw new ArgumentException("Object is not an IndexEntry");
}
}
/// <summary>
/// B+Tree node for index storage.
/// Uses struct for node metadata to minimize allocations.
/// </summary>
}
}
/// <summary>
/// B+Tree node for index storage.
/// Uses struct for node metadata to minimize allocations.
/// </summary>
public struct BTreeNodeHeader
{
/// <summary>
/// Gets or sets the page identifier.
/// Gets or sets the page identifier.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether this node is a leaf node.
/// Gets or sets a value indicating whether this node is a leaf node.
/// </summary>
public bool IsLeaf { get; set; }
/// <summary>
/// Gets or sets the number of entries in the node.
/// Gets or sets the number of entries in the node.
/// </summary>
public ushort EntryCount { get; set; }
/// <summary>
/// Gets or sets the parent page identifier.
/// Gets or sets the parent page identifier.
/// </summary>
public uint ParentPageId { get; set; }
/// <summary>
/// Gets or sets the next leaf page identifier.
/// Gets or sets the next leaf page identifier.
/// </summary>
public uint NextLeafPageId { get; set; } // For leaf nodes only
public uint NextLeafPageId { get; set; } // For leaf nodes only
/// <summary>
/// Gets or sets the previous leaf page identifier.
/// Gets or sets the previous leaf page identifier.
/// </summary>
public uint PrevLeafPageId { get; set; } // For leaf nodes only (added for reverse scan)
public uint PrevLeafPageId { get; set; } // For leaf nodes only (added for reverse scan)
/// <summary>
/// Writes the header to a byte span.
/// Writes the header to a byte span.
/// </summary>
/// <param name="destination">The destination span.</param>
public void WriteTo(Span<byte> destination)
{
if (destination.Length < 20)
throw new ArgumentException("Destination must be at least 20 bytes");
BitConverter.TryWriteBytes(destination[0..4], PageId);
destination[4] = (byte)(IsLeaf ? 1 : 0);
BitConverter.TryWriteBytes(destination[5..7], EntryCount);
BitConverter.TryWriteBytes(destination[7..11], ParentPageId);
BitConverter.TryWriteBytes(destination[..4], PageId);
destination[4] = (byte)(IsLeaf ? 1 : 0);
BitConverter.TryWriteBytes(destination[5..7], EntryCount);
BitConverter.TryWriteBytes(destination[7..11], ParentPageId);
BitConverter.TryWriteBytes(destination[11..15], NextLeafPageId);
BitConverter.TryWriteBytes(destination[15..19], PrevLeafPageId);
}
/// <summary>
/// Reads a node header from a byte span.
/// Reads a node header from a byte span.
/// </summary>
/// <param name="source">The source span.</param>
/// <returns>The parsed node header.</returns>
@@ -137,21 +136,18 @@ public struct BTreeNodeHeader
{
if (source.Length < 20)
throw new ArgumentException("Source must be at least 16 bytes");
var header = new BTreeNodeHeader
{
PageId = BitConverter.ToUInt32(source[0..4]),
IsLeaf = source[4] != 0,
EntryCount = BitConverter.ToUInt16(source[5..7]),
ParentPageId = BitConverter.ToUInt32(source[7..11]),
NextLeafPageId = BitConverter.ToUInt32(source[11..15])
};
if (source.Length >= 20)
{
header.PrevLeafPageId = BitConverter.ToUInt32(source[15..19]);
}
return header;
}
}
var header = new BTreeNodeHeader
{
PageId = BitConverter.ToUInt32(source[..4]),
IsLeaf = source[4] != 0,
EntryCount = BitConverter.ToUInt16(source[5..7]),
ParentPageId = BitConverter.ToUInt32(source[7..11]),
NextLeafPageId = BitConverter.ToUInt32(source[11..15])
};
if (source.Length >= 20) header.PrevLeafPageId = BitConverter.ToUInt32(source[15..19]);
return header;
}
}

267
src/CBDD.Core/Indexing/CollectionIndexDefinition.cs
View File

@@ -1,71 +1,26 @@
using System;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// High-level metadata and configuration for a custom index on a document collection.
/// Wraps low-level IndexOptions and provides strongly-typed expression-based key extraction.
/// High-level metadata and configuration for a custom index on a document collection.
/// Wraps low-level IndexOptions and provides strongly-typed expression-based key extraction.
/// </summary>
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionIndexDefinition<T> where T : class
{
/// <summary>
/// Unique name for this index (auto-generated or user-specified)
/// </summary>
public string Name { get; }
/// <summary>
/// Property paths that make up this index key.
/// Examples: ["Age"] for simple index, ["City", "Age"] for compound index
/// </summary>
public string[] PropertyPaths { get; }
/// <summary>
/// If true, enforces uniqueness constraint on the indexed values
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Type of index structure (from existing IndexType enum)
/// </summary>
public IndexType Type { get; }
/// <summary>Vector dimensions (only for Vector index)</summary>
public int Dimensions { get; }
/// <summary>Distance metric (only for Vector index)</summary>
public VectorMetric Metric { get; }
/// <summary>
/// Compiled function to extract the index key from a document.
/// Compiled for maximum performance (10-100x faster than interpreting Expression).
/// </summary>
public Func<T, object> KeySelector { get; }
/// <summary>
/// Original expression for the key selector (for analysis and serialization)
/// </summary>
public Expression<Func<T, object>> KeySelectorExpression { get; }
/// <summary>
/// If true, this is the primary key index (_id)
/// </summary>
public bool IsPrimary { get; }
/// <summary>
/// Creates a new index definition
/// Creates a new index definition
/// </summary>
/// <param name="name">Index name</param>
/// <param name="propertyPaths">Property paths for the index</param>
/// <param name="keySelectorExpression">Expression to extract key from document</param>
/// <param name="isUnique">Enforce uniqueness</param>
/// <param name="type">Index structure type (BTree or Hash)</param>
/// <param name="isPrimary">Is this the primary key index</param>
/// <param name="dimensions">The vector dimensions for vector indexes.</param>
/// <param name="metric">The distance metric for vector indexes.</param>
public CollectionIndexDefinition(
/// <param name="isUnique">Enforce uniqueness</param>
/// <param name="type">Index structure type (BTree or Hash)</param>
/// <param name="isPrimary">Is this the primary key index</param>
/// <param name="dimensions">The vector dimensions for vector indexes.</param>
/// <param name="metric">The distance metric for vector indexes.</param>
public CollectionIndexDefinition(
string name,
string[] propertyPaths,
Expression<Func<T, object>> keySelectorExpression,
@@ -76,11 +31,11 @@ public sealed class CollectionIndexDefinition<T> where T : class
VectorMetric metric = VectorMetric.Cosine)
{
if (string.IsNullOrWhiteSpace(name))
throw new ArgumentException("Index name cannot be empty", nameof(name));
throw new ArgumentException("Index name cannot be empty", nameof(name));
if (propertyPaths == null || propertyPaths.Length == 0)
throw new ArgumentException("Property paths cannot be empty", nameof(propertyPaths));
throw new ArgumentException("Property paths cannot be empty", nameof(propertyPaths));
Name = name;
PropertyPaths = propertyPaths;
KeySelectorExpression = keySelectorExpression ?? throw new ArgumentNullException(nameof(keySelectorExpression));
@@ -90,10 +45,53 @@ public sealed class CollectionIndexDefinition<T> where T : class
IsPrimary = isPrimary;
Dimensions = dimensions;
Metric = metric;
}
}
/// <summary>
/// Converts this high-level definition to low-level IndexOptions for BTreeIndex
/// Unique name for this index (auto-generated or user-specified)
/// </summary>
public string Name { get; }
/// <summary>
/// Property paths that make up this index key.
/// Examples: ["Age"] for simple index, ["City", "Age"] for compound index
/// </summary>
public string[] PropertyPaths { get; }
/// <summary>
/// If true, enforces uniqueness constraint on the indexed values
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Type of index structure (from existing IndexType enum)
/// </summary>
public IndexType Type { get; }
/// <summary>Vector dimensions (only for Vector index)</summary>
public int Dimensions { get; }
/// <summary>Distance metric (only for Vector index)</summary>
public VectorMetric Metric { get; }
/// <summary>
/// Compiled function to extract the index key from a document.
/// Compiled for maximum performance (10-100x faster than interpreting Expression).
/// </summary>
public Func<T, object> KeySelector { get; }
/// <summary>
/// Original expression for the key selector (for analysis and serialization)
/// </summary>
public Expression<Func<T, object>> KeySelectorExpression { get; }
/// <summary>
/// If true, this is the primary key index (_id)
/// </summary>
public bool IsPrimary { get; }
/// <summary>
/// Converts this high-level definition to low-level IndexOptions for BTreeIndex
/// </summary>
public IndexOptions ToIndexOptions()
{
@@ -105,98 +103,97 @@ public sealed class CollectionIndexDefinition<T> where T : class
Dimensions = Dimensions,
Metric = Metric
};
}
/// <summary>
/// Checks if this index can be used for a query on the specified property path
/// </summary>
/// <param name="propertyPath">The property path to validate.</param>
public bool CanSupportQuery(string propertyPath)
}
/// <summary>
/// Checks if this index can be used for a query on the specified property path
/// </summary>
/// <param name="propertyPath">The property path to validate.</param>
public bool CanSupportQuery(string propertyPath)
{
// Simple index: exact match required
if (PropertyPaths.Length == 1)
return PropertyPaths[0].Equals(propertyPath, StringComparison.OrdinalIgnoreCase);
// Compound index: can support if queried property is the first component
// e.g., index on ["City", "Age"] can support query on "City" but not just "Age"
return PropertyPaths[0].Equals(propertyPath, StringComparison.OrdinalIgnoreCase);
// Compound index: can support if queried property is the first component
// e.g., index on ["City", "Age"] can support query on "City" but not just "Age"
return PropertyPaths[0].Equals(propertyPath, StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// Checks if this index can support queries on multiple properties (compound queries)
/// </summary>
/// <param name="propertyPaths">The ordered property paths to validate.</param>
public bool CanSupportCompoundQuery(string[] propertyPaths)
}
/// <summary>
/// Checks if this index can support queries on multiple properties (compound queries)
/// </summary>
/// <param name="propertyPaths">The ordered property paths to validate.</param>
public bool CanSupportCompoundQuery(string[] propertyPaths)
{
if (propertyPaths == null || propertyPaths.Length == 0)
return false;
// Check if queried paths are a prefix of this index
// e.g., index on ["City", "Age", "Name"] can support ["City"] or ["City", "Age"]
return false;
// Check if queried paths are a prefix of this index
// e.g., index on ["City", "Age", "Name"] can support ["City"] or ["City", "Age"]
if (propertyPaths.Length > PropertyPaths.Length)
return false;
for (int i = 0; i < propertyPaths.Length; i++)
{
return false;
for (var i = 0; i < propertyPaths.Length; i++)
if (!PropertyPaths[i].Equals(propertyPaths[i], StringComparison.OrdinalIgnoreCase))
return false;
}
return true;
}
/// <inheritdoc />
public override string ToString()
/// <inheritdoc />
public override string ToString()
{
var uniqueStr = IsUnique ? "Unique" : "Non-Unique";
var paths = string.Join(", ", PropertyPaths);
string uniqueStr = IsUnique ? "Unique" : "Non-Unique";
string paths = string.Join(", ", PropertyPaths);
return $"{Name} ({uniqueStr} {Type} on [{paths}])";
}
}
/// <summary>
/// Information about an existing index (for querying index metadata)
/// Information about an existing index (for querying index metadata)
/// </summary>
public sealed class CollectionIndexInfo
{
/// <summary>
/// Gets the index name.
/// </summary>
public string Name { get; init; } = string.Empty;
/// <summary>
/// Gets the indexed property paths.
/// </summary>
public string[] PropertyPaths { get; init; } = Array.Empty<string>();
/// <summary>
/// Gets a value indicating whether the index is unique.
/// </summary>
public bool IsUnique { get; init; }
/// <summary>
/// Gets the index type.
/// </summary>
public IndexType Type { get; init; }
/// <summary>
/// Gets a value indicating whether this index is the primary index.
/// </summary>
public bool IsPrimary { get; init; }
/// <summary>
/// Gets the estimated number of indexed documents.
/// </summary>
public long EstimatedDocumentCount { get; init; }
/// <summary>
/// Gets the estimated storage size, in bytes.
/// </summary>
public long EstimatedSizeBytes { get; init; }
/// <inheritdoc />
public override string ToString()
public sealed class CollectionIndexInfo
{
/// <summary>
/// Gets the index name.
/// </summary>
public string Name { get; init; } = string.Empty;
/// <summary>
/// Gets the indexed property paths.
/// </summary>
public string[] PropertyPaths { get; init; } = Array.Empty<string>();
/// <summary>
/// Gets a value indicating whether the index is unique.
/// </summary>
public bool IsUnique { get; init; }
/// <summary>
/// Gets the index type.
/// </summary>
public IndexType Type { get; init; }
/// <summary>
/// Gets a value indicating whether this index is the primary index.
/// </summary>
public bool IsPrimary { get; init; }
/// <summary>
/// Gets the estimated number of indexed documents.
/// </summary>
public long EstimatedDocumentCount { get; init; }
/// <summary>
/// Gets the estimated storage size, in bytes.
/// </summary>
public long EstimatedSizeBytes { get; init; }
/// <inheritdoc />
public override string ToString()
{
return $"{Name}: {string.Join(", ", PropertyPaths)} ({EstimatedDocumentCount} docs, {EstimatedSizeBytes:N0} bytes)";
return
$"{Name}: {string.Join(", ", PropertyPaths)} ({EstimatedDocumentCount} docs, {EstimatedSizeBytes:N0} bytes)";
}
}
}

579
src/CBDD.Core/Indexing/CollectionIndexManager.cs
View File

@@ -1,7 +1,4 @@
using System;
using System.Collections.Generic;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
@@ -9,57 +6,94 @@ using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Manages a collection of secondary indexes on a document collection.
/// Handles index creation, deletion, automatic selection, and maintenance.
/// Manages a collection of secondary indexes on a document collection.
/// Handles index creation, deletion, automatic selection, and maintenance.
/// </summary>
/// <typeparam name="TId">Primary key type</typeparam>
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
{
private readonly Dictionary<string, CollectionSecondaryIndex<TId, T>> _indexes;
private readonly IStorageEngine _storage;
private readonly IDocumentMapper<TId, T> _mapper;
public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
{
private readonly string _collectionName;
private readonly Dictionary<string, CollectionSecondaryIndex<TId, T>> _indexes;
private readonly object _lock = new();
private readonly IDocumentMapper<TId, T> _mapper;
private readonly IStorageEngine _storage;
private bool _disposed;
private readonly string _collectionName;
private CollectionMetadata _metadata;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}"/> class.
/// </summary>
/// <param name="storage">The storage engine used to persist index data and metadata.</param>
/// <param name="mapper">The document mapper for the collection type.</param>
/// <param name="collectionName">The optional collection name override.</param>
public CollectionIndexManager(StorageEngine storage, IDocumentMapper<TId, T> mapper, string? collectionName = null)
: this((IStorageEngine)storage, mapper, collectionName)
{
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}"/> class from the storage abstraction.
/// </summary>
/// <param name="storage">The storage abstraction used to persist index state.</param>
/// <param name="mapper">The document mapper for the collection.</param>
/// <param name="collectionName">An optional collection name override.</param>
internal CollectionIndexManager(IStorageEngine storage, IDocumentMapper<TId, T> mapper, string? collectionName = null)
{
private CollectionMetadata _metadata;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}" /> class.
/// </summary>
/// <param name="storage">The storage engine used to persist index data and metadata.</param>
/// <param name="mapper">The document mapper for the collection type.</param>
/// <param name="collectionName">The optional collection name override.</param>
public CollectionIndexManager(StorageEngine storage, IDocumentMapper<TId, T> mapper, string? collectionName = null)
: this((IStorageEngine)storage, mapper, collectionName)
{
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}" /> class from the storage abstraction.
/// </summary>
/// <param name="storage">The storage abstraction used to persist index state.</param>
/// <param name="mapper">The document mapper for the collection.</param>
/// <param name="collectionName">An optional collection name override.</param>
internal CollectionIndexManager(IStorageEngine storage, IDocumentMapper<TId, T> mapper,
string? collectionName = null)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_mapper = mapper ?? throw new ArgumentNullException(nameof(mapper));
_collectionName = collectionName ?? _mapper.CollectionName;
_indexes = new Dictionary<string, CollectionSecondaryIndex<TId, T>>(StringComparer.OrdinalIgnoreCase);
// Load existing metadata via storage
_metadata = _storage.GetCollectionMetadata(_collectionName) ?? new CollectionMetadata { Name = _collectionName };
// Initialize indexes from metadata
_indexes = new Dictionary<string, CollectionSecondaryIndex<TId, T>>(StringComparer.OrdinalIgnoreCase);
// Load existing metadata via storage
_metadata = _storage.GetCollectionMetadata(_collectionName) ??
new CollectionMetadata { Name = _collectionName };
// Initialize indexes from metadata
foreach (var idxMeta in _metadata.Indexes)
{
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type, idxMeta.Dimensions, idxMeta.Metric);
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type,
idxMeta.Dimensions, idxMeta.Metric);
var index = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper, idxMeta.RootPageId);
_indexes[idxMeta.Name] = index;
}
}
/// <summary>
/// Gets the root page identifier for the primary index.
/// </summary>
public uint PrimaryRootPageId => _metadata.PrimaryRootPageId;
/// <summary>
/// Releases resources used by the index manager.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// No auto-save on dispose to avoid unnecessary I/O if no changes
lock (_lock)
{
foreach (var index in _indexes.Values)
try
{
index.Dispose();
}
catch
{
/* Best effort */
}
_indexes.Clear();
_disposed = true;
}
GC.SuppressFinalize(this);
}
private void UpdateMetadata()
{
_metadata.Indexes.Clear();
@@ -80,7 +114,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Creates a new secondary index
/// Creates a new secondary index
/// </summary>
/// <param name="definition">Index definition</param>
/// <returns>The created secondary index</returns>
@@ -100,9 +134,9 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
// Create secondary index
var secondaryIndex = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper);
_indexes[definition.Name] = secondaryIndex;
// Persist metadata
_indexes[definition.Name] = secondaryIndex;
// Persist metadata
UpdateMetadata();
_storage.SaveCollectionMetadata(_metadata);
@@ -113,7 +147,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
// ... methods ...
/// <summary>
/// Creates a simple index on a single property
/// Creates a simple index on a single property
/// </summary>
/// <typeparam name="TKey">Key type</typeparam>
/// <param name="keySelector">Expression to extract key from document</param>
@@ -129,9 +163,9 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
throw new ArgumentNullException(nameof(keySelector));
// Extract property paths from expression
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
// Generate name if not provided
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
// Generate name if not provided
name ??= GenerateIndexName(propertyPaths);
// Convert expression to object-returning expression (required for definition)
@@ -149,52 +183,51 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return CreateIndex(definition);
}
/// <summary>
/// Creates a vector index for a collection property.
/// </summary>
/// <typeparam name="TKey">The selected key type.</typeparam>
/// <param name="keySelector">Expression to extract the indexed field.</param>
/// <param name="dimensions">Vector dimensionality.</param>
/// <param name="metric">Distance metric used by the vector index.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
/// <summary>
/// Creates a vector index for a collection property.
/// </summary>
/// <typeparam name="TKey">The selected key type.</typeparam>
/// <param name="keySelector">Expression to extract the indexed field.</param>
/// <param name="dimensions">Vector dimensionality.</param>
/// <param name="metric">Distance metric used by the vector index.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector,
int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
{
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
if (_indexes.TryGetValue(indexName, out var existing))
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
// Reuse the original parameter from keySelector to avoid invalid expression trees.
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector, false, dimensions, metric);
return CreateIndex(definition);
}
}
var body = keySelector.Body;
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
/// <summary>
/// Ensures that an index exists for the specified key selector.
/// </summary>
/// <param name="keySelector">Expression to extract the indexed field.</param>
/// <param name="name">Optional index name.</param>
/// <param name="unique">Whether the index enforces uniqueness.</param>
/// <returns>The existing or newly created index.</returns>
public CollectionSecondaryIndex<TId, T> EnsureIndex(
Expression<Func<T, object>> keySelector,
string? name = null,
bool unique = false)
// Reuse the original parameter from keySelector to avoid invalid expression trees.
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector,
false, dimensions, metric);
return CreateIndex(definition);
}
}
/// <summary>
/// Ensures that an index exists for the specified key selector.
/// </summary>
/// <param name="keySelector">Expression to extract the indexed field.</param>
/// <param name="name">Optional index name.</param>
/// <param name="unique">Whether the index enforces uniqueness.</param>
/// <returns>The existing or newly created index.</returns>
public CollectionSecondaryIndex<TId, T> EnsureIndex(
Expression<Func<T, object>> keySelector,
string? name = null,
bool unique = false)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
name ??= GenerateIndexName(propertyPaths);
lock (_lock)
@@ -206,46 +239,43 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
}
/// <summary>
/// Ensures that an index exists for the specified untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="name">Optional index name.</param>
/// <param name="unique">Whether the index enforces uniqueness.</param>
/// <returns>The existing or newly created index.</returns>
internal CollectionSecondaryIndex<TId, T> EnsureIndexUntyped(
LambdaExpression keySelector,
string? name = null,
bool unique = false)
/// <summary>
/// Ensures that an index exists for the specified untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="name">Optional index name.</param>
/// <param name="unique">Whether the index enforces uniqueness.</param>
/// <returns>The existing or newly created index.</returns>
internal CollectionSecondaryIndex<TId, T> EnsureIndexUntyped(
LambdaExpression keySelector,
string? name = null,
bool unique = false)
{
// Convert LambdaExpression to Expression<Func<T, object>> properly by sharing parameters
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
return EnsureIndex(lambda, name, unique);
}
/// <summary>
/// Creates a vector index from an untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="dimensions">Vector dimensionality.</param>
/// <param name="metric">Distance metric used by the vector index.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateVectorIndexUntyped(
LambdaExpression keySelector,
int dimensions,
VectorMetric metric = VectorMetric.Cosine,
string? name = null)
/// <summary>
/// Creates a vector index from an untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="dimensions">Vector dimensionality.</param>
/// <param name="metric">Distance metric used by the vector index.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateVectorIndexUntyped(
LambdaExpression keySelector,
int dimensions,
VectorMetric metric = VectorMetric.Cosine,
string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
@@ -253,51 +283,47 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector, false, dimensions, metric);
return CreateIndex(definition);
}
}
/// <summary>
/// Creates a spatial index from an untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateSpatialIndexUntyped(
LambdaExpression keySelector,
string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
if (_indexes.TryGetValue(indexName, out var existing))
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Spatial);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector,
false, dimensions, metric);
return CreateIndex(definition);
}
}
/// <summary>
/// Drops an existing index by name
/// Creates a spatial index from an untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateSpatialIndexUntyped(
LambdaExpression keySelector,
string? name = null)
{
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
if (_indexes.TryGetValue(indexName, out var existing))
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition =
new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Spatial);
return CreateIndex(definition);
}
}
/// <summary>
/// Drops an existing index by name
/// </summary>
/// <param name="name">Index name</param>
/// <returns>True if index was found and dropped, false otherwise</returns>
@@ -311,10 +337,10 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
if (_indexes.TryGetValue(name, out var index))
{
index.Dispose();
_indexes.Remove(name);
// TODO: Free pages used by index in PageFile
_indexes.Remove(name);
// TODO: Free pages used by index in PageFile
SaveMetadata(); // Save metadata after dropping index
return true;
}
@@ -323,11 +349,11 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
}
/// <summary>
/// Gets an index by name
/// </summary>
/// <param name="name">The index name.</param>
public CollectionSecondaryIndex<TId, T>? GetIndex(string name)
/// <summary>
/// Gets an index by name
/// </summary>
/// <param name="name">The index name.</param>
public CollectionSecondaryIndex<TId, T>? GetIndex(string name)
{
lock (_lock)
{
@@ -336,7 +362,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Gets all indexes
/// Gets all indexes
/// </summary>
public IEnumerable<CollectionSecondaryIndex<TId, T>> GetAllIndexes()
{
@@ -347,7 +373,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Gets information about all indexes
/// Gets information about all indexes
/// </summary>
public IEnumerable<CollectionIndexInfo> GetIndexInfo()
{
@@ -358,8 +384,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Finds the best index to use for a query on the specified property.
/// Returns null if no suitable index found (requires full scan).
/// Finds the best index to use for a query on the specified property.
/// Returns null if no suitable index found (requires full scan).
/// </summary>
/// <param name="propertyPath">Property path being queried</param>
/// <returns>Best index for the query, or null if none suitable</returns>
@@ -386,11 +412,11 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
}
/// <summary>
/// Finds the best index for a compound query on multiple properties
/// </summary>
/// <param name="propertyPaths">The ordered list of queried property paths.</param>
public CollectionSecondaryIndex<TId, T>? FindBestCompoundIndex(string[] propertyPaths)
/// <summary>
/// Finds the best index for a compound query on multiple properties
/// </summary>
/// <param name="propertyPaths">The ordered list of queried property paths.</param>
public CollectionSecondaryIndex<TId, T>? FindBestCompoundIndex(string[] propertyPaths)
{
if (propertyPaths == null || propertyPaths.Length == 0)
return null;
@@ -413,7 +439,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Inserts a document into all indexes
/// Inserts a document into all indexes
/// </summary>
/// <param name="document">Document to insert</param>
/// <param name="location">Physical location of the document</param>
@@ -425,22 +451,20 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Insert(document, location, transaction);
}
foreach (var index in _indexes.Values) index.Insert(document, location, transaction);
}
}
/// <summary>
/// Updates a document in all indexes
/// Updates a document in all indexes
/// </summary>
/// <param name="oldDocument">Old version of document</param>
/// <param name="newDocument">New version of document</param>
/// <param name="oldLocation">Physical location of old document</param>
/// <param name="newLocation">Physical location of new document</param>
/// <param name="transaction">Transaction context</param>
public void UpdateInAll(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation, ITransaction transaction)
public void UpdateInAll(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation,
ITransaction transaction)
{
if (oldDocument == null)
throw new ArgumentNullException(nameof(oldDocument));
@@ -450,14 +474,12 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Update(oldDocument, newDocument, oldLocation, newLocation, transaction);
}
}
}
/// <summary>
/// Deletes a document from all indexes
/// Deletes a document from all indexes
/// </summary>
/// <param name="document">Document to delete</param>
/// <param name="location">Physical location of the document</param>
@@ -469,83 +491,78 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Delete(document, location, transaction);
}
foreach (var index in _indexes.Values) index.Delete(document, location, transaction);
}
}
/// <summary>
/// Generates an index name from property paths
/// Generates an index name from property paths
/// </summary>
private static string GenerateIndexName(string[] propertyPaths)
{
return $"idx_{string.Join("_", propertyPaths)}";
}
private CollectionIndexDefinition<T> RebuildDefinition(string name, string[] paths, bool isUnique, IndexType type, int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
private CollectionIndexDefinition<T> RebuildDefinition(string name, string[] paths, bool isUnique, IndexType type,
int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
{
var param = Expression.Parameter(typeof(T), "u");
Expression body;
Expression body;
if (paths.Length == 1)
{
body = Expression.PropertyOrField(param, paths[0]);
}
else
{
body = Expression.NewArrayInit(typeof(object),
body = Expression.NewArrayInit(typeof(object),
paths.Select(p => Expression.Convert(Expression.PropertyOrField(param, p), typeof(object))));
}
var objectBody = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(objectBody, param);
var lambda = Expression.Lambda<Func<T, object>>(objectBody, param);
return new CollectionIndexDefinition<T>(name, paths, lambda, isUnique, type, false, dimensions, metric);
}
/// <summary>
/// Gets the root page identifier for the primary index.
/// </summary>
public uint PrimaryRootPageId => _metadata.PrimaryRootPageId;
/// <summary>
/// Rebinds cached metadata and index instances from persisted metadata.
/// </summary>
/// <param name="metadata">The collection metadata used to rebuild index state.</param>
internal void RebindFromMetadata(CollectionMetadata metadata)
{
if (metadata == null)
throw new ArgumentNullException(nameof(metadata));
lock (_lock)
{
if (_disposed)
throw new ObjectDisposedException(nameof(CollectionIndexManager<TId, T>));
foreach (var index in _indexes.Values)
{
try { index.Dispose(); } catch { /* Best effort */ }
}
_indexes.Clear();
_metadata = metadata;
foreach (var idxMeta in _metadata.Indexes)
{
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type, idxMeta.Dimensions, idxMeta.Metric);
var index = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper, idxMeta.RootPageId);
_indexes[idxMeta.Name] = index;
}
}
}
/// <summary>
/// Sets the root page identifier for the primary index.
/// </summary>
/// <param name="pageId">The root page identifier.</param>
public void SetPrimaryRootPageId(uint pageId)
/// <summary>
/// Rebinds cached metadata and index instances from persisted metadata.
/// </summary>
/// <param name="metadata">The collection metadata used to rebuild index state.</param>
internal void RebindFromMetadata(CollectionMetadata metadata)
{
if (metadata == null)
throw new ArgumentNullException(nameof(metadata));
lock (_lock)
{
if (_disposed)
throw new ObjectDisposedException(nameof(CollectionIndexManager<TId, T>));
foreach (var index in _indexes.Values)
try
{
index.Dispose();
}
catch
{
/* Best effort */
}
_indexes.Clear();
_metadata = metadata;
foreach (var idxMeta in _metadata.Indexes)
{
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type,
idxMeta.Dimensions, idxMeta.Metric);
var index = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper, idxMeta.RootPageId);
_indexes[idxMeta.Name] = index;
}
}
}
/// <summary>
/// Sets the root page identifier for the primary index.
/// </summary>
/// <param name="pageId">The root page identifier.</param>
public void SetPrimaryRootPageId(uint pageId)
{
lock (_lock)
{
@@ -557,88 +574,62 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
}
/// <summary>
/// Gets the current collection metadata.
/// </summary>
/// <returns>The collection metadata.</returns>
public CollectionMetadata GetMetadata() => _metadata;
private void SaveMetadata()
{
UpdateMetadata();
_storage.SaveCollectionMetadata(_metadata);
}
/// <summary>
/// Releases resources used by the index manager.
/// </summary>
public void Dispose()
/// <summary>
/// Gets the current collection metadata.
/// </summary>
/// <returns>The collection metadata.</returns>
public CollectionMetadata GetMetadata()
{
if (_disposed)
return;
// No auto-save on dispose to avoid unnecessary I/O if no changes
lock (_lock)
{
foreach (var index in _indexes.Values)
{
try { index.Dispose(); } catch { /* Best effort */ }
}
_indexes.Clear();
_disposed = true;
}
return _metadata;
}
GC.SuppressFinalize(this);
private void SaveMetadata()
{
UpdateMetadata();
_storage.SaveCollectionMetadata(_metadata);
}
}
/// <summary>
/// Helper class to analyze LINQ expressions and extract property paths
/// Helper class to analyze LINQ expressions and extract property paths
/// </summary>
public static class ExpressionAnalyzer
{
/// <summary>
/// Extracts property paths from a lambda expression.
/// Supports simple property access (p => p.Age) and anonymous types (p => new { p.City, p.Age }).
/// </summary>
/// <param name="expression">The lambda expression to analyze.</param>
public static string[] ExtractPropertyPaths(LambdaExpression expression)
/// <summary>
/// Extracts property paths from a lambda expression.
/// Supports simple property access (p => p.Age) and anonymous types (p => new { p.City, p.Age }).
/// </summary>
/// <param name="expression">The lambda expression to analyze.</param>
public static string[] ExtractPropertyPaths(LambdaExpression expression)
{
if (expression.Body is MemberExpression memberExpr)
{
// Simple property: p => p.Age
return new[] { memberExpr.Member.Name };
}
else if (expression.Body is NewExpression newExpr)
{
if (expression.Body is NewExpression newExpr)
// Compound key via anonymous type: p => new { p.City, p.Age }
return newExpr.Arguments
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
else if (expression.Body is UnaryExpression { NodeType: ExpressionType.Convert } unaryExpr)
if (expression.Body is UnaryExpression { NodeType: ExpressionType.Convert } unaryExpr)
{
// Handle Convert(Member) or Convert(New)
if (unaryExpr.Operand is MemberExpression innerMember)
{
// Wrapped property: p => (object)p.Age
return new[] { innerMember.Member.Name };
}
else if (unaryExpr.Operand is NewExpression innerNew)
{
// Wrapped anonymous type: p => (object)new { p.City, p.Age }
return innerNew.Arguments
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
if (unaryExpr.Operand is NewExpression innerNew)
// Wrapped anonymous type: p => (object)new { p.City, p.Age }
return innerNew.Arguments
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
throw new ArgumentException(
"Expression must be a property accessor (p => p.Property) or anonymous type (p => new { p.Prop1, p.Prop2 })",
nameof(expression));
}
}
}

531
src/CBDD.Core/Indexing/CollectionSecondaryIndex.cs
View File

@@ -1,101 +1,112 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using System;
using System.Linq;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents a secondary (non-primary) index on a document collection.
/// Provides a high-level, strongly-typed wrapper around the low-level BTreeIndex.
/// Handles automatic key extraction from documents using compiled expressions.
/// Represents a secondary (non-primary) index on a document collection.
/// Provides a high-level, strongly-typed wrapper around the low-level BTreeIndex.
/// Handles automatic key extraction from documents using compiled expressions.
/// </summary>
/// <typeparam name="TId">Primary key type</typeparam>
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : class
{
private readonly CollectionIndexDefinition<T> _definition;
private readonly BTreeIndex? _btreeIndex;
private readonly VectorSearchIndex? _vectorIndex;
private readonly RTreeIndex? _spatialIndex;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly RTreeIndex? _spatialIndex;
private readonly VectorSearchIndex? _vectorIndex;
private bool _disposed;
/// <summary>
/// Gets the index definition
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}" /> class.
/// </summary>
public CollectionIndexDefinition<T> Definition => _definition;
/// <summary>
/// Gets the underlying BTree index (for advanced scenarios)
/// </summary>
public BTreeIndex? BTreeIndex => _btreeIndex;
/// <summary>
/// Gets the root page identifier for the underlying index structure.
/// </summary>
public uint RootPageId => _btreeIndex?.RootPageId ?? _vectorIndex?.RootPageId ?? _spatialIndex?.RootPageId ?? 0;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}"/> class.
/// </summary>
/// <param name="definition">The index definition.</param>
/// <param name="storage">The storage engine.</param>
/// <param name="mapper">The document mapper.</param>
/// <param name="rootPageId">The existing root page ID, or <c>0</c> to create a new one.</param>
public CollectionSecondaryIndex(
CollectionIndexDefinition<T> definition,
StorageEngine storage,
IDocumentMapper<TId, T> mapper,
uint rootPageId = 0)
: this(definition, (IStorageEngine)storage, mapper, rootPageId)
{
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}"/> class from index storage abstractions.
/// </summary>
/// <param name="definition">The index definition.</param>
/// <param name="storage">The index storage abstraction.</param>
/// <param name="mapper">The document mapper.</param>
/// <param name="rootPageId">The existing root page identifier, if any.</param>
internal CollectionSecondaryIndex(
CollectionIndexDefinition<T> definition,
IIndexStorage storage,
IDocumentMapper<TId, T> mapper,
uint rootPageId = 0)
/// <param name="definition">The index definition.</param>
/// <param name="storage">The storage engine.</param>
/// <param name="mapper">The document mapper.</param>
/// <param name="rootPageId">The existing root page ID, or <c>0</c> to create a new one.</param>
public CollectionSecondaryIndex(
CollectionIndexDefinition<T> definition,
StorageEngine storage,
IDocumentMapper<TId, T> mapper,
uint rootPageId = 0)
: this(definition, (IStorageEngine)storage, mapper, rootPageId)
{
_definition = definition ?? throw new ArgumentNullException(nameof(definition));
_mapper = mapper ?? throw new ArgumentNullException(nameof(mapper));
var indexOptions = definition.ToIndexOptions();
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}" /> class from index storage
/// abstractions.
/// </summary>
/// <param name="definition">The index definition.</param>
/// <param name="storage">The index storage abstraction.</param>
/// <param name="mapper">The document mapper.</param>
/// <param name="rootPageId">The existing root page identifier, if any.</param>
internal CollectionSecondaryIndex(
CollectionIndexDefinition<T> definition,
IIndexStorage storage,
IDocumentMapper<TId, T> mapper,
uint rootPageId = 0)
{
Definition = definition ?? throw new ArgumentNullException(nameof(definition));
_mapper = mapper ?? throw new ArgumentNullException(nameof(mapper));
var indexOptions = definition.ToIndexOptions();
if (indexOptions.Type == IndexType.Vector)
{
_vectorIndex = new VectorSearchIndex(storage, indexOptions, rootPageId);
_btreeIndex = null;
BTreeIndex = null;
_spatialIndex = null;
}
else if (indexOptions.Type == IndexType.Spatial)
{
_spatialIndex = new RTreeIndex(storage, indexOptions, rootPageId);
_btreeIndex = null;
BTreeIndex = null;
_vectorIndex = null;
}
else
{
_btreeIndex = new BTreeIndex(storage, indexOptions, rootPageId);
BTreeIndex = new BTreeIndex(storage, indexOptions, rootPageId);
_vectorIndex = null;
_spatialIndex = null;
}
}
/// <summary>
/// Inserts a document into this index
/// Gets the index definition
/// </summary>
public CollectionIndexDefinition<T> Definition { get; }
/// <summary>
/// Gets the underlying BTree index (for advanced scenarios)
/// </summary>
public BTreeIndex? BTreeIndex { get; }
/// <summary>
/// Gets the root page identifier for the underlying index structure.
/// </summary>
public uint RootPageId => BTreeIndex?.RootPageId ?? _vectorIndex?.RootPageId ?? _spatialIndex?.RootPageId ?? 0;
/// <summary>
/// Releases resources used by this index wrapper.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// BTreeIndex doesn't currently implement IDisposable
// Future: may need to flush buffers, close resources
_disposed = true;
GC.SuppressFinalize(this);
}
/// <summary>
/// Inserts a document into this index
/// </summary>
/// <param name="document">Document to index</param>
/// <param name="location">Physical location of the document</param>
@@ -103,91 +114,84 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
public void Insert(T document, DocumentLocation location, ITransaction transaction)
{
if (document == null)
throw new ArgumentNullException(nameof(document));
// Extract key using compiled selector (fast!)
var keyValue = _definition.KeySelector(document);
throw new ArgumentNullException(nameof(document));
// Extract key using compiled selector (fast!)
object? keyValue = Definition.KeySelector(document);
if (keyValue == null)
return; // Skip null keys
return; // Skip null keys
if (_vectorIndex != null)
{
// Vector Index Support
if (keyValue is float[] singleVector)
{
_vectorIndex.Insert(singleVector, location, transaction);
}
else if (keyValue is IEnumerable<float[]> vectors)
{
foreach (var v in vectors)
{
foreach (float[] v in vectors)
_vectorIndex.Insert(v, location, transaction);
}
}
}
else if (_spatialIndex != null)
{
// Geospatial Index Support
if (keyValue is ValueTuple<double, double> t)
{
_spatialIndex.Insert(GeoBox.FromPoint(new GeoPoint(t.Item1, t.Item2)), location, transaction);
}
}
else if (_btreeIndex != null)
else if (BTreeIndex != null)
{
// BTree Index logic
var userKey = ConvertToIndexKey(keyValue);
var documentId = _mapper.GetId(document);
var compositeKey = CreateCompositeKey(userKey, _mapper.ToIndexKey(documentId));
_btreeIndex.Insert(compositeKey, location, transaction?.TransactionId);
BTreeIndex.Insert(compositeKey, location, transaction?.TransactionId);
}
}
/// <summary>
/// Updates a document in this index (delete old, insert new).
/// Only updates if the indexed key has changed.
/// Updates a document in this index (delete old, insert new).
/// Only updates if the indexed key has changed.
/// </summary>
/// <param name="oldDocument">Old version of document</param>
/// <param name="newDocument">New version of document</param>
/// <param name="oldLocation">Physical location of old document</param>
/// <param name="newLocation">Physical location of new document</param>
/// <param name="transaction">Optional transaction</param>
public void Update(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation, ITransaction transaction)
public void Update(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation,
ITransaction transaction)
{
if (oldDocument == null)
throw new ArgumentNullException(nameof(oldDocument));
if (newDocument == null)
throw new ArgumentNullException(nameof(newDocument));
// Extract keys from both versions
var oldKey = _definition.KeySelector(oldDocument);
var newKey = _definition.KeySelector(newDocument);
// If keys are the same, no index update needed (optimization)
throw new ArgumentNullException(nameof(newDocument));
// Extract keys from both versions
object? oldKey = Definition.KeySelector(oldDocument);
object? newKey = Definition.KeySelector(newDocument);
// If keys are the same, no index update needed (optimization)
if (Equals(oldKey, newKey))
return;
var documentId = _mapper.GetId(oldDocument);
// Delete old entry if it had a key
return;
var documentId = _mapper.GetId(oldDocument);
// Delete old entry if it had a key
if (oldKey != null)
{
var oldUserKey = ConvertToIndexKey(oldKey);
var oldCompositeKey = CreateCompositeKey(oldUserKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Delete(oldCompositeKey, oldLocation, transaction?.TransactionId);
}
// Insert new entry if it has a key
BTreeIndex?.Delete(oldCompositeKey, oldLocation, transaction?.TransactionId);
}
// Insert new entry if it has a key
if (newKey != null)
{
var newUserKey = ConvertToIndexKey(newKey);
var newCompositeKey = CreateCompositeKey(newUserKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Insert(newCompositeKey, newLocation, transaction?.TransactionId);
BTreeIndex?.Insert(newCompositeKey, newLocation, transaction?.TransactionId);
}
}
/// <summary>
/// Deletes a document from this index
/// Deletes a document from this index
/// </summary>
/// <param name="document">Document to remove from index</param>
/// <param name="location">Physical location of the document</param>
@@ -195,23 +199,23 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
public void Delete(T document, DocumentLocation location, ITransaction transaction)
{
if (document == null)
throw new ArgumentNullException(nameof(document));
// Extract key
var keyValue = _definition.KeySelector(document);
throw new ArgumentNullException(nameof(document));
// Extract key
object? keyValue = Definition.KeySelector(document);
if (keyValue == null)
return; // Nothing to delete
return; // Nothing to delete
var userKey = ConvertToIndexKey(keyValue);
var documentId = _mapper.GetId(document);
// Create composite key and delete
var documentId = _mapper.GetId(document);
// Create composite key and delete
var compositeKey = CreateCompositeKey(userKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Delete(compositeKey, location, transaction?.TransactionId);
BTreeIndex?.Delete(compositeKey, location, transaction?.TransactionId);
}
/// <summary>
/// Seeks a single document by exact key match (O(log n))
/// Seeks a single document by exact key match (O(log n))
/// </summary>
/// <param name="key">Key value to seek</param>
/// <param name="transaction">Optional transaction to read uncommitted changes</param>
@@ -219,68 +223,67 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
public DocumentLocation? Seek(object key, ITransaction? transaction = null)
{
if (key == null)
return null;
if (_vectorIndex != null && key is float[] query)
{
return _vectorIndex.Search(query, 1, transaction: transaction).FirstOrDefault().Location;
}
return null;
if (_btreeIndex != null)
if (_vectorIndex != null && key is float[] query)
return _vectorIndex.Search(query, 1, transaction: transaction).FirstOrDefault().Location;
if (BTreeIndex != null)
{
var userKey = ConvertToIndexKey(key);
var minComposite = CreateCompositeKeyBoundary(userKey, useMinObjectId: true);
var maxComposite = CreateCompositeKeyBoundary(userKey, useMinObjectId: false);
var firstEntry = _btreeIndex.Range(minComposite, maxComposite, IndexDirection.Forward, transaction?.TransactionId).FirstOrDefault();
return firstEntry.Location.PageId == 0 ? null : (DocumentLocation?)firstEntry.Location;
}
return null;
}
/// <summary>
/// Performs a vector nearest-neighbor search.
/// </summary>
/// <param name="query">The query vector.</param>
/// <param name="k">The number of results to return.</param>
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction.</param>
/// <returns>The matching vector search results.</returns>
public IEnumerable<VectorSearchResult> VectorSearch(float[] query, int k, int efSearch = 100, ITransaction? transaction = null)
{
if (_vectorIndex == null)
throw new InvalidOperationException("This index is not a vector index.");
var minComposite = CreateCompositeKeyBoundary(userKey, true);
var maxComposite = CreateCompositeKeyBoundary(userKey, false);
var firstEntry = BTreeIndex
.Range(minComposite, maxComposite, IndexDirection.Forward, transaction?.TransactionId).FirstOrDefault();
return firstEntry.Location.PageId == 0 ? null : firstEntry.Location;
}
return null;
}
/// <summary>
/// Performs a vector nearest-neighbor search.
/// </summary>
/// <param name="query">The query vector.</param>
/// <param name="k">The number of results to return.</param>
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction.</param>
/// <returns>The matching vector search results.</returns>
public IEnumerable<VectorSearchResult> VectorSearch(float[] query, int k, int efSearch = 100,
ITransaction? transaction = null)
{
if (_vectorIndex == null)
throw new InvalidOperationException("This index is not a vector index.");
return _vectorIndex.Search(query, k, efSearch, transaction);
}
/// <summary>
/// Performs geospatial distance search
/// </summary>
/// <param name="center">The center point.</param>
/// <param name="radiusKm">The search radius in kilometers.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Near((double Latitude, double Longitude) center, double radiusKm, ITransaction? transaction = null)
{
if (_spatialIndex == null)
/// <summary>
/// Performs geospatial distance search
/// </summary>
/// <param name="center">The center point.</param>
/// <param name="radiusKm">The search radius in kilometers.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Near((double Latitude, double Longitude) center, double radiusKm,
ITransaction? transaction = null)
{
if (_spatialIndex == null)
throw new InvalidOperationException("This index is not a spatial index.");
var queryBox = SpatialMath.BoundingBox(center.Latitude, center.Longitude, radiusKm);
foreach (var loc in _spatialIndex.Search(queryBox, transaction))
{
yield return loc;
}
foreach (var loc in _spatialIndex.Search(queryBox, transaction)) yield return loc;
}
/// <summary>
/// Performs geospatial bounding box search
/// </summary>
/// <param name="min">The minimum latitude/longitude corner.</param>
/// <param name="max">The maximum latitude/longitude corner.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Within((double Latitude, double Longitude) min, (double Latitude, double Longitude) max, ITransaction? transaction = null)
{
if (_spatialIndex == null)
/// <summary>
/// Performs geospatial bounding box search
/// </summary>
/// <param name="min">The minimum latitude/longitude corner.</param>
/// <param name="max">The maximum latitude/longitude corner.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Within((double Latitude, double Longitude) min,
(double Latitude, double Longitude) max, ITransaction? transaction = null)
{
if (_spatialIndex == null)
throw new InvalidOperationException("This index is not a spatial index.");
var area = new GeoBox(min.Latitude, min.Longitude, max.Latitude, max.Longitude);
@@ -288,21 +291,22 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
}
/// <summary>
/// Scans a range of keys (O(log n + k) where k is result count)
/// Scans a range of keys (O(log n + k) where k is result count)
/// </summary>
/// <param name="minKey">Minimum key (inclusive), null for unbounded</param>
/// <param name="maxKey">Maximum key (inclusive), null for unbounded</param>
/// <param name="direction">Scan direction.</param>
/// <param name="transaction">Optional transaction to read uncommitted changes</param>
/// <returns>Enumerable of document locations in key order</returns>
public IEnumerable<DocumentLocation> Range(object? minKey, object? maxKey, IndexDirection direction = IndexDirection.Forward, ITransaction? transaction = null)
/// <param name="minKey">Minimum key (inclusive), null for unbounded</param>
/// <param name="maxKey">Maximum key (inclusive), null for unbounded</param>
/// <param name="direction">Scan direction.</param>
/// <param name="transaction">Optional transaction to read uncommitted changes</param>
/// <returns>Enumerable of document locations in key order</returns>
public IEnumerable<DocumentLocation> Range(object? minKey, object? maxKey,
IndexDirection direction = IndexDirection.Forward, ITransaction? transaction = null)
{
if (_btreeIndex == null) yield break;
// Handle unbounded ranges
if (BTreeIndex == null) yield break;
// Handle unbounded ranges
IndexKey actualMinKey;
IndexKey actualMaxKey;
IndexKey actualMaxKey;
if (minKey == null && maxKey == null)
{
// Full scan - use extreme values
@@ -313,108 +317,53 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
actualMinKey = new IndexKey(new byte[0]);
var userMaxKey = ConvertToIndexKey(maxKey!);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, useMinObjectId: false); // Max boundary
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, false); // Max boundary
}
else if (maxKey == null)
{
var userMinKey = ConvertToIndexKey(minKey);
actualMinKey = CreateCompositeKeyBoundary(userMinKey, useMinObjectId: true); // Min boundary
actualMinKey = CreateCompositeKeyBoundary(userMinKey, true); // Min boundary
actualMaxKey = new IndexKey(Enumerable.Repeat((byte)0xFF, 255).ToArray());
}
else
{
// Both bounds specified
var userMinKey = ConvertToIndexKey(minKey);
var userMaxKey = ConvertToIndexKey(maxKey);
// Create composite boundaries:
// Min: (userMinKey, ObjectId.Empty) - captures all docs with key >= userMinKey
// Max: (userMaxKey, ObjectId.MaxValue) - captures all docs with key <= userMaxKey
actualMinKey = CreateCompositeKeyBoundary(userMinKey, useMinObjectId: true);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, useMinObjectId: false);
}
// Use BTreeIndex.Range with WAL-aware reads and direction
// Extract DocumentLocation from each entry
foreach (var entry in _btreeIndex.Range(actualMinKey, actualMaxKey, direction, transaction?.TransactionId))
{
yield return entry.Location;
var userMaxKey = ConvertToIndexKey(maxKey);
// Create composite boundaries:
// Min: (userMinKey, ObjectId.Empty) - captures all docs with key >= userMinKey
// Max: (userMaxKey, ObjectId.MaxValue) - captures all docs with key <= userMaxKey
actualMinKey = CreateCompositeKeyBoundary(userMinKey, true);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, false);
}
// Use BTreeIndex.Range with WAL-aware reads and direction
// Extract DocumentLocation from each entry
foreach (var entry in BTreeIndex.Range(actualMinKey, actualMaxKey, direction, transaction?.TransactionId))
yield return entry.Location;
}
/// <summary>
/// Gets statistics about this index
/// Gets statistics about this index
/// </summary>
public CollectionIndexInfo GetInfo()
{
return new CollectionIndexInfo
{
Name = _definition.Name,
PropertyPaths = _definition.PropertyPaths,
IsUnique = _definition.IsUnique,
Type = _definition.Type,
IsPrimary = _definition.IsPrimary,
Name = Definition.Name,
PropertyPaths = Definition.PropertyPaths,
IsUnique = Definition.IsUnique,
Type = Definition.Type,
IsPrimary = Definition.IsPrimary,
EstimatedDocumentCount = 0, // TODO: Track or calculate document count
EstimatedSizeBytes = 0 // TODO: Calculate index size
EstimatedSizeBytes = 0 // TODO: Calculate index size
};
}
#region Composite Key Support (SQLite-style for Duplicate Keys)
/// <summary>
/// Creates a composite key by concatenating user key with document ID.
/// This allows duplicate user keys while maintaining BTree uniqueness.
/// Format: [UserKeyBytes] + [DocumentIdKey]
/// </summary>
private IndexKey CreateCompositeKey(IndexKey userKey, IndexKey documentIdKey)
{
// Allocate buffer: user key + document ID key length
var compositeBytes = new byte[userKey.Data.Length + documentIdKey.Data.Length];
// Copy user key
userKey.Data.CopyTo(compositeBytes.AsSpan(0, userKey.Data.Length));
// Append document ID key
documentIdKey.Data.CopyTo(compositeBytes.AsSpan(userKey.Data.Length));
return new IndexKey(compositeBytes);
}
/// <summary>
/// Creates a composite key for range query boundary.
/// Uses MIN or MAX ID representation to capture all documents with the user key.
/// </summary>
private IndexKey CreateCompositeKeyBoundary(IndexKey userKey, bool useMinObjectId)
{
// For range boundaries, we use an empty key for Min and a very large key for Max
// to wrap around all possible IDs for this user key.
IndexKey idBoundary = useMinObjectId
? new IndexKey(Array.Empty<byte>())
: new IndexKey(Enumerable.Repeat((byte)0xFF, 16).ToArray()); // Using 16 as a safe max for GUID/ObjectId
return CreateCompositeKey(userKey, idBoundary);
}
/// <summary>
/// Extracts the original user key from a composite key by removing the ObjectId suffix.
/// Used when we need to return the original indexed value.
/// </summary>
private IndexKey ExtractUserKey(IndexKey compositeKey)
{
// Composite key = UserKey + ObjectId(12 bytes)
var userKeyLength = compositeKey.Data.Length - 12;
if (userKeyLength <= 0)
return compositeKey; // Fallback for malformed keys
var userKeyBytes = compositeKey.Data.Slice(0, userKeyLength);
return new IndexKey(userKeyBytes);
}
#endregion
/// <summary>
/// Converts a CLR value to an IndexKey for BTree storage.
/// Supports all common .NET types.
/// Converts a CLR value to an IndexKey for BTree storage.
/// Supports all common .NET types.
/// </summary>
private IndexKey ConvertToIndexKey(object value)
{
@@ -426,26 +375,64 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
long longVal => new IndexKey(longVal),
DateTime dateTime => new IndexKey(dateTime.Ticks),
bool boolVal => new IndexKey(boolVal ? 1 : 0),
byte[] byteArray => new IndexKey(byteArray),
// For compound keys or complex types, use ToString and serialize
// TODO: Better compound key serialization
byte[] byteArray => new IndexKey(byteArray),
// For compound keys or complex types, use ToString and serialize
// TODO: Better compound key serialization
_ => new IndexKey(value.ToString() ?? string.Empty)
};
}
/// <summary>
/// Releases resources used by this index wrapper.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// BTreeIndex doesn't currently implement IDisposable
// Future: may need to flush buffers, close resources
_disposed = true;
GC.SuppressFinalize(this);
};
}
}
#region Composite Key Support (SQLite-style for Duplicate Keys)
/// <summary>
/// Creates a composite key by concatenating user key with document ID.
/// This allows duplicate user keys while maintaining BTree uniqueness.
/// Format: [UserKeyBytes] + [DocumentIdKey]
/// </summary>
private IndexKey CreateCompositeKey(IndexKey userKey, IndexKey documentIdKey)
{
// Allocate buffer: user key + document ID key length
var compositeBytes = new byte[userKey.Data.Length + documentIdKey.Data.Length];
// Copy user key
userKey.Data.CopyTo(compositeBytes.AsSpan(0, userKey.Data.Length));
// Append document ID key
documentIdKey.Data.CopyTo(compositeBytes.AsSpan(userKey.Data.Length));
return new IndexKey(compositeBytes);
}
/// <summary>
/// Creates a composite key for range query boundary.
/// Uses MIN or MAX ID representation to capture all documents with the user key.
/// </summary>
private IndexKey CreateCompositeKeyBoundary(IndexKey userKey, bool useMinObjectId)
{
// For range boundaries, we use an empty key for Min and a very large key for Max
// to wrap around all possible IDs for this user key.
var idBoundary = useMinObjectId
? new IndexKey(Array.Empty<byte>())
: new IndexKey(Enumerable.Repeat((byte)0xFF, 16).ToArray()); // Using 16 as a safe max for GUID/ObjectId
return CreateCompositeKey(userKey, idBoundary);
}
/// <summary>
/// Extracts the original user key from a composite key by removing the ObjectId suffix.
/// Used when we need to return the original indexed value.
/// </summary>
private IndexKey ExtractUserKey(IndexKey compositeKey)
{
// Composite key = UserKey + ObjectId(12 bytes)
int userKeyLength = compositeKey.Data.Length - 12;
if (userKeyLength <= 0)
return compositeKey; // Fallback for malformed keys
var userKeyBytes = compositeKey.Data.Slice(0, userKeyLength);
return new IndexKey(userKeyBytes);
}
#endregion
}

18
src/CBDD.Core/Indexing/GeoSpatialExtensions.cs
View File

@@ -3,28 +3,30 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
public static class GeoSpatialExtensions
{
/// <summary>
/// Performs a geospatial proximity search (Near) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// Performs a geospatial proximity search (Near) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// </summary>
/// <param name="point">The coordinate tuple (Latitude, Longitude) property of the entity.</param>
/// <param name="center">The center point (Latitude, Longitude) for the proximity search.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>True if the point is within the specified radius.</returns>
public static bool Near(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) center, double radiusKm)
public static bool Near(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) center,
double radiusKm)
{
return true;
return true;
}
/// <summary>
/// Performs a geospatial bounding box search (Within) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// Performs a geospatial bounding box search (Within) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// </summary>
/// <param name="point">The coordinate tuple (Latitude, Longitude) property of the entity.</param>
/// <param name="min">The minimum (Latitude, Longitude) of the bounding box.</param>
/// <param name="max">The maximum (Latitude, Longitude) of the bounding box.</param>
/// <returns>True if the point is within the specified bounding box.</returns>
public static bool Within(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) min, (double Latitude, double Longitude) max)
public static bool Within(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) min,
(double Latitude, double Longitude) max)
{
return true;
}
}
}

41
src/CBDD.Core/Indexing/HashIndex.cs
View File

@@ -1,13 +1,10 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Hash-based index for exact-match lookups.
/// Uses simple bucket-based hashing with collision handling.
/// Hash-based index for exact-match lookups.
/// Uses simple bucket-based hashing with collision handling.
/// </summary>
public sealed class HashIndex
{
@@ -15,7 +12,7 @@ public sealed class HashIndex
private readonly IndexOptions _options;
/// <summary>
/// Initializes a new instance of the <see cref="HashIndex"/> class.
/// Initializes a new instance of the <see cref="HashIndex" /> class.
/// </summary>
/// <param name="options">The index options.</param>
public HashIndex(IndexOptions options)
@@ -25,16 +22,16 @@ public sealed class HashIndex
}
/// <summary>
/// Inserts a key-location pair into the hash index
/// Inserts a key-location pair into the hash index
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
public void Insert(IndexKey key, DocumentLocation location)
{
if (_options.Unique && TryFind(key, out _))
throw new InvalidOperationException($"Duplicate key violation for unique index");
throw new InvalidOperationException("Duplicate key violation for unique index");
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
{
@@ -46,46 +43,43 @@ public sealed class HashIndex
}
/// <summary>
/// Finds a document location by exact key match
/// Finds a document location by exact key match
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">When this method returns, contains the matched document location if found.</param>
/// <returns><see langword="true"/> if a matching entry is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if a matching entry is found; otherwise, <see langword="false" />.</returns>
public bool TryFind(IndexKey key, out DocumentLocation location)
{
location = default;
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
return false;
foreach (var entry in bucket)
{
if (entry.Key == key)
{
location = entry.Location;
return true;
}
}
return false;
}
/// <summary>
/// Removes an entry from the index
/// Removes an entry from the index
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
/// <returns><see langword="true"/> if an entry is removed; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if an entry is removed; otherwise, <see langword="false" />.</returns>
public bool Remove(IndexKey key, DocumentLocation location)
{
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
return false;
for (int i = 0; i < bucket.Count; i++)
{
for (var i = 0; i < bucket.Count; i++)
if (bucket[i].Key == key &&
bucket[i].Location.PageId == location.PageId &&
bucket[i].Location.SlotIndex == location.SlotIndex)
@@ -97,27 +91,24 @@ public sealed class HashIndex
return true;
}
}
return false;
}
/// <summary>
/// Gets all entries matching the key
/// Gets all entries matching the key
/// </summary>
/// <param name="key">The index key.</param>
/// <returns>All matching index entries.</returns>
public IEnumerable<IndexEntry> FindAll(IndexKey key)
{
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
yield break;
foreach (var entry in bucket)
{
if (entry.Key == key)
yield return entry;
}
}
}
}

27
src/CBDD.Core/Indexing/IBTreeCursor.cs
View File

@@ -1,50 +1,47 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents a cursor for traversing a B+Tree index.
/// Provides low-level primitives for building complex queries.
/// Represents a cursor for traversing a B+Tree index.
/// Provides low-level primitives for building complex queries.
/// </summary>
public interface IBTreeCursor : IDisposable
{
/// <summary>
/// Gets the current entry at the cursor position.
/// Throws InvalidOperationException if cursor is invalid or uninitialized.
/// Gets the current entry at the cursor position.
/// Throws InvalidOperationException if cursor is invalid or uninitialized.
/// </summary>
IndexEntry Current { get; }
/// <summary>
/// Moves the cursor to the first entry in the index.
/// Moves the cursor to the first entry in the index.
/// </summary>
/// <returns>True if the index is not empty; otherwise, false.</returns>
bool MoveToFirst();
/// <summary>
/// Moves the cursor to the last entry in the index.
/// Moves the cursor to the last entry in the index.
/// </summary>
/// <returns>True if the index is not empty; otherwise, false.</returns>
bool MoveToLast();
/// <summary>
/// Seeks to the specified key.
/// If exact match found, positions there and returns true.
/// If not found, positions at the next greater key and returns false.
/// Seeks to the specified key.
/// If exact match found, positions there and returns true.
/// If not found, positions at the next greater key and returns false.
/// </summary>
/// <param name="key">Key to seek</param>
/// <returns>True if exact match found; false if positioned at next greater key.</returns>
bool Seek(IndexKey key);
/// <summary>
/// Advances the cursor to the next entry.
/// Advances the cursor to the next entry.
/// </summary>
/// <returns>True if successfully moved; false if end of index reached.</returns>
bool MoveNext();
/// <summary>
/// Moves the cursor to the previous entry.
/// Moves the cursor to the previous entry.
/// </summary>
/// <returns>True if successfully moved; false if start of index reached.</returns>
bool MovePrev();
}
}

2
src/CBDD.Core/Indexing/IndexDirection.cs
View File

@@ -4,4 +4,4 @@ public enum IndexDirection
{
Forward,
Backward
}
}

203
src/CBDD.Core/Indexing/IndexKey.cs
View File

@@ -1,31 +1,30 @@
using ZB.MOM.WW.CBDD.Bson;
using System;
using System.Linq;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents a key in an index.
/// Implemented as struct for efficient index operations.
/// Note: Contains byte array so cannot be readonly struct.
/// </summary>
public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
{
private readonly byte[] _data;
private readonly int _hashCode;
/// <summary>
/// Gets the minimum possible index key.
/// </summary>
public static IndexKey MinKey => new IndexKey(Array.Empty<byte>());
using System.Text;
using ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents a key in an index.
/// Implemented as struct for efficient index operations.
/// Note: Contains byte array so cannot be readonly struct.
/// </summary>
public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
{
private readonly byte[] _data;
private readonly int _hashCode;
/// <summary>
/// Gets the maximum possible index key.
/// Gets the minimum possible index key.
/// </summary>
public static IndexKey MaxKey => new IndexKey(Enumerable.Repeat((byte)0xFF, 32).ToArray());
public static IndexKey MinKey => new(Array.Empty<byte>());
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from raw key bytes.
/// Gets the maximum possible index key.
/// </summary>
public static IndexKey MaxKey => new(Enumerable.Repeat((byte)0xFF, 32).ToArray());
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey" /> struct from raw key bytes.
/// </summary>
/// <param name="data">The key bytes.</param>
public IndexKey(ReadOnlySpan<byte> data)
@@ -35,7 +34,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from an object identifier.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from an object identifier.
/// </summary>
/// <param name="objectId">The object identifier value.</param>
public IndexKey(ObjectId objectId)
@@ -46,7 +45,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a 32-bit integer.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a 32-bit integer.
/// </summary>
/// <param name="value">The integer value.</param>
public IndexKey(int value)
@@ -56,7 +55,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a 64-bit integer.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a 64-bit integer.
/// </summary>
/// <param name="value">The integer value.</param>
public IndexKey(long value)
@@ -66,17 +65,17 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a string.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a string.
/// </summary>
/// <param name="value">The string value.</param>
public IndexKey(string value)
{
_data = System.Text.Encoding.UTF8.GetBytes(value);
_data = Encoding.UTF8.GetBytes(value);
_hashCode = ComputeHashCode(_data);
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a GUID.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a GUID.
/// </summary>
/// <param name="value">The GUID value.</param>
public IndexKey(Guid value)
@@ -86,72 +85,102 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Gets the raw byte data for this key.
/// Gets the raw byte data for this key.
/// </summary>
public readonly ReadOnlySpan<byte> Data => _data;
/// <summary>
/// Compares this key to another key.
/// Compares this key to another key.
/// </summary>
/// <param name="other">The key to compare with.</param>
/// <returns>
/// A value less than zero if this key is less than <paramref name="other"/>, zero if equal, or greater than zero if greater.
/// A value less than zero if this key is less than <paramref name="other" />, zero if equal, or greater than zero if
/// greater.
/// </returns>
public readonly int CompareTo(IndexKey other)
{
if (_data == null) return other._data == null ? 0 : -1;
if (other._data == null) return 1;
var minLength = Math.Min(_data.Length, other._data.Length);
for (int i = 0; i < minLength; i++)
{
var cmp = _data[i].CompareTo(other._data[i]);
if (cmp != 0)
return cmp;
}
int minLength = Math.Min(_data.Length, other._data.Length);
for (var i = 0; i < minLength; i++)
{
int cmp = _data[i].CompareTo(other._data[i]);
if (cmp != 0)
return cmp;
}
return _data.Length.CompareTo(other._data.Length);
}
/// <summary>
/// Determines whether this key equals another key.
/// Determines whether this key equals another key.
/// </summary>
/// <param name="other">The key to compare with.</param>
/// <returns><see langword="true"/> if the keys are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the keys are equal; otherwise, <see langword="false" />.</returns>
public readonly bool Equals(IndexKey other)
{
if (_hashCode != other._hashCode)
return false;
if (_data == null) return other._data == null;
if (other._data == null) return false;
if (_data == null) return other._data == null;
if (other._data == null) return false;
return _data.AsSpan().SequenceEqual(other._data);
}
/// <inheritdoc />
public override readonly bool Equals(object? obj) => obj is IndexKey other && Equals(other);
public readonly override bool Equals(object? obj)
{
return obj is IndexKey other && Equals(other);
}
/// <inheritdoc />
public override readonly int GetHashCode() => _hashCode;
public static bool operator ==(IndexKey left, IndexKey right) => left.Equals(right);
public static bool operator !=(IndexKey left, IndexKey right) => !left.Equals(right);
public static bool operator <(IndexKey left, IndexKey right) => left.CompareTo(right) < 0;
public static bool operator >(IndexKey left, IndexKey right) => left.CompareTo(right) > 0;
public static bool operator <=(IndexKey left, IndexKey right) => left.CompareTo(right) <= 0;
public static bool operator >=(IndexKey left, IndexKey right) => left.CompareTo(right) >= 0;
private static int ComputeHashCode(ReadOnlySpan<byte> data)
{
var hash = new HashCode();
hash.AddBytes(data);
return hash.ToHashCode();
}
public readonly override int GetHashCode()
{
return _hashCode;
}
public static bool operator ==(IndexKey left, IndexKey right)
{
return left.Equals(right);
}
public static bool operator !=(IndexKey left, IndexKey right)
{
return !left.Equals(right);
}
public static bool operator <(IndexKey left, IndexKey right)
{
return left.CompareTo(right) < 0;
}
public static bool operator >(IndexKey left, IndexKey right)
{
return left.CompareTo(right) > 0;
}
public static bool operator <=(IndexKey left, IndexKey right)
{
return left.CompareTo(right) <= 0;
}
public static bool operator >=(IndexKey left, IndexKey right)
{
return left.CompareTo(right) >= 0;
}
private static int ComputeHashCode(ReadOnlySpan<byte> data)
{
var hash = new HashCode();
hash.AddBytes(data);
return hash.ToHashCode();
}
/// <summary>
/// Creates an <see cref="IndexKey"/> from a supported CLR value.
/// Creates an <see cref="IndexKey" /> from a supported CLR value.
/// </summary>
/// <typeparam name="T">The CLR type of the value.</typeparam>
/// <param name="value">The value to convert.</param>
@@ -159,33 +188,35 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
public static IndexKey Create<T>(T value)
{
if (value == null) return default;
if (typeof(T) == typeof(ObjectId)) return new IndexKey((ObjectId)(object)value);
if (typeof(T) == typeof(int)) return new IndexKey((int)(object)value);
if (typeof(T) == typeof(long)) return new IndexKey((long)(object)value);
if (typeof(T) == typeof(string)) return new IndexKey((string)(object)value);
if (typeof(T) == typeof(Guid)) return new IndexKey((Guid)(object)value);
if (typeof(T) == typeof(byte[])) return new IndexKey((byte[])(object)value);
throw new NotSupportedException($"Type {typeof(T).Name} is not supported as an IndexKey. Provide a custom mapping.");
if (typeof(T) == typeof(ObjectId)) return new IndexKey((ObjectId)(object)value);
if (typeof(T) == typeof(int)) return new IndexKey((int)(object)value);
if (typeof(T) == typeof(long)) return new IndexKey((long)(object)value);
if (typeof(T) == typeof(string)) return new IndexKey((string)(object)value);
if (typeof(T) == typeof(Guid)) return new IndexKey((Guid)(object)value);
if (typeof(T) == typeof(byte[])) return new IndexKey((byte[])(object)value);
throw new NotSupportedException(
$"Type {typeof(T).Name} is not supported as an IndexKey. Provide a custom mapping.");
}
/// <summary>
/// Converts this key to a CLR value of type <typeparamref name="T"/>.
/// Converts this key to a CLR value of type <typeparamref name="T" />.
/// </summary>
/// <typeparam name="T">The CLR type to read from this key.</typeparam>
/// <returns>The converted value.</returns>
public readonly T As<T>()
{
if (_data == null) return default!;
if (typeof(T) == typeof(ObjectId)) return (T)(object)new ObjectId(_data);
if (typeof(T) == typeof(int)) return (T)(object)BitConverter.ToInt32(_data);
if (typeof(T) == typeof(long)) return (T)(object)BitConverter.ToInt64(_data);
if (typeof(T) == typeof(string)) return (T)(object)System.Text.Encoding.UTF8.GetString(_data);
if (typeof(T) == typeof(Guid)) return (T)(object)new Guid(_data);
if (typeof(T) == typeof(byte[])) return (T)(object)_data;
throw new NotSupportedException($"Type {typeof(T).Name} cannot be extracted from IndexKey. Provide a custom mapping.");
}
}
if (typeof(T) == typeof(ObjectId)) return (T)(object)new ObjectId(_data);
if (typeof(T) == typeof(int)) return (T)(object)BitConverter.ToInt32(_data);
if (typeof(T) == typeof(long)) return (T)(object)BitConverter.ToInt64(_data);
if (typeof(T) == typeof(string)) return (T)(object)Encoding.UTF8.GetString(_data);
if (typeof(T) == typeof(Guid)) return (T)(object)new Guid(_data);
if (typeof(T) == typeof(byte[])) return (T)(object)_data;
throw new NotSupportedException(
$"Type {typeof(T).Name} cannot be extracted from IndexKey. Provide a custom mapping.");
}
}

176
src/CBDD.Core/Indexing/IndexOptions.cs
View File

@@ -1,121 +1,130 @@
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Types of indices supported
/// </summary>
public enum IndexType : byte
{
/// <summary>B+Tree index for range queries and ordering</summary>
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Types of indices supported
/// </summary>
public enum IndexType : byte
{
/// <summary>B+Tree index for range queries and ordering</summary>
BTree = 1,
/// <summary>Hash index for exact match lookups</summary>
/// <summary>Hash index for exact match lookups</summary>
Hash = 2,
/// <summary>Unique index constraint</summary>
Unique = 3,
/// <summary>Vector index (HNSW) for similarity search</summary>
Vector = 4,
/// <summary>Geospatial index (R-Tree) for spatial queries</summary>
Spatial = 5
}
/// <summary>
/// Distance metrics for vector search
/// </summary>
public enum VectorMetric : byte
{
/// <summary>Cosine Similarity (Standard for embeddings)</summary>
Cosine = 1,
/// <summary>Euclidean Distance (L2)</summary>
L2 = 2,
/// <summary>Dot Product</summary>
DotProduct = 3
}
/// <summary>
/// Index options and configuration.
/// Implemented as readonly struct for efficiency.
/// </summary>
/// <summary>Unique index constraint</summary>
Unique = 3,
/// <summary>Vector index (HNSW) for similarity search</summary>
Vector = 4,
/// <summary>Geospatial index (R-Tree) for spatial queries</summary>
Spatial = 5
}
/// <summary>
/// Distance metrics for vector search
/// </summary>
public enum VectorMetric : byte
{
/// <summary>Cosine Similarity (Standard for embeddings)</summary>
Cosine = 1,
/// <summary>Euclidean Distance (L2)</summary>
L2 = 2,
/// <summary>Dot Product</summary>
DotProduct = 3
}
/// <summary>
/// Index options and configuration.
/// Implemented as readonly struct for efficiency.
/// </summary>
public readonly struct IndexOptions
{
/// <summary>
/// Gets the configured index type.
/// Gets the configured index type.
/// </summary>
public IndexType Type { get; init; }
/// <summary>
/// Gets a value indicating whether the index enforces uniqueness.
/// Gets a value indicating whether the index enforces uniqueness.
/// </summary>
public bool Unique { get; init; }
/// <summary>
/// Gets the indexed field names.
/// Gets the indexed field names.
/// </summary>
public string[] Fields { get; init; }
// Vector search options
/// <summary>
/// Gets the vector dimensionality for vector indexes.
/// Gets the vector dimensionality for vector indexes.
/// </summary>
public int Dimensions { get; init; }
/// <summary>
/// Gets the distance metric used for vector similarity.
/// Gets the distance metric used for vector similarity.
/// </summary>
public VectorMetric Metric { get; init; }
/// <summary>
/// Gets the minimum number of graph connections per node.
/// Gets the minimum number of graph connections per node.
/// </summary>
public int M { get; init; } // Min number of connections per node
/// <summary>
/// Gets the size of the dynamic candidate list during index construction.
/// Gets the size of the dynamic candidate list during index construction.
/// </summary>
public int EfConstruction { get; init; } // Size of dynamic candidate list for construction
/// <summary>
/// Creates non-unique B+Tree index options.
/// Creates non-unique B+Tree index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateBTree(params string[] fields) => new()
public static IndexOptions CreateBTree(params string[] fields)
{
Type = IndexType.BTree,
Unique = false,
Fields = fields
};
return new IndexOptions
{
Type = IndexType.BTree,
Unique = false,
Fields = fields
};
}
/// <summary>
/// Creates unique B+Tree index options.
/// Creates unique B+Tree index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateUnique(params string[] fields) => new()
public static IndexOptions CreateUnique(params string[] fields)
{
Type = IndexType.BTree,
Unique = true,
Fields = fields
};
return new IndexOptions
{
Type = IndexType.BTree,
Unique = true,
Fields = fields
};
}
/// <summary>
/// Creates hash index options.
/// Creates hash index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateHash(params string[] fields) => new()
public static IndexOptions CreateHash(params string[] fields)
{
Type = IndexType.Hash,
Unique = false,
Fields = fields
};
return new IndexOptions
{
Type = IndexType.Hash,
Unique = false,
Fields = fields
};
}
/// <summary>
/// Creates vector index options.
/// Creates vector index options.
/// </summary>
/// <param name="dimensions">The vector dimensionality.</param>
/// <param name="metric">The similarity metric.</param>
@@ -123,26 +132,33 @@ public readonly struct IndexOptions
/// <param name="ef">The candidate list size used during index construction.</param>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateVector(int dimensions, VectorMetric metric = VectorMetric.Cosine, int m = 16, int ef = 200, params string[] fields) => new()
public static IndexOptions CreateVector(int dimensions, VectorMetric metric = VectorMetric.Cosine, int m = 16,
int ef = 200, params string[] fields)
{
Type = IndexType.Vector,
Unique = false,
Fields = fields,
Dimensions = dimensions,
Metric = metric,
M = m,
EfConstruction = ef
};
return new IndexOptions
{
Type = IndexType.Vector,
Unique = false,
Fields = fields,
Dimensions = dimensions,
Metric = metric,
M = m,
EfConstruction = ef
};
}
/// <summary>
/// Creates spatial index options.
/// Creates spatial index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateSpatial(params string[] fields) => new()
public static IndexOptions CreateSpatial(params string[] fields)
{
Type = IndexType.Spatial,
Unique = false,
Fields = fields
};
}
return new IndexOptions
{
Type = IndexType.Spatial,
Unique = false,
Fields = fields
};
}
}

110
src/CBDD.Core/Indexing/Internal/GeoTypes.cs
View File

@@ -1,90 +1,90 @@
namespace ZB.MOM.WW.CBDD.Core.Indexing.Internal;
/// <summary>
/// Basic spatial point (Latitude/Longitude)
/// Internal primitive for R-Tree logic.
/// </summary>
namespace ZB.MOM.WW.CBDD.Core.Indexing.Internal;
/// <summary>
/// Basic spatial point (Latitude/Longitude)
/// Internal primitive for R-Tree logic.
/// </summary>
internal record struct GeoPoint(double Latitude, double Longitude)
{
/// <summary>
/// Gets an empty point at coordinate origin.
/// Gets an empty point at coordinate origin.
/// </summary>
public static GeoPoint Empty => new(0, 0);
}
/// <summary>
/// Minimum Bounding Box (MBR) for spatial indexing
/// Internal primitive for R-Tree logic.
/// </summary>
/// <summary>
/// Minimum Bounding Box (MBR) for spatial indexing
/// Internal primitive for R-Tree logic.
/// </summary>
internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, double MaxLon)
{
/// <summary>
/// Gets an empty bounding box sentinel value.
/// Gets an empty bounding box sentinel value.
/// </summary>
public static GeoBox Empty => new(double.MaxValue, double.MaxValue, double.MinValue, double.MinValue);
/// <summary>
/// Determines whether this box contains the specified point.
/// Gets the area of this bounding box.
/// </summary>
public double Area => Math.Max(0, MaxLat - MinLat) * Math.Max(0, MaxLon - MinLon);
/// <summary>
/// Determines whether this box contains the specified point.
/// </summary>
/// <param name="point">The point to test.</param>
/// <returns><see langword="true"/> if the point is inside this box; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the point is inside this box; otherwise, <see langword="false" />.</returns>
public bool Contains(GeoPoint point)
{
return point.Latitude >= MinLat && point.Latitude <= MaxLat &&
point.Longitude >= MinLon && point.Longitude <= MaxLon;
}
{
return point.Latitude >= MinLat && point.Latitude <= MaxLat &&
point.Longitude >= MinLon && point.Longitude <= MaxLon;
}
/// <summary>
/// Determines whether this box intersects another box.
/// Determines whether this box intersects another box.
/// </summary>
/// <param name="other">The other box to test.</param>
/// <returns><see langword="true"/> if the boxes intersect; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the boxes intersect; otherwise, <see langword="false" />.</returns>
public bool Intersects(GeoBox other)
{
return !(other.MinLat > MaxLat || other.MaxLat < MinLat ||
other.MinLon > MaxLon || other.MaxLon < MinLon);
}
{
return !(other.MinLat > MaxLat || other.MaxLat < MinLat ||
other.MinLon > MaxLon || other.MaxLon < MinLon);
}
/// <summary>
/// Creates a box that contains a single point.
/// Creates a box that contains a single point.
/// </summary>
/// <param name="point">The point to convert.</param>
/// <returns>A bounding box containing the specified point.</returns>
public static GeoBox FromPoint(GeoPoint point)
{
return new GeoBox(point.Latitude, point.Longitude, point.Latitude, point.Longitude);
}
{
return new GeoBox(point.Latitude, point.Longitude, point.Latitude, point.Longitude);
}
/// <summary>
/// Expands this box to include the specified point.
/// Expands this box to include the specified point.
/// </summary>
/// <param name="point">The point to include.</param>
/// <returns>A new expanded bounding box.</returns>
public GeoBox ExpandTo(GeoPoint point)
{
return new GeoBox(
Math.Min(MinLat, point.Latitude),
Math.Min(MinLon, point.Longitude),
Math.Max(MaxLat, point.Latitude),
Math.Max(MaxLon, point.Longitude));
}
{
return new GeoBox(
Math.Min(MinLat, point.Latitude),
Math.Min(MinLon, point.Longitude),
Math.Max(MaxLat, point.Latitude),
Math.Max(MaxLon, point.Longitude));
}
/// <summary>
/// Expands this box to include the specified box.
/// Expands this box to include the specified box.
/// </summary>
/// <param name="other">The box to include.</param>
/// <returns>A new expanded bounding box.</returns>
public GeoBox ExpandTo(GeoBox other)
{
return new GeoBox(
Math.Min(MinLat, other.MinLat),
Math.Min(MinLon, other.MinLon),
Math.Max(MaxLat, other.MaxLat),
Math.Max(MaxLon, other.MaxLon));
}
/// <summary>
/// Gets the area of this bounding box.
/// </summary>
public double Area => Math.Max(0, MaxLat - MinLat) * Math.Max(0, MaxLon - MinLon);
}
{
return new GeoBox(
Math.Min(MinLat, other.MinLat),
Math.Min(MinLon, other.MinLon),
Math.Max(MaxLat, other.MaxLat),
Math.Max(MaxLon, other.MaxLon));
}
}

22
src/CBDD.Core/Indexing/InternalEntry.cs
View File

@@ -1,27 +1,25 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
public struct InternalEntry
{
/// <summary>
/// Gets or sets the separator key.
/// Gets or sets the separator key.
/// </summary>
public IndexKey Key { get; set; }
/// <summary>
/// Gets or sets the child page identifier.
/// Gets or sets the child page identifier.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Initializes a new instance of the <see cref="InternalEntry"/> struct.
/// Initializes a new instance of the <see cref="InternalEntry" /> struct.
/// </summary>
/// <param name="key">The separator key.</param>
/// <param name="pageId">The child page identifier.</param>
public InternalEntry(IndexKey key, uint pageId)
{
Key = key;
PageId = pageId;
}
}
{
Key = key;
PageId = pageId;
}
}

245
src/CBDD.Core/Indexing/RTreeIndex.cs
View File

@@ -1,73 +1,78 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using System.Buffers;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// R-Tree Index implementation for Geospatial Indexing.
/// Uses Quadratic Split algorithm for simplicity and efficiency in paged storage.
/// R-Tree Index implementation for Geospatial Indexing.
/// Uses Quadratic Split algorithm for simplicity and efficiency in paged storage.
/// </summary>
internal class RTreeIndex : IDisposable
{
private readonly IIndexStorage _storage;
private readonly IndexOptions _options;
private uint _rootPageId;
internal class RTreeIndex : IDisposable
{
private readonly object _lock = new();
private readonly IndexOptions _options;
private readonly int _pageSize;
private readonly IIndexStorage _storage;
/// <summary>
/// Initializes a new instance of the <see cref="RTreeIndex"/> class.
/// </summary>
/// <param name="storage">The storage engine used for page operations.</param>
/// <param name="options">The index options.</param>
/// <param name="rootPageId">The root page identifier, or <c>0</c> to create a new root.</param>
public RTreeIndex(IIndexStorage storage, IndexOptions options, uint rootPageId)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
_rootPageId = rootPageId;
/// <summary>
/// Initializes a new instance of the <see cref="RTreeIndex" /> class.
/// </summary>
/// <param name="storage">The storage engine used for page operations.</param>
/// <param name="options">The index options.</param>
/// <param name="rootPageId">The root page identifier, or <c>0</c> to create a new root.</param>
public RTreeIndex(IIndexStorage storage, IndexOptions options, uint rootPageId)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
RootPageId = rootPageId;
_pageSize = _storage.PageSize;
if (_rootPageId == 0)
{
InitializeNewIndex();
}
}
/// <summary>
/// Gets the current root page identifier.
/// </summary>
public uint RootPageId => _rootPageId;
if (RootPageId == 0) InitializeNewIndex();
}
/// <summary>
/// Gets the current root page identifier.
/// </summary>
public uint RootPageId { get; private set; }
/// <summary>
/// Releases resources used by the index.
/// </summary>
public void Dispose()
{
}
private void InitializeNewIndex()
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_rootPageId = _storage.AllocatePage();
SpatialPage.Initialize(buffer, _rootPageId, true, 0);
_storage.WritePageImmediate(_rootPageId, buffer);
RootPageId = _storage.AllocatePage();
SpatialPage.Initialize(buffer, RootPageId, true, 0);
_storage.WritePageImmediate(RootPageId, buffer);
}
finally
{
ReturnPageBuffer(buffer);
}
finally { ReturnPageBuffer(buffer); }
}
/// <summary>
/// Searches for document locations whose minimum bounding rectangles intersect the specified area.
/// </summary>
/// <param name="area">The area to search.</param>
/// <param name="transaction">The optional transaction context.</param>
/// <returns>A sequence of matching document locations.</returns>
public IEnumerable<DocumentLocation> Search(GeoBox area, ITransaction? transaction = null)
{
if (_rootPageId == 0) yield break;
/// <summary>
/// Searches for document locations whose minimum bounding rectangles intersect the specified area.
/// </summary>
/// <param name="area">The area to search.</param>
/// <param name="transaction">The optional transaction context.</param>
/// <returns>A sequence of matching document locations.</returns>
public IEnumerable<DocumentLocation> Search(GeoBox area, ITransaction? transaction = null)
{
if (RootPageId == 0) yield break;
var stack = new Stack<uint>();
stack.Push(_rootPageId);
stack.Push(RootPageId);
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
while (stack.Count > 0)
@@ -78,38 +83,37 @@ internal class RTreeIndex : IDisposable
bool isLeaf = SpatialPage.GetIsLeaf(buffer);
ushort count = SpatialPage.GetEntryCount(buffer);
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var mbr, out var pointer);
if (area.Intersects(mbr))
{
if (isLeaf)
{
yield return pointer;
}
else
{
stack.Push(pointer.PageId);
}
}
}
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
/// <summary>
/// Inserts a bounding rectangle and document location into the index.
/// </summary>
/// <param name="mbr">The minimum bounding rectangle to index.</param>
/// <param name="loc">The document location associated with the rectangle.</param>
/// <param name="transaction">The optional transaction context.</param>
public void Insert(GeoBox mbr, DocumentLocation loc, ITransaction? transaction = null)
{
lock (_lock)
{
var leafPageId = ChooseLeaf(_rootPageId, mbr, transaction);
/// <summary>
/// Inserts a bounding rectangle and document location into the index.
/// </summary>
/// <param name="mbr">The minimum bounding rectangle to index.</param>
/// <param name="loc">The document location associated with the rectangle.</param>
/// <param name="transaction">The optional transaction context.</param>
public void Insert(GeoBox mbr, DocumentLocation loc, ITransaction? transaction = null)
{
lock (_lock)
{
uint leafPageId = ChooseLeaf(RootPageId, mbr, transaction);
InsertIntoNode(leafPageId, mbr, loc, transaction);
}
}
@@ -117,7 +121,7 @@ internal class RTreeIndex : IDisposable
private uint ChooseLeaf(uint rootId, GeoBox mbr, ITransaction? transaction)
{
uint currentId = rootId;
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
while (true)
@@ -127,13 +131,13 @@ internal class RTreeIndex : IDisposable
ushort count = SpatialPage.GetEntryCount(buffer);
uint bestChild = 0;
double minEnlargement = double.MaxValue;
double minArea = double.MaxValue;
var minEnlargement = double.MaxValue;
var minArea = double.MaxValue;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var childMbr, out var pointer);
SpatialPage.ReadEntry(buffer, i, out var childMbr, out var pointer);
var expanded = childMbr.ExpandTo(mbr);
double enlargement = expanded.Area - childMbr.Area;
@@ -156,12 +160,15 @@ internal class RTreeIndex : IDisposable
currentId = bestChild;
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void InsertIntoNode(uint pageId, GeoBox mbr, DocumentLocation pointer, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -171,8 +178,8 @@ internal class RTreeIndex : IDisposable
if (count < maxEntries)
{
SpatialPage.WriteEntry(buffer, count, mbr, pointer);
SpatialPage.SetEntryCount(buffer, (ushort)(count + 1));
SpatialPage.SetEntryCount(buffer, (ushort)(count + 1));
if (transaction != null)
_storage.WritePage(pageId, transaction.TransactionId, buffer);
else
@@ -186,17 +193,20 @@ internal class RTreeIndex : IDisposable
SplitNode(pageId, mbr, pointer, transaction);
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void UpdateMBRUpwards(uint pageId, ITransaction? transaction)
{
var buffer = RentPageBuffer();
var parentBuffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
byte[] parentBuffer = RentPageBuffer();
try
{
uint currentId = pageId;
while (currentId != _rootPageId)
while (currentId != RootPageId)
{
_storage.ReadPage(currentId, transaction?.TransactionId, buffer);
var currentMbr = SpatialPage.CalculateMBR(buffer);
@@ -206,9 +216,9 @@ internal class RTreeIndex : IDisposable
_storage.ReadPage(parentId, transaction?.TransactionId, parentBuffer);
ushort count = SpatialPage.GetEntryCount(parentBuffer);
bool changed = false;
var changed = false;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(parentBuffer, i, out var mbr, out var pointer);
if (pointer.PageId == currentId)
@@ -218,6 +228,7 @@ internal class RTreeIndex : IDisposable
SpatialPage.WriteEntry(parentBuffer, i, currentMbr, pointer);
changed = true;
}
break;
}
}
@@ -232,17 +243,17 @@ internal class RTreeIndex : IDisposable
currentId = parentId;
}
}
finally
{
ReturnPageBuffer(buffer);
finally
{
ReturnPageBuffer(buffer);
ReturnPageBuffer(parentBuffer);
}
}
private void SplitNode(uint pageId, GeoBox newMbr, DocumentLocation newPointer, ITransaction? transaction)
{
var buffer = RentPageBuffer();
var newBuffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
byte[] newBuffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -253,11 +264,12 @@ internal class RTreeIndex : IDisposable
// Collect all entries including the new one
var entries = new List<(GeoBox Mbr, DocumentLocation Pointer)>();
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var m, out var p);
entries.Add((m, p));
}
entries.Add((newMbr, newPointer));
// Pick Seeds
@@ -277,8 +289,8 @@ internal class RTreeIndex : IDisposable
SpatialPage.WriteEntry(newBuffer, 0, seed2.Mbr, seed2.Pointer);
SpatialPage.SetEntryCount(newBuffer, 1);
GeoBox mbr1 = seed1.Mbr;
GeoBox mbr2 = seed2.Mbr;
var mbr1 = seed1.Mbr;
var mbr2 = seed2.Mbr;
// Distribute remaining entries
while (entries.Count > 0)
@@ -320,23 +332,23 @@ internal class RTreeIndex : IDisposable
}
// Propagate split upwards
if (pageId == _rootPageId)
if (pageId == RootPageId)
{
// New Root
uint newRootId = _storage.AllocatePage();
SpatialPage.Initialize(buffer, newRootId, false, (byte)(level + 1));
SpatialPage.WriteEntry(buffer, 0, mbr1, new DocumentLocation(pageId, 0));
SpatialPage.WriteEntry(buffer, 1, mbr2, new DocumentLocation(newPageId, 0));
SpatialPage.SetEntryCount(buffer, 2);
SpatialPage.SetEntryCount(buffer, 2);
if (transaction != null)
_storage.WritePage(newRootId, transaction.TransactionId, buffer);
else
_storage.WritePageImmediate(newRootId, buffer);
_rootPageId = newRootId;
// Update parent pointers
RootPageId = newRootId;
// Update parent pointers
UpdateParentPointer(pageId, newRootId, transaction);
UpdateParentPointer(newPageId, newRootId, transaction);
}
@@ -347,16 +359,16 @@ internal class RTreeIndex : IDisposable
UpdateMBRUpwards(pageId, transaction);
}
}
finally
{
ReturnPageBuffer(buffer);
finally
{
ReturnPageBuffer(buffer);
ReturnPageBuffer(newBuffer);
}
}
private void UpdateParentPointer(uint pageId, uint parentId, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -366,27 +378,29 @@ internal class RTreeIndex : IDisposable
else
_storage.WritePageImmediate(pageId, buffer);
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void PickSeeds(List<(GeoBox Mbr, DocumentLocation Pointer)> entries, out (GeoBox Mbr, DocumentLocation Pointer) s1, out (GeoBox Mbr, DocumentLocation Pointer) s2)
private void PickSeeds(List<(GeoBox Mbr, DocumentLocation Pointer)> entries,
out (GeoBox Mbr, DocumentLocation Pointer) s1, out (GeoBox Mbr, DocumentLocation Pointer) s2)
{
double maxWaste = double.MinValue;
var maxWaste = double.MinValue;
s1 = entries[0];
s2 = entries[1];
for (int i = 0; i < entries.Count; i++)
for (var i = 0; i < entries.Count; i++)
for (int j = i + 1; j < entries.Count; j++)
{
for (int j = i + 1; j < entries.Count; j++)
var combined = entries[i].Mbr.ExpandTo(entries[j].Mbr);
double waste = combined.Area - entries[i].Mbr.Area - entries[j].Mbr.Area;
if (waste > maxWaste)
{
var combined = entries[i].Mbr.ExpandTo(entries[j].Mbr);
double waste = combined.Area - entries[i].Mbr.Area - entries[j].Mbr.Area;
if (waste > maxWaste)
{
maxWaste = waste;
s1 = entries[i];
s2 = entries[j];
}
maxWaste = waste;
s1 = entries[i];
s2 = entries[j];
}
}
}
@@ -400,11 +414,4 @@ internal class RTreeIndex : IDisposable
{
ArrayPool<byte>.Shared.Return(buffer);
}
/// <summary>
/// Releases resources used by the index.
/// </summary>
public void Dispose()
{
}
}
}

90
src/CBDD.Core/Indexing/SpatialMath.cs
View File

@@ -1,22 +1,25 @@
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
public static class SpatialMath
{
private const double EarthRadiusKm = 6371.0;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
public static class SpatialMath
{
private const double EarthRadiusKm = 6371.0;
/// <summary>
/// Calculates distance between two points on Earth using Haversine formula.
/// Result in kilometers.
/// Calculates distance between two points on Earth using Haversine formula.
/// Result in kilometers.
/// </summary>
/// <param name="p1">The first point.</param>
/// <param name="p2">The second point.</param>
/// <returns>The distance in kilometers.</returns>
internal static double DistanceKm(GeoPoint p1, GeoPoint p2) => DistanceKm(p1.Latitude, p1.Longitude, p2.Latitude, p2.Longitude);
internal static double DistanceKm(GeoPoint p1, GeoPoint p2)
{
return DistanceKm(p1.Latitude, p1.Longitude, p2.Latitude, p2.Longitude);
}
/// <summary>
/// Calculates distance between two coordinates on Earth using Haversine formula.
/// Calculates distance between two coordinates on Earth using Haversine formula.
/// </summary>
/// <param name="lat1">The latitude of the first point.</param>
/// <param name="lon1">The longitude of the first point.</param>
@@ -27,34 +30,40 @@ public static class SpatialMath
{
double dLat = ToRadians(lat2 - lat1);
double dLon = ToRadians(lon2 - lon1);
double a = Math.Sin(dLat / 2) * Math.Sin(dLat / 2) +
Math.Cos(ToRadians(lat1)) * Math.Cos(ToRadians(lat2)) *
Math.Sin(dLon / 2) * Math.Sin(dLon / 2);
double c = 2 * Math.Atan2(Math.Sqrt(a), Math.Sqrt(1 - a));
return EarthRadiusKm * c;
}
double a = Math.Sin(dLat / 2) * Math.Sin(dLat / 2) +
Math.Cos(ToRadians(lat1)) * Math.Cos(ToRadians(lat2)) *
Math.Sin(dLon / 2) * Math.Sin(dLon / 2);
double c = 2 * Math.Atan2(Math.Sqrt(a), Math.Sqrt(1 - a));
return EarthRadiusKm * c;
}
/// <summary>
/// Creates a Bounding Box (MBR) centered at a point with a given radius in km.
/// Creates a Bounding Box (MBR) centered at a point with a given radius in km.
/// </summary>
/// <param name="center">The center point.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>The bounding box.</returns>
internal static GeoBox BoundingBox(GeoPoint center, double radiusKm) => BoundingBox(center.Latitude, center.Longitude, radiusKm);
internal static GeoBox BoundingBox(GeoPoint center, double radiusKm)
{
return BoundingBox(center.Latitude, center.Longitude, radiusKm);
}
/// <summary>
/// Creates a bounding box from a coordinate and radius.
/// Creates a bounding box from a coordinate and radius.
/// </summary>
/// <param name="lat">The center latitude.</param>
/// <param name="lon">The center longitude.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>The bounding box.</returns>
internal static GeoBox InternalBoundingBox(double lat, double lon, double radiusKm) => BoundingBox(lat, lon, radiusKm);
internal static GeoBox InternalBoundingBox(double lat, double lon, double radiusKm)
{
return BoundingBox(lat, lon, radiusKm);
}
/// <summary>
/// Creates a bounding box centered at a coordinate with a given radius in kilometers.
/// Creates a bounding box centered at a coordinate with a given radius in kilometers.
/// </summary>
/// <param name="lat">The center latitude.</param>
/// <param name="lon">The center longitude.</param>
@@ -64,14 +73,21 @@ public static class SpatialMath
{
double dLat = ToDegrees(radiusKm / EarthRadiusKm);
double dLon = ToDegrees(radiusKm / (EarthRadiusKm * Math.Cos(ToRadians(lat))));
return new GeoBox(
lat - dLat,
lon - dLon,
lat + dLat,
lon + dLon);
}
private static double ToRadians(double degrees) => degrees * Math.PI / 180.0;
private static double ToDegrees(double radians) => radians * 180.0 / Math.PI;
}
return new GeoBox(
lat - dLat,
lon - dLon,
lat + dLat,
lon + dLon);
}
private static double ToRadians(double degrees)
{
return degrees * Math.PI / 180.0;
}
private static double ToDegrees(double radians)
{
return radians * 180.0 / Math.PI;
}
}

88
src/CBDD.Core/Indexing/VectorMath.cs
View File

@@ -1,23 +1,21 @@
using System.Runtime.Intrinsics;
using System.Runtime.Intrinsics.X86;
using System.Runtime.InteropServices;
using System.Numerics;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Optimized vector math utilities using SIMD if available.
/// Optimized vector math utilities using SIMD if available.
/// </summary>
public static class VectorMath
{
/// <summary>
/// Computes vector distance according to the selected metric.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <param name="metric">The metric used to compute distance.</param>
/// <returns>The distance value for the selected metric.</returns>
public static float Distance(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2, VectorMetric metric)
public static class VectorMath
{
/// <summary>
/// Computes vector distance according to the selected metric.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <param name="metric">The metric used to compute distance.</param>
/// <returns>The distance value for the selected metric.</returns>
public static float Distance(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2, VectorMetric metric)
{
return metric switch
{
@@ -28,13 +26,13 @@ public static class VectorMath
};
}
/// <summary>
/// Computes cosine similarity between two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <returns>The cosine similarity in the range [-1, 1].</returns>
public static float CosineSimilarity(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2)
/// <summary>
/// Computes cosine similarity between two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <returns>The cosine similarity in the range [-1, 1].</returns>
public static float CosineSimilarity(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2)
{
float dot = DotProduct(v1, v2);
float mag1 = DotProduct(v1, v1);
@@ -44,19 +42,19 @@ public static class VectorMath
return dot / (MathF.Sqrt(mag1) * MathF.Sqrt(mag2));
}
/// <summary>
/// Computes the dot product of two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <returns>The dot product value.</returns>
public static float DotProduct(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2)
/// <summary>
/// Computes the dot product of two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <returns>The dot product value.</returns>
public static float DotProduct(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2)
{
if (v1.Length != v2.Length)
throw new ArgumentException("Vectors must have same length");
float dot = 0;
int i = 0;
var i = 0;
// SIMD Optimization for .NET
if (Vector.IsHardwareAccelerated && v1.Length >= Vector<float>.Count)
@@ -65,37 +63,31 @@ public static class VectorMath
var v1Span = MemoryMarshal.Cast<float, Vector<float>>(v1);
var v2Span = MemoryMarshal.Cast<float, Vector<float>>(v2);
foreach (var chunk in Enumerable.Range(0, v1Span.Length))
{
vDot += v1Span[chunk] * v2Span[chunk];
}
foreach (int chunk in Enumerable.Range(0, v1Span.Length)) vDot += v1Span[chunk] * v2Span[chunk];
dot = Vector.Dot(vDot, Vector<float>.One);
i = v1Span.Length * Vector<float>.Count;
}
// Remaining elements
for (; i < v1.Length; i++)
{
dot += v1[i] * v2[i];
}
for (; i < v1.Length; i++) dot += v1[i] * v2[i];
return dot;
}
/// <summary>
/// Computes squared Euclidean distance between two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <returns>The squared Euclidean distance.</returns>
public static float EuclideanDistanceSquared(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2)
/// <summary>
/// Computes squared Euclidean distance between two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
/// <returns>The squared Euclidean distance.</returns>
public static float EuclideanDistanceSquared(ReadOnlySpan<float> v1, ReadOnlySpan<float> v2)
{
if (v1.Length != v2.Length)
throw new ArgumentException("Vectors must have same length");
float dist = 0;
int i = 0;
var i = 0;
if (Vector.IsHardwareAccelerated && v1.Length >= Vector<float>.Count)
{
@@ -103,7 +95,7 @@ public static class VectorMath
var v1Span = MemoryMarshal.Cast<float, Vector<float>>(v1);
var v2Span = MemoryMarshal.Cast<float, Vector<float>>(v2);
foreach (var chunk in Enumerable.Range(0, v1Span.Length))
foreach (int chunk in Enumerable.Range(0, v1Span.Length))
{
var diff = v1Span[chunk] - v2Span[chunk];
vDist += diff * diff;
@@ -121,4 +113,4 @@ public static class VectorMath
return dist;
}
}
}

38
src/CBDD.Core/Indexing/VectorSearchExtensions.cs
View File

@@ -3,28 +3,34 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
public static class VectorSearchExtensions
{
/// <summary>
/// Performs a similarity search on a vector property.
/// This method is a marker for the LINQ query provider and is optimized using HNSW indexes if available.
/// Performs a similarity search on a vector property.
/// This method is a marker for the LINQ query provider and is optimized using HNSW indexes if available.
/// </summary>
/// <param name="vector">The vector property of the entity.</param>
/// <param name="query">The query vector to compare against.</param>
/// <param name="k">Number of nearest neighbors to return.</param>
/// <returns>True if the document is part of the top-k results (always returns true when evaluated in memory for compilation purposes).</returns>
/// <returns>
/// True if the document is part of the top-k results (always returns true when evaluated in memory for
/// compilation purposes).
/// </returns>
public static bool VectorSearch(this float[] vector, float[] query, int k)
{
return true;
}
/// <summary>
/// Performs a similarity search on a collection of vector properties.
/// Used for entities with multiple vectors per document.
/// </summary>
/// <param name="vectors">The vector collection of the entity.</param>
/// <param name="query">The query vector to compare against.</param>
/// <param name="k">Number of nearest neighbors to return.</param>
/// <returns>True if the document is part of the top-k results (always returns true when evaluated in memory for compilation purposes).</returns>
public static bool VectorSearch(this IEnumerable<float[]> vectors, float[] query, int k)
{
return true;
}
}
/// <summary>
/// Performs a similarity search on a collection of vector properties.
/// Used for entities with multiple vectors per document.
/// </summary>
/// <param name="vectors">The vector collection of the entity.</param>
/// <param name="query">The query vector to compare against.</param>
/// <param name="k">Number of nearest neighbors to return.</param>
/// <returns>
/// True if the document is part of the top-k results (always returns true when evaluated in memory for
/// compilation purposes).
/// </returns>
public static bool VectorSearch(this IEnumerable<float[]> vectors, float[] query, int k)
{
return true;
}
}

305
src/CBDD.Core/Indexing/VectorSearchIndex.cs
View File

@@ -1,87 +1,85 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// HNSW (Hierarchical Navigable Small World) index implementation.
/// Handles multi-vector indexing and similarity searches.
/// HNSW (Hierarchical Navigable Small World) index implementation.
/// Handles multi-vector indexing and similarity searches.
/// </summary>
public sealed class VectorSearchIndex
{
private struct NodeReference
{
public uint PageId;
public int NodeIndex;
public int MaxLevel;
}
private readonly IIndexStorage _storage;
public sealed class VectorSearchIndex
{
private readonly IndexOptions _options;
private uint _rootPageId;
private readonly Random _random = new(42);
/// <summary>
/// Initializes a new vector search index.
/// </summary>
/// <param name="storage">The storage engine used by the index.</param>
/// <param name="options">Index configuration options.</param>
/// <param name="rootPageId">Optional existing root page identifier.</param>
public VectorSearchIndex(StorageEngine storage, IndexOptions options, uint rootPageId = 0)
: this((IStorageEngine)storage, options, rootPageId)
{
}
/// <summary>
/// Initializes a new vector search index.
/// </summary>
/// <param name="storage">The index storage abstraction used by the index.</param>
/// <param name="options">Index configuration options.</param>
/// <param name="rootPageId">Optional existing root page identifier.</param>
internal VectorSearchIndex(IIndexStorage storage, IndexOptions options, uint rootPageId = 0)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
_rootPageId = rootPageId;
}
private readonly IIndexStorage _storage;
/// <summary>
/// Gets the root page identifier of the index.
/// </summary>
public uint RootPageId => _rootPageId;
/// <summary>
/// Initializes a new vector search index.
/// </summary>
/// <param name="storage">The storage engine used by the index.</param>
/// <param name="options">Index configuration options.</param>
/// <param name="rootPageId">Optional existing root page identifier.</param>
public VectorSearchIndex(StorageEngine storage, IndexOptions options, uint rootPageId = 0)
: this((IStorageEngine)storage, options, rootPageId)
{
}
/// <summary>
/// Inserts a vector and its document location into the index.
/// </summary>
/// <param name="vector">The vector values to index.</param>
/// <param name="docLocation">The document location associated with the vector.</param>
/// <param name="transaction">Optional transaction context.</param>
public void Insert(float[] vector, DocumentLocation docLocation, ITransaction? transaction = null)
/// <summary>
/// Initializes a new vector search index.
/// </summary>
/// <param name="storage">The index storage abstraction used by the index.</param>
/// <param name="options">Index configuration options.</param>
/// <param name="rootPageId">Optional existing root page identifier.</param>
internal VectorSearchIndex(IIndexStorage storage, IndexOptions options, uint rootPageId = 0)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
RootPageId = rootPageId;
}
/// <summary>
/// Gets the root page identifier of the index.
/// </summary>
public uint RootPageId { get; private set; }
/// <summary>
/// Inserts a vector and its document location into the index.
/// </summary>
/// <param name="vector">The vector values to index.</param>
/// <param name="docLocation">The document location associated with the vector.</param>
/// <param name="transaction">Optional transaction context.</param>
public void Insert(float[] vector, DocumentLocation docLocation, ITransaction? transaction = null)
{
if (vector.Length != _options.Dimensions)
throw new ArgumentException($"Vector dimension mismatch. Expected {_options.Dimensions}, got {vector.Length}");
throw new ArgumentException(
$"Vector dimension mismatch. Expected {_options.Dimensions}, got {vector.Length}");
// 1. Determine level for new node
int targetLevel = GetRandomLevel();
// 2. If index is empty, create first page and first node
if (_rootPageId == 0)
if (RootPageId == 0)
{
_rootPageId = CreateNewPage(transaction);
var pageBuffer = RentPageBuffer();
RootPageId = CreateNewPage(transaction);
byte[] pageBuffer = RentPageBuffer();
try
{
_storage.ReadPage(_rootPageId, transaction?.TransactionId, pageBuffer);
_storage.ReadPage(RootPageId, transaction?.TransactionId, pageBuffer);
VectorPage.WriteNode(pageBuffer, 0, docLocation, targetLevel, vector, _options.Dimensions);
VectorPage.IncrementNodeCount(pageBuffer); // Helper needs to be added or handled
VectorPage.IncrementNodeCount(pageBuffer); // Helper needs to be added or handled
if (transaction != null)
_storage.WritePage(_rootPageId, transaction.TransactionId, pageBuffer);
_storage.WritePage(RootPageId, transaction.TransactionId, pageBuffer);
else
_storage.WritePageImmediate(_rootPageId, pageBuffer);
_storage.WritePageImmediate(RootPageId, pageBuffer);
}
finally { ReturnPageBuffer(pageBuffer); }
finally
{
ReturnPageBuffer(pageBuffer);
}
return;
}
@@ -92,9 +90,7 @@ public sealed class VectorSearchIndex
// 4. Greedy search down to targetLevel+1
for (int l = entryPoint.MaxLevel; l > targetLevel; l--)
{
currentPoint = GreedySearch(currentPoint, vector, l, transaction);
}
// 5. Create the new node
var newNode = AllocateNode(vector, docLocation, targetLevel, transaction);
@@ -103,25 +99,20 @@ public sealed class VectorSearchIndex
for (int l = Math.Min(targetLevel, entryPoint.MaxLevel); l >= 0; l--)
{
var neighbors = SearchLayer(currentPoint, vector, _options.EfConstruction, l, transaction);
var selectedNeighbors = SelectNeighbors(neighbors, vector, _options.M, l, transaction);
foreach (var neighbor in selectedNeighbors)
{
AddBidirectionalLink(newNode, neighbor, l, transaction);
}
// Move currentPoint down for next level if available
var selectedNeighbors = SelectNeighbors(neighbors, vector, _options.M, l, transaction);
foreach (var neighbor in selectedNeighbors) AddBidirectionalLink(newNode, neighbor, l, transaction);
// Move currentPoint down for next level if available
currentPoint = GreedySearch(currentPoint, vector, l, transaction);
}
// 7. Update entry point if new node is higher
if (targetLevel > entryPoint.MaxLevel)
{
UpdateEntryPoint(newNode, transaction);
}
if (targetLevel > entryPoint.MaxLevel) UpdateEntryPoint(newNode, transaction);
}
private IEnumerable<NodeReference> SelectNeighbors(IEnumerable<NodeReference> candidates, float[] query, int m, int level, ITransaction? transaction)
private IEnumerable<NodeReference> SelectNeighbors(IEnumerable<NodeReference> candidates, float[] query, int m,
int level, ITransaction? transaction)
{
// Simple heuristic: just take top M nearest.
// HNSW Paper suggests more complex heuristic to maintain connectivity diversity.
@@ -136,20 +127,20 @@ public sealed class VectorSearchIndex
private void Link(NodeReference from, NodeReference to, int level, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(from.PageId, transaction?.TransactionId, buffer);
var links = VectorPage.GetLinksSpan(buffer, from.NodeIndex, level, _options.Dimensions, _options.M);
// Find first empty slot (PageId == 0)
for (int i = 0; i < links.Length; i += 6)
var links = VectorPage.GetLinksSpan(buffer, from.NodeIndex, level, _options.Dimensions, _options.M);
// Find first empty slot (PageId == 0)
for (var i = 0; i < links.Length; i += 6)
{
var existing = DocumentLocation.ReadFrom(links.Slice(i, 6));
if (existing.PageId == 0)
{
new DocumentLocation(to.PageId, (ushort)to.NodeIndex).WriteTo(links.Slice(i, 6));
new DocumentLocation(to.PageId, (ushort)to.NodeIndex).WriteTo(links.Slice(i, 6));
if (transaction != null)
_storage.WritePage(from.PageId, transaction.TransactionId, buffer);
else
@@ -160,7 +151,10 @@ public sealed class VectorSearchIndex
// If full, we should technically prune or redistribute links as per HNSW paper.
// For now, we assume M is large enough or we skip (limited connectivity).
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private NodeReference AllocateNode(float[] vector, DocumentLocation docLoc, int level, ITransaction? transaction)
@@ -168,24 +162,27 @@ public sealed class VectorSearchIndex
// Find a page with space or create new
// For simplicity, we search for a page with available slots or append to a new one.
// Implementation omitted for brevity but required for full persistence.
uint pageId = _rootPageId; // Placeholder: need allocation strategy
int index = 0;
var buffer = RentPageBuffer();
uint pageId = RootPageId; // Placeholder: need allocation strategy
var index = 0;
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
index = VectorPage.GetNodeCount(buffer);
VectorPage.WriteNode(buffer, index, docLoc, level, vector, _options.Dimensions);
VectorPage.IncrementNodeCount(buffer);
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
index = VectorPage.GetNodeCount(buffer);
VectorPage.WriteNode(buffer, index, docLoc, level, vector, _options.Dimensions);
VectorPage.IncrementNodeCount(buffer);
if (transaction != null)
_storage.WritePage(pageId, transaction.TransactionId, buffer);
_storage.WritePage(pageId, transaction.TransactionId, buffer);
else
_storage.WritePageImmediate(pageId, buffer);
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
return new NodeReference { PageId = pageId, NodeIndex = index, MaxLevel = level };
}
@@ -197,7 +194,7 @@ public sealed class VectorSearchIndex
private NodeReference GreedySearch(NodeReference entryPoint, float[] query, int level, ITransaction? transaction)
{
bool changed = true;
var changed = true;
var current = entryPoint;
float currentDist = VectorMath.Distance(query, LoadVector(current, transaction), _options.Metric);
@@ -215,10 +212,12 @@ public sealed class VectorSearchIndex
}
}
}
return current;
}
private IEnumerable<NodeReference> SearchLayer(NodeReference entryPoint, float[] query, int ef, int level, ITransaction? transaction)
private IEnumerable<NodeReference> SearchLayer(NodeReference entryPoint, float[] query, int ef, int level,
ITransaction? transaction)
{
var visited = new HashSet<NodeReference>();
var candidates = new PriorityQueue<NodeReference, float>();
@@ -233,14 +232,13 @@ public sealed class VectorSearchIndex
{
float d_c = 0;
candidates.TryPeek(out var c, out d_c);
result.TryPeek(out var f, out var d_f);
result.TryPeek(out var f, out float d_f);
if (d_c > -d_f) break;
candidates.Dequeue();
foreach (var e in GetNeighbors(c, level, transaction))
{
if (!visited.Contains(e))
{
visited.Add(e);
@@ -254,7 +252,6 @@ public sealed class VectorSearchIndex
if (result.Count > ef) result.Dequeue();
}
}
}
}
// Convert result to list (ordered by distance)
@@ -268,52 +265,53 @@ public sealed class VectorSearchIndex
{
// For now, assume a fixed location or track it in page 0 of index
// TODO: Real implementation
return new NodeReference { PageId = _rootPageId, NodeIndex = 0, MaxLevel = 0 };
return new NodeReference { PageId = RootPageId, NodeIndex = 0, MaxLevel = 0 };
}
private float[] LoadVector(NodeReference node, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
float[] vector = new float[_options.Dimensions];
var vector = new float[_options.Dimensions];
VectorPage.ReadNodeData(buffer, node.NodeIndex, out _, out _, vector);
return vector;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
/// <summary>
/// Searches the index for the nearest vectors to the query.
/// </summary>
/// <param name="query">The query vector.</param>
/// <param name="k">The number of nearest results to return.</param>
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction context.</param>
/// <returns>The nearest vector search results.</returns>
public IEnumerable<VectorSearchResult> Search(float[] query, int k, int efSearch = 100, ITransaction? transaction = null)
/// <summary>
/// Searches the index for the nearest vectors to the query.
/// </summary>
/// <param name="query">The query vector.</param>
/// <param name="k">The number of nearest results to return.</param>
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction context.</param>
/// <returns>The nearest vector search results.</returns>
public IEnumerable<VectorSearchResult> Search(float[] query, int k, int efSearch = 100,
ITransaction? transaction = null)
{
if (_rootPageId == 0) yield break;
if (RootPageId == 0) yield break;
var entryPoint = GetEntryPoint();
var currentPoint = entryPoint;
// 1. Greedy search through higher layers to find entry point for level 0
for (int l = entryPoint.MaxLevel; l > 0; l--)
{
currentPoint = GreedySearch(currentPoint, query, l, transaction);
}
for (int l = entryPoint.MaxLevel; l > 0; l--) currentPoint = GreedySearch(currentPoint, query, l, transaction);
// 2. Comprehensive search on level 0
var nearest = SearchLayer(currentPoint, query, Math.Max(efSearch, k), 0, transaction);
// 3. Return top-k results
int count = 0;
var nearest = SearchLayer(currentPoint, query, Math.Max(efSearch, k), 0, transaction);
// 3. Return top-k results
var count = 0;
foreach (var node in nearest)
{
if (count++ >= k) break;
if (count++ >= k) break;
float dist = VectorMath.Distance(query, LoadVector(node, transaction), _options.Metric);
var loc = LoadDocumentLocation(node, transaction);
yield return new VectorSearchResult(loc, dist);
@@ -322,34 +320,41 @@ public sealed class VectorSearchIndex
private DocumentLocation LoadDocumentLocation(NodeReference node, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
VectorPage.ReadNodeData(buffer, node.NodeIndex, out var loc, out _, new float[0]); // Vector not needed here
return loc;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private IEnumerable<NodeReference> GetNeighbors(NodeReference node, int level, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
var results = new List<NodeReference>();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
var links = VectorPage.GetLinksSpan(buffer, node.NodeIndex, level, _options.Dimensions, _options.M);
for (int i = 0; i < links.Length; i += 6)
var links = VectorPage.GetLinksSpan(buffer, node.NodeIndex, level, _options.Dimensions, _options.M);
for (var i = 0; i < links.Length; i += 6)
{
var loc = DocumentLocation.ReadFrom(links.Slice(i, 6));
if (loc.PageId == 0) break; // End of links
if (loc.PageId == 0) break; // End of links
results.Add(new NodeReference { PageId = loc.PageId, NodeIndex = loc.SlotIndex });
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
return results;
}
@@ -357,29 +362,43 @@ public sealed class VectorSearchIndex
{
// Probability p = 1/M for each level
double p = 1.0 / _options.M;
int level = 0;
while (_random.NextDouble() < p && level < 15)
{
level++;
}
var level = 0;
while (_random.NextDouble() < p && level < 15) level++;
return level;
}
private uint CreateNewPage(ITransaction? transaction)
{
uint pageId = _storage.AllocatePage();
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
VectorPage.Initialize(buffer, pageId, _options.Dimensions, _options.M);
_storage.WritePageImmediate(pageId, buffer);
return pageId;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private byte[] RentPageBuffer() => System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
private void ReturnPageBuffer(byte[] buffer) => System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
private byte[] RentPageBuffer()
{
return ArrayPool<byte>.Shared.Rent(_storage.PageSize);
}
private void ReturnPageBuffer(byte[] buffer)
{
ArrayPool<byte>.Shared.Return(buffer);
}
private struct NodeReference
{
public uint PageId;
public int NodeIndex;
public int MaxLevel;
}
}
public record struct VectorSearchResult(DocumentLocation Location, float Distance);
public record struct VectorSearchResult(DocumentLocation Location, float Distance);

158
src/CBDD.Core/Metadata/EntityTypeBuilder.cs
View File

@@ -1,42 +1,42 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
using System.Linq.Expressions;
namespace ZB.MOM.WW.CBDD.Core.Metadata;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Metadata;
public class EntityTypeBuilder<T> where T : class
{
/// <summary>
/// Gets the configured collection name for the entity type.
/// Gets the configured collection name for the entity type.
/// </summary>
public string? CollectionName { get; private set; }
/// <summary>
/// Gets the configured indexes for the entity type.
/// Gets the configured indexes for the entity type.
/// </summary>
public List<IndexBuilder<T>> Indexes { get; } = new();
/// <summary>
/// Gets the primary key selector expression.
/// Gets the primary key selector expression.
/// </summary>
public LambdaExpression? PrimaryKeySelector { get; private set; }
/// <summary>
/// Gets a value indicating whether the primary key value is generated on add.
/// Gets a value indicating whether the primary key value is generated on add.
/// </summary>
public bool ValueGeneratedOnAdd { get; private set; }
/// <summary>
/// Gets the configured primary key property name.
/// Gets the configured primary key property name.
/// </summary>
public string? PrimaryKeyName { get; private set; }
/// <summary>
/// Gets the configured property converter types keyed by property name.
/// Gets the configured property converter types keyed by property name.
/// </summary>
public Dictionary<string, Type> PropertyConverters { get; } = new();
/// <summary>
/// Sets the collection name for the entity type.
/// Sets the collection name for the entity type.
/// </summary>
/// <param name="name">The collection name.</param>
/// <returns>The current entity type builder.</returns>
@@ -47,21 +47,22 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Adds an index for the specified key selector.
/// Adds an index for the specified key selector.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The key selector expression.</param>
/// <param name="name">The optional index name.</param>
/// <param name="unique">A value indicating whether the index is unique.</param>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasIndex<TKey>(Expression<Func<T, TKey>> keySelector, string? name = null, bool unique = false)
public EntityTypeBuilder<T> HasIndex<TKey>(Expression<Func<T, TKey>> keySelector, string? name = null,
bool unique = false)
{
Indexes.Add(new IndexBuilder<T>(keySelector, name, unique));
return this;
}
/// <summary>
/// Adds a vector index for the specified key selector.
/// Adds a vector index for the specified key selector.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The key selector expression.</param>
@@ -69,14 +70,15 @@ public class EntityTypeBuilder<T> where T : class
/// <param name="metric">The vector similarity metric.</param>
/// <param name="name">The optional index name.</param>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
public EntityTypeBuilder<T> HasVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions,
VectorMetric metric = VectorMetric.Cosine, string? name = null)
{
Indexes.Add(new IndexBuilder<T>(keySelector, name, false, IndexType.Vector, dimensions, metric));
return this;
}
/// <summary>
/// Adds a spatial index for the specified key selector.
/// Adds a spatial index for the specified key selector.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The key selector expression.</param>
@@ -89,7 +91,7 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Sets the primary key selector for the entity type.
/// Sets the primary key selector for the entity type.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The primary key selector expression.</param>
@@ -102,38 +104,35 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Configures a converter for the primary key property.
/// Configures a converter for the primary key property.
/// </summary>
/// <typeparam name="TConverter">The converter type.</typeparam>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasConversion<TConverter>()
{
if (!string.IsNullOrEmpty(PrimaryKeyName))
{
PropertyConverters[PrimaryKeyName] = typeof(TConverter);
}
if (!string.IsNullOrEmpty(PrimaryKeyName)) PropertyConverters[PrimaryKeyName] = typeof(TConverter);
return this;
}
/// <summary>
/// Configures a specific property on the entity type.
/// Configures a specific property on the entity type.
/// </summary>
/// <typeparam name="TProperty">The property type.</typeparam>
/// <param name="propertyExpression">The property expression.</param>
/// <returns>A builder for the selected property.</returns>
public PropertyBuilder Property<TProperty>(Expression<Func<T, TProperty>> propertyExpression)
{
var propertyName = ExpressionAnalyzer.ExtractPropertyPaths(propertyExpression).FirstOrDefault();
string? propertyName = ExpressionAnalyzer.ExtractPropertyPaths(propertyExpression).FirstOrDefault();
return new PropertyBuilder(this, propertyName);
}
public class PropertyBuilder
{
public class PropertyBuilder
{
private readonly EntityTypeBuilder<T> _parent;
private readonly string? _propertyName;
/// <summary>
/// Initializes a new instance of the <see cref="PropertyBuilder"/> class.
/// Initializes a new instance of the <see cref="PropertyBuilder" /> class.
/// </summary>
/// <param name="parent">The parent entity type builder.</param>
/// <param name="propertyName">The property name.</param>
@@ -144,68 +143,32 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Marks the configured property as value generated on add.
/// Marks the configured property as value generated on add.
/// </summary>
/// <returns>The current property builder.</returns>
public PropertyBuilder ValueGeneratedOnAdd()
{
if (_propertyName == _parent.PrimaryKeyName)
{
_parent.ValueGeneratedOnAdd = true;
}
if (_propertyName == _parent.PrimaryKeyName) _parent.ValueGeneratedOnAdd = true;
return this;
}
/// <summary>
/// Configures a converter for the configured property.
/// Configures a converter for the configured property.
/// </summary>
/// <typeparam name="TConverter">The converter type.</typeparam>
/// <returns>The current property builder.</returns>
public PropertyBuilder HasConversion<TConverter>()
{
if (!string.IsNullOrEmpty(_propertyName))
{
_parent.PropertyConverters[_propertyName] = typeof(TConverter);
}
return this;
}
}
}
if (!string.IsNullOrEmpty(_propertyName)) _parent.PropertyConverters[_propertyName] = typeof(TConverter);
return this;
}
}
}
public class IndexBuilder<T>
{
/// <summary>
/// Gets the index key selector expression.
/// </summary>
public LambdaExpression KeySelector { get; }
/// <summary>
/// Gets the configured index name.
/// </summary>
public string? Name { get; }
/// <summary>
/// Gets a value indicating whether the index is unique.
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Gets the index type.
/// </summary>
public IndexType Type { get; }
/// <summary>
/// Gets the vector dimensions.
/// </summary>
public int Dimensions { get; }
/// <summary>
/// Gets the vector metric.
/// </summary>
public VectorMetric Metric { get; }
/// <summary>
/// Initializes a new instance of the <see cref="IndexBuilder{T}"/> class.
/// Initializes a new instance of the <see cref="IndexBuilder{T}" /> class.
/// </summary>
/// <param name="keySelector">The index key selector expression.</param>
/// <param name="name">The optional index name.</param>
@@ -213,13 +176,44 @@ public class IndexBuilder<T>
/// <param name="type">The index type.</param>
/// <param name="dimensions">The vector dimensions.</param>
/// <param name="metric">The vector metric.</param>
public IndexBuilder(LambdaExpression keySelector, string? name, bool unique, IndexType type = IndexType.BTree, int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
public IndexBuilder(LambdaExpression keySelector, string? name, bool unique, IndexType type = IndexType.BTree,
int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
{
KeySelector = keySelector;
Name = name;
IsUnique = unique;
Type = type;
Dimensions = dimensions;
Metric = metric;
}
}
IsUnique = unique;
Type = type;
Dimensions = dimensions;
Metric = metric;
}
/// <summary>
/// Gets the index key selector expression.
/// </summary>
public LambdaExpression KeySelector { get; }
/// <summary>
/// Gets the configured index name.
/// </summary>
public string? Name { get; }
/// <summary>
/// Gets a value indicating whether the index is unique.
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Gets the index type.
/// </summary>
public IndexType Type { get; }
/// <summary>
/// Gets the vector dimensions.
/// </summary>
public int Dimensions { get; }
/// <summary>
/// Gets the vector metric.
/// </summary>
public VectorMetric Metric { get; }
}

33
src/CBDD.Core/Metadata/ModelBuilder.cs
View File

@@ -1,30 +1,31 @@
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Metadata;
public class ModelBuilder
{
private readonly Dictionary<Type, object> _entityBuilders = new();
/// <summary>
/// Gets or creates the entity builder for the specified entity type.
/// </summary>
/// <typeparam name="T">The entity type.</typeparam>
/// <returns>The entity builder for <typeparamref name="T"/>.</returns>
public EntityTypeBuilder<T> Entity<T>() where T : class
/// <summary>
/// Gets or creates the entity builder for the specified entity type.
/// </summary>
/// <typeparam name="T">The entity type.</typeparam>
/// <returns>The entity builder for <typeparamref name="T" />.</returns>
public EntityTypeBuilder<T> Entity<T>() where T : class
{
if (!_entityBuilders.TryGetValue(typeof(T), out var builder))
if (!_entityBuilders.TryGetValue(typeof(T), out object? builder))
{
builder = new EntityTypeBuilder<T>();
_entityBuilders[typeof(T)] = builder;
}
return (EntityTypeBuilder<T>)builder;
}
/// <summary>
/// Gets all registered entity builders.
/// </summary>
/// <returns>A read-only dictionary of entity builders keyed by entity type.</returns>
public IReadOnlyDictionary<Type, object> GetEntityBuilders() => _entityBuilders;
}
/// <summary>
/// Gets all registered entity builders.
/// </summary>
/// <returns>A read-only dictionary of entity builders keyed by entity type.</returns>
public IReadOnlyDictionary<Type, object> GetEntityBuilders()
{
return _entityBuilders;
}
}

22
src/CBDD.Core/Metadata/ValueConverter.cs
View File

@@ -1,20 +1,20 @@
namespace ZB.MOM.WW.CBDD.Core.Metadata;
/// <summary>
/// Defines a bidirectional conversion between a model type (e.g. ValueObject)
/// and a provider type supported by the storage engine (e.g. string, int, Guid, ObjectId).
/// </summary>
public abstract class ValueConverter<TModel, TProvider>
{
namespace ZB.MOM.WW.CBDD.Core.Metadata;
/// <summary>
/// Defines a bidirectional conversion between a model type (e.g. ValueObject)
/// and a provider type supported by the storage engine (e.g. string, int, Guid, ObjectId).
/// </summary>
public abstract class ValueConverter<TModel, TProvider>
{
/// <summary>
/// Converts the model value to the provider value.
/// Converts the model value to the provider value.
/// </summary>
/// <param name="model">The model value to convert.</param>
public abstract TProvider ConvertToProvider(TModel model);
/// <summary>
/// Converts the provider value back to the model value.
/// Converts the provider value back to the model value.
/// </summary>
/// <param name="provider">The provider value to convert.</param>
public abstract TModel ConvertFromProvider(TProvider provider);
}
}

31
src/CBDD.Core/Query/BTreeExpressionVisitor.cs
View File

@@ -4,18 +4,20 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal class BTreeExpressionVisitor : ExpressionVisitor
{
private readonly QueryModel _model = new();
/// <summary>
/// Gets the query model built while visiting an expression tree.
/// </summary>
public QueryModel GetModel() => _model;
/// <inheritdoc />
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Queryable))
{
private readonly QueryModel _model = new();
/// <summary>
/// Gets the query model built while visiting an expression tree.
/// </summary>
public QueryModel GetModel()
{
return _model;
}
/// <inheritdoc />
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Queryable))
switch (node.Method.Name)
{
case "Where":
@@ -35,8 +37,7 @@ internal class BTreeExpressionVisitor : ExpressionVisitor
VisitSkip(node);
break;
}
}
return base.VisitMethodCall(node);
}
@@ -94,4 +95,4 @@ internal class BTreeExpressionVisitor : ExpressionVisitor
if (countExpression.Value != null)
_model.Skip = (int)countExpression.Value;
}
}
}

128
src/CBDD.Core/Query/BTreeQueryProvider.cs
View File

@@ -1,17 +1,17 @@
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
using ZB.MOM.WW.CBDD.Core.Collections;
using static ZB.MOM.WW.CBDD.Core.Query.IndexOptimizer;
namespace ZB.MOM.WW.CBDD.Core.Query;
using System.Linq.Expressions;
using System.Reflection;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using static ZB.MOM.WW.CBDD.Core.Query.IndexOptimizer;
namespace ZB.MOM.WW.CBDD.Core.Query;
public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
{
private readonly DocumentCollection<TId, T> _collection;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeQueryProvider{TId, T}"/> class.
/// Initializes a new instance of the <see cref="BTreeQueryProvider{TId, T}" /> class.
/// </summary>
/// <param name="collection">The backing document collection.</param>
public BTreeQueryProvider(DocumentCollection<TId, T> collection)
@@ -20,38 +20,37 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
}
/// <summary>
/// Creates a query from the specified expression.
/// Creates a query from the specified expression.
/// </summary>
/// <param name="expression">The query expression.</param>
/// <returns>An <see cref="IQueryable"/> representing the query.</returns>
/// <returns>An <see cref="IQueryable" /> representing the query.</returns>
public IQueryable CreateQuery(Expression expression)
{
var elementType = expression.Type.GetGenericArguments()[0];
try
{
return (IQueryable)Activator.CreateInstance(
typeof(BTreeQueryable<>).MakeGenericType(elementType),
new object[] { this, expression })!;
}
catch (TargetInvocationException ex)
{
{
return (IQueryable)Activator.CreateInstance(
typeof(BTreeQueryable<>).MakeGenericType(elementType), this, expression)!;
}
catch (TargetInvocationException ex)
{
throw ex.InnerException ?? ex;
}
}
/// <summary>
/// Creates a strongly typed query from the specified expression.
/// Creates a strongly typed query from the specified expression.
/// </summary>
/// <typeparam name="TElement">The element type of the query.</typeparam>
/// <param name="expression">The query expression.</param>
/// <returns>An <see cref="IQueryable{T}"/> representing the query.</returns>
/// <returns>An <see cref="IQueryable{T}" /> representing the query.</returns>
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
return new BTreeQueryable<TElement>(this, expression);
}
/// <summary>
/// Executes a query expression.
/// Executes a query expression.
/// </summary>
/// <param name="expression">The query expression.</param>
/// <returns>The query result.</returns>
@@ -61,7 +60,7 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
}
/// <summary>
/// Executes a query expression and returns a strongly typed result.
/// Executes a query expression and returns a strongly typed result.
/// </summary>
/// <typeparam name="TResult">The result type.</typeparam>
/// <param name="expression">The query expression.</param>
@@ -72,88 +71,71 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
// We only care about WHERE clause for optimization.
// GroupBy, Select, OrderBy, etc. are handled by EnumerableRewriter.
var visitor = new BTreeExpressionVisitor();
visitor.Visit(expression);
var visitor = new BTreeExpressionVisitor();
visitor.Visit(expression);
var model = visitor.GetModel();
// 2. Data Fetching Strategy (Optimized or Full Scan)
IEnumerable<T> sourceData = null!;
// A. Try Index Optimization (Only if Where clause exists)
var indexOpt = IndexOptimizer.TryOptimize<T>(model, _collection.GetIndexes());
if (indexOpt != null)
var indexOpt = TryOptimize<T>(model, _collection.GetIndexes());
if (indexOpt != null)
{
if (indexOpt.IsVectorSearch)
{
sourceData = _collection.VectorSearch(indexOpt.IndexName, indexOpt.VectorQuery!, indexOpt.K);
}
else if (indexOpt.IsSpatialSearch)
{
sourceData = indexOpt.SpatialType == SpatialQueryType.Near
? _collection.Near(indexOpt.IndexName, indexOpt.SpatialPoint, indexOpt.RadiusKm)
: _collection.Within(indexOpt.IndexName, indexOpt.SpatialMin, indexOpt.SpatialMax);
}
else
{
sourceData = _collection.QueryIndex(indexOpt.IndexName, indexOpt.MinValue, indexOpt.MaxValue);
}
}
// B. Try Scan Optimization (if no index used)
if (sourceData == null)
{
Func<ZB.MOM.WW.CBDD.Bson.BsonSpanReader, bool>? bsonPredicate = null;
if (model.WhereClause != null)
{
bsonPredicate = BsonExpressionEvaluator.TryCompile<T>(model.WhereClause);
}
if (bsonPredicate != null)
{
sourceData = _collection.Scan(bsonPredicate);
}
}
// C. Fallback to Full Scan
if (sourceData == null)
{
sourceData = _collection.FindAll();
if (sourceData == null)
{
Func<BsonSpanReader, bool>? bsonPredicate = null;
if (model.WhereClause != null) bsonPredicate = BsonExpressionEvaluator.TryCompile<T>(model.WhereClause);
if (bsonPredicate != null) sourceData = _collection.Scan(bsonPredicate);
}
// C. Fallback to Full Scan
if (sourceData == null) sourceData = _collection.FindAll();
// 3. Rewrite Expression Tree to use Enumerable
// Replace the "Root" IQueryable with our sourceData IEnumerable
// We need to find the root IQueryable in the expression to replace it.
// It's likely the first argument of the first method call, or a constant.
var rootFinder = new RootFinder();
rootFinder.Visit(expression);
var rootFinder = new RootFinder();
rootFinder.Visit(expression);
var root = rootFinder.Root;
if (root == null) throw new InvalidOperationException("Could not find root Queryable in expression");
var rewriter = new EnumerableRewriter(root, sourceData);
if (root == null) throw new InvalidOperationException("Could not find root Queryable in expression");
var rewriter = new EnumerableRewriter(root, sourceData);
var rewrittenExpression = rewriter.Visit(expression);
// 4. Compile and Execute
// The rewritten expression is now a tree of IEnumerable calls returning TResult.
// We need to turn it into a Func<TResult> and invoke it.
if (rewrittenExpression.Type != typeof(TResult))
{
if (rewrittenExpression.Type != typeof(TResult))
// If TResult is object (non-generic Execute), we need to cast
rewrittenExpression = Expression.Convert(rewrittenExpression, typeof(TResult));
}
var lambda = Expression.Lambda<Func<TResult>>(rewrittenExpression);
var compiled = lambda.Compile();
return compiled();
rewrittenExpression = Expression.Convert(rewrittenExpression, typeof(TResult));
var lambda = Expression.Lambda<Func<TResult>>(rewrittenExpression);
var compiled = lambda.Compile();
return compiled();
}
private class RootFinder : ExpressionVisitor
{
/// <summary>
/// Gets the root queryable found in the expression tree.
/// Gets the root queryable found in the expression tree.
/// </summary>
public IQueryable? Root { get; private set; }
@@ -161,13 +143,11 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
protected override Expression VisitConstant(ConstantExpression node)
{
// If we found a Queryable, that's our root source
if (Root == null && node.Value is IQueryable q)
{
// We typically want the "base" queryable (the BTreeQueryable instance)
// In a chain like Coll.Where.Select, the root is Coll.
Root = q;
}
return base.VisitConstant(node);
}
}
}
if (Root == null && node.Value is IQueryable q)
// We typically want the "base" queryable (the BTreeQueryable instance)
// In a chain like Coll.Where.Select, the root is Coll.
Root = q;
return base.VisitConstant(node);
}
}
}

23
src/CBDD.Core/Query/BTreeQueryable.cs
View File

@@ -1,13 +1,12 @@
using System.Collections;
using System.Linq;
using System.Linq.Expressions;
namespace ZB.MOM.WW.CBDD.Core.Query;
using System.Collections;
using System.Linq.Expressions;
namespace ZB.MOM.WW.CBDD.Core.Query;
internal class BTreeQueryable<T> : IOrderedQueryable<T>
{
/// <summary>
/// Initializes a new queryable wrapper for the specified provider and expression.
/// Initializes a new queryable wrapper for the specified provider and expression.
/// </summary>
/// <param name="provider">The query provider.</param>
/// <param name="expression">The expression tree.</param>
@@ -18,7 +17,7 @@ internal class BTreeQueryable<T> : IOrderedQueryable<T>
}
/// <summary>
/// Initializes a new queryable wrapper for the specified provider.
/// Initializes a new queryable wrapper for the specified provider.
/// </summary>
/// <param name="provider">The query provider.</param>
public BTreeQueryable(IQueryProvider provider)
@@ -28,17 +27,17 @@ internal class BTreeQueryable<T> : IOrderedQueryable<T>
}
/// <summary>
/// Gets the element type returned by this query.
/// Gets the element type returned by this query.
/// </summary>
public Type ElementType => typeof(T);
/// <summary>
/// Gets the expression tree associated with this query.
/// Gets the expression tree associated with this query.
/// </summary>
public Expression Expression { get; }
/// <summary>
/// Gets the query provider for this query.
/// Gets the query provider for this query.
/// </summary>
public IQueryProvider Provider { get; }
@@ -53,4 +52,4 @@ internal class BTreeQueryable<T> : IOrderedQueryable<T>
{
return GetEnumerator();
}
}
}

45
src/CBDD.Core/Query/BsonExpressionEvaluator.cs
View File

@@ -6,11 +6,11 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal static class BsonExpressionEvaluator
{
/// <summary>
/// Attempts to compile a LINQ predicate expression into a BSON reader predicate.
/// Attempts to compile a LINQ predicate expression into a BSON reader predicate.
/// </summary>
/// <typeparam name="T">The entity type of the original expression.</typeparam>
/// <param name="expression">The lambda expression to compile.</param>
/// <returns>A compiled predicate when supported; otherwise, <see langword="null"/>.</returns>
/// <returns>A compiled predicate when supported; otherwise, <see langword="null" />.</returns>
public static Func<BsonSpanReader, bool>? TryCompile<T>(LambdaExpression expression)
{
// Simple optimization for: x => x.Prop op Constant
@@ -29,12 +29,11 @@ internal static class BsonExpressionEvaluator
}
if (left is MemberExpression member && right is ConstantExpression constant)
{
// Check if member is property of parameter
if (member.Expression == expression.Parameters[0])
{
var propertyName = member.Member.Name.ToLowerInvariant();
var value = constant.Value;
string propertyName = member.Member.Name.ToLowerInvariant();
object? value = constant.Value;
// Handle Id mapping?
// If property is "id", Bson field is "_id"
@@ -42,22 +41,25 @@ internal static class BsonExpressionEvaluator
return CreatePredicate(propertyName, value, nodeType);
}
}
}
return null;
}
private static ExpressionType Flip(ExpressionType type) => type switch
private static ExpressionType Flip(ExpressionType type)
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
return type switch
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}
private static Func<BsonSpanReader, bool>? CreatePredicate(string propertyName, object? targetValue, ExpressionType op)
private static Func<BsonSpanReader, bool>? CreatePredicate(string propertyName, object? targetValue,
ExpressionType op)
{
// We need to return a delegate that searches for propertyName in BsonSpanReader and compares
@@ -71,13 +73,11 @@ internal static class BsonExpressionEvaluator
var type = reader.ReadBsonType();
if (type == 0) break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == propertyName)
{
// Found -> read value and compare
return Compare(ref reader, type, targetValue, op);
}
reader.SkipValue(type);
}
@@ -86,6 +86,7 @@ internal static class BsonExpressionEvaluator
{
return false;
}
return false; // Not found
};
}
@@ -97,9 +98,8 @@ internal static class BsonExpressionEvaluator
if (type == BsonType.Int32)
{
var val = reader.ReadInt32();
int val = reader.ReadInt32();
if (target is int targetInt)
{
return op switch
{
ExpressionType.Equal => val == targetInt,
@@ -110,14 +110,13 @@ internal static class BsonExpressionEvaluator
ExpressionType.LessThanOrEqual => val <= targetInt,
_ => false
};
}
}
else if (type == BsonType.String)
{
var val = reader.ReadString();
string val = reader.ReadString();
if (target is string targetStr)
{
var cmp = string.Compare(val, targetStr, StringComparison.Ordinal);
int cmp = string.Compare(val, targetStr, StringComparison.Ordinal);
return op switch
{
ExpressionType.Equal => cmp == 0,
@@ -140,4 +139,4 @@ internal static class BsonExpressionEvaluator
return false;
}
}
}

57
src/CBDD.Core/Query/EnumerableRewriter.cs
View File

@@ -1,6 +1,3 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
@@ -8,50 +5,48 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal class EnumerableRewriter : ExpressionVisitor
{
private readonly IQueryable _source;
private readonly object _target;
/// <summary>
/// Initializes a new instance of the <see cref="EnumerableRewriter"/> class.
/// </summary>
/// <param name="source">The original queryable source to replace.</param>
/// <param name="target">The target enumerable-backed object.</param>
public EnumerableRewriter(IQueryable source, object target)
{
_source = source;
_target = target;
}
/// <inheritdoc />
protected override Expression VisitConstant(ConstantExpression node)
private readonly IQueryable _source;
private readonly object _target;
/// <summary>
/// Initializes a new instance of the <see cref="EnumerableRewriter" /> class.
/// </summary>
/// <param name="source">The original queryable source to replace.</param>
/// <param name="target">The target enumerable-backed object.</param>
public EnumerableRewriter(IQueryable source, object target)
{
_source = source;
_target = target;
}
/// <inheritdoc />
protected override Expression VisitConstant(ConstantExpression node)
{
// Replace the IQueryable source with the materialized IEnumerable
if (node.Value == _source)
{
return Expression.Constant(_target);
}
if (node.Value == _source) return Expression.Constant(_target);
return base.VisitConstant(node);
}
/// <inheritdoc />
protected override Expression VisitMethodCall(MethodCallExpression node)
/// <inheritdoc />
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Queryable))
{
var methodName = node.Method.Name;
string methodName = node.Method.Name;
var typeArgs = node.Method.GetGenericArguments();
var args = new Expression[node.Arguments.Count];
for (int i = 0; i < node.Arguments.Count; i++)
for (var i = 0; i < node.Arguments.Count; i++)
{
var arg = Visit(node.Arguments[i]);
// Strip Quote from lambda arguments
var arg = Visit(node.Arguments[i]);
// Strip Quote from lambda arguments
if (arg is UnaryExpression quote && quote.NodeType == ExpressionType.Quote)
{
var lambda = (LambdaExpression)quote.Operand;
arg = Expression.Constant(lambda.Compile());
}
args[i] = arg;
}
@@ -83,4 +78,4 @@ internal class EnumerableRewriter : ExpressionVisitor
return base.VisitMethodCall(node);
}
}
}

292
src/CBDD.Core/Query/IndexOptimizer.cs
View File

@@ -3,96 +3,30 @@ using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Query;
internal static class IndexOptimizer
{
/// <summary>
/// Represents the selected index and bounds for an optimized query.
/// </summary>
public class OptimizationResult
{
/// <summary>
/// Gets or sets the selected index name.
/// </summary>
public string IndexName { get; set; } = "";
/// <summary>
/// Gets or sets the minimum bound value.
/// </summary>
public object? MinValue { get; set; }
/// <summary>
/// Gets or sets the maximum bound value.
/// </summary>
public object? MaxValue { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses a range.
/// </summary>
public bool IsRange { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses vector search.
/// </summary>
public bool IsVectorSearch { get; set; }
/// <summary>
/// Gets or sets the vector query values.
/// </summary>
public float[]? VectorQuery { get; set; }
/// <summary>
/// Gets or sets the number of nearest neighbors for vector search.
/// </summary>
public int K { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses spatial search.
/// </summary>
public bool IsSpatialSearch { get; set; }
/// <summary>
/// Gets or sets the center point for near queries.
/// </summary>
public (double Latitude, double Longitude) SpatialPoint { get; set; }
/// <summary>
/// Gets or sets the search radius in kilometers.
/// </summary>
public double RadiusKm { get; set; }
/// <summary>
/// Gets or sets the minimum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMin { get; set; }
/// <summary>
/// Gets or sets the maximum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMax { get; set; }
/// <summary>
/// Gets or sets the spatial query type.
/// </summary>
public SpatialQueryType SpatialType { get; set; }
}
internal static class IndexOptimizer
{
public enum SpatialQueryType
{
Near,
Within
}
public enum SpatialQueryType { Near, Within }
/// <summary>
/// Attempts to optimize a query model using available indexes.
/// </summary>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="model">The query model.</param>
/// <param name="indexes">The available collection indexes.</param>
/// <returns>An optimization result when optimization is possible; otherwise, <see langword="null"/>.</returns>
public static OptimizationResult? TryOptimize<T>(QueryModel model, IEnumerable<CollectionIndexInfo> indexes)
{
if (model.WhereClause == null) return null;
/// <summary>
/// Attempts to optimize a query model using available indexes.
/// </summary>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="model">The query model.</param>
/// <param name="indexes">The available collection indexes.</param>
/// <returns>An optimization result when optimization is possible; otherwise, <see langword="null" />.</returns>
public static OptimizationResult? TryOptimize<T>(QueryModel model, IEnumerable<CollectionIndexInfo> indexes)
{
if (model.WhereClause == null) return null;
return OptimizeExpression(model.WhereClause.Body, model.WhereClause.Parameters[0], indexes);
}
private static OptimizationResult? OptimizeExpression(Expression expression, ParameterExpression parameter, IEnumerable<CollectionIndexInfo> indexes)
private static OptimizationResult? OptimizeExpression(Expression expression, ParameterExpression parameter,
IEnumerable<CollectionIndexInfo> indexes)
{
// ... (Existing AndAlso logic remains the same) ...
if (expression is BinaryExpression binary && binary.NodeType == ExpressionType.AndAlso)
@@ -101,7 +35,6 @@ internal static class IndexOptimizer
var right = OptimizeExpression(binary.Right, parameter, indexes);
if (left != null && right != null && left.IndexName == right.IndexName)
{
return new OptimizationResult
{
IndexName = left.IndexName,
@@ -109,12 +42,11 @@ internal static class IndexOptimizer
MaxValue = left.MaxValue ?? right.MaxValue,
IsRange = true
};
}
return left ?? right;
}
// Handle Simple Binary Predicates
var (propertyName, value, op) = ParseSimplePredicate(expression, parameter);
(string? propertyName, object? value, var op) = ParseSimplePredicate(expression, parameter);
if (propertyName != null)
{
var index = indexes.FirstOrDefault(i => Matches(i, propertyName));
@@ -128,55 +60,56 @@ internal static class IndexOptimizer
result.MaxValue = value;
result.IsRange = false;
break;
case ExpressionType.GreaterThan:
case ExpressionType.GreaterThan:
case ExpressionType.GreaterThanOrEqual:
result.MinValue = value;
result.MaxValue = null;
result.IsRange = true;
break;
case ExpressionType.LessThan:
break;
case ExpressionType.LessThan:
case ExpressionType.LessThanOrEqual:
result.MinValue = null;
result.MaxValue = value;
result.IsRange = true;
break;
}
return result;
}
}
// Handle StartsWith
if (expression is MethodCallExpression call && call.Method.Name == "StartsWith" && call.Object is MemberExpression member)
{
if (member.Expression == parameter && call.Arguments[0] is ConstantExpression constant && constant.Value is string prefix)
{
var index = indexes.FirstOrDefault(i => Matches(i, member.Member.Name));
if (index != null && index.Type == IndexType.BTree)
{
var nextPrefix = IncrementPrefix(prefix);
return new OptimizationResult
{
IndexName = index.Name,
MinValue = prefix,
MaxValue = nextPrefix,
IsRange = true
};
}
}
}
// Handle StartsWith
if (expression is MethodCallExpression call && call.Method.Name == "StartsWith" &&
call.Object is MemberExpression member)
if (member.Expression == parameter && call.Arguments[0] is ConstantExpression constant &&
constant.Value is string prefix)
{
var index = indexes.FirstOrDefault(i => Matches(i, member.Member.Name));
if (index != null && index.Type == IndexType.BTree)
{
string nextPrefix = IncrementPrefix(prefix);
return new OptimizationResult
{
IndexName = index.Name,
MinValue = prefix,
MaxValue = nextPrefix,
IsRange = true
};
}
}
// Handle Method Calls (VectorSearch, Near, Within)
if (expression is MethodCallExpression mcall)
{
// VectorSearch(this float[] vector, float[] query, int k)
if (mcall.Method.Name == "VectorSearch" && mcall.Arguments[0] is MemberExpression vMember && vMember.Expression == parameter)
if (mcall.Method.Name == "VectorSearch" && mcall.Arguments[0] is MemberExpression vMember &&
vMember.Expression == parameter)
{
var query = EvaluateExpression<float[]>(mcall.Arguments[1]);
var k = EvaluateExpression<int>(mcall.Arguments[2]);
float[] query = EvaluateExpression<float[]>(mcall.Arguments[1]);
var k = EvaluateExpression<int>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Vector && Matches(i, vMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
@@ -184,18 +117,17 @@ internal static class IndexOptimizer
VectorQuery = query,
K = k
};
}
}
// Near(this (double, double) point, (double, double) center, double radiusKm)
if (mcall.Method.Name == "Near" && mcall.Arguments[0] is MemberExpression nMember && nMember.Expression == parameter)
}
// Near(this (double, double) point, (double, double) center, double radiusKm)
if (mcall.Method.Name == "Near" && mcall.Arguments[0] is MemberExpression nMember &&
nMember.Expression == parameter)
{
var center = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var radius = EvaluateExpression<double>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, nMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
@@ -204,18 +136,17 @@ internal static class IndexOptimizer
SpatialPoint = center,
RadiusKm = radius
};
}
}
// Within(this (double, double) point, (double, double) min, (double, double) max)
if (mcall.Method.Name == "Within" && mcall.Arguments[0] is MemberExpression wMember && wMember.Expression == parameter)
if (mcall.Method.Name == "Within" && mcall.Arguments[0] is MemberExpression wMember &&
wMember.Expression == parameter)
{
var min = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var max = EvaluateExpression<(double, double)>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, wMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
@@ -224,7 +155,6 @@ internal static class IndexOptimizer
SpatialMin = min,
SpatialMax = max
};
}
}
}
@@ -241,10 +171,7 @@ internal static class IndexOptimizer
private static T EvaluateExpression<T>(Expression expression)
{
if (expression is ConstantExpression constant)
{
return (T)constant.Value!;
}
if (expression is ConstantExpression constant) return (T)constant.Value!;
// Evaluate more complex expressions (closures, properties, etc.)
var lambda = Expression.Lambda(expression);
@@ -258,7 +185,8 @@ internal static class IndexOptimizer
return index.PropertyPaths[0].Equals(propertyName, StringComparison.OrdinalIgnoreCase);
}
private static (string? propertyName, object? value, ExpressionType op) ParseSimplePredicate(Expression expression, ParameterExpression parameter)
private static (string? propertyName, object? value, ExpressionType op) ParseSimplePredicate(Expression expression,
ParameterExpression parameter)
{
if (expression is BinaryExpression binary)
{
@@ -273,27 +201,99 @@ internal static class IndexOptimizer
}
if (left is MemberExpression member && right is ConstantExpression constant)
{
if (member.Expression == parameter)
return (member.Member.Name, constant.Value, nodeType);
}
// Handle Convert
if (left is UnaryExpression unary && unary.Operand is MemberExpression member2 && right is ConstantExpression constant2)
{
// Handle Convert
if (left is UnaryExpression unary && unary.Operand is MemberExpression member2 &&
right is ConstantExpression constant2)
if (member2.Expression == parameter)
return (member2.Member.Name, constant2.Value, nodeType);
}
}
return (null, null, ExpressionType.Default);
}
private static ExpressionType Flip(ExpressionType type) => type switch
private static ExpressionType Flip(ExpressionType type)
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}
return type switch
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}
/// <summary>
/// Represents the selected index and bounds for an optimized query.
/// </summary>
public class OptimizationResult
{
/// <summary>
/// Gets or sets the selected index name.
/// </summary>
public string IndexName { get; set; } = "";
/// <summary>
/// Gets or sets the minimum bound value.
/// </summary>
public object? MinValue { get; set; }
/// <summary>
/// Gets or sets the maximum bound value.
/// </summary>
public object? MaxValue { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses a range.
/// </summary>
public bool IsRange { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses vector search.
/// </summary>
public bool IsVectorSearch { get; set; }
/// <summary>
/// Gets or sets the vector query values.
/// </summary>
public float[]? VectorQuery { get; set; }
/// <summary>
/// Gets or sets the number of nearest neighbors for vector search.
/// </summary>
public int K { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses spatial search.
/// </summary>
public bool IsSpatialSearch { get; set; }
/// <summary>
/// Gets or sets the center point for near queries.
/// </summary>
public (double Latitude, double Longitude) SpatialPoint { get; set; }
/// <summary>
/// Gets or sets the search radius in kilometers.
/// </summary>
public double RadiusKm { get; set; }
/// <summary>
/// Gets or sets the minimum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMin { get; set; }
/// <summary>
/// Gets or sets the maximum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMax { get; set; }
/// <summary>
/// Gets or sets the spatial query type.
/// </summary>
public SpatialQueryType SpatialType { get; set; }
}
}

22
src/CBDD.Core/Query/QueryModel.cs
View File

@@ -1,36 +1,36 @@
using System.Linq.Expressions;
namespace ZB.MOM.WW.CBDD.Core.Query;
using System.Linq.Expressions;
namespace ZB.MOM.WW.CBDD.Core.Query;
internal class QueryModel
{
/// <summary>
/// Gets or sets the filter expression.
/// Gets or sets the filter expression.
/// </summary>
public LambdaExpression? WhereClause { get; set; }
/// <summary>
/// Gets or sets the projection expression.
/// Gets or sets the projection expression.
/// </summary>
public LambdaExpression? SelectClause { get; set; }
/// <summary>
/// Gets or sets the ordering expression.
/// Gets or sets the ordering expression.
/// </summary>
public LambdaExpression? OrderByClause { get; set; }
/// <summary>
/// Gets or sets the maximum number of results to return.
/// Gets or sets the maximum number of results to return.
/// </summary>
public int? Take { get; set; }
/// <summary>
/// Gets or sets the number of results to skip.
/// Gets or sets the number of results to skip.
/// </summary>
public int? Skip { get; set; }
/// <summary>
/// Gets or sets a value indicating whether ordering is descending.
/// Gets or sets a value indicating whether ordering is descending.
/// </summary>
public bool OrderDescending { get; set; }
}
}

272
src/CBDD.Core/Storage/DictionaryPage.cs
View File

@@ -1,13 +1,13 @@
using System.Runtime.InteropServices;
using System.Buffers;
using System.Buffers.Binary;
using System.Text;
using ZB.MOM.WW.CBDD.Core;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page for storing dictionary entries (Key -> Value map).
/// Uses a sorted list of keys for binary search within the page.
/// Supports chaining via PageHeader.NextPageId for dictionaries larger than one page.
/// Page for storing dictionary entries (Key -> Value map).
/// Uses a sorted list of keys for binary search within the page.
/// Supports chaining via PageHeader.NextPageId for dictionaries larger than one page.
/// </summary>
public struct DictionaryPage
{
@@ -25,16 +25,16 @@ public struct DictionaryPage
private const int OffsetsStart = 36;
/// <summary>
/// Values 0-100 are reserved for internal system keys (e.g. _id, _v).
/// Values 0-100 are reserved for internal system keys (e.g. _id, _v).
/// </summary>
public const ushort ReservedValuesEnd = 100;
/// <summary>
/// Initialize a new dictionary page
/// </summary>
/// <param name="page">The page buffer to initialize.</param>
/// <param name="pageId">The page identifier.</param>
public static void Initialize(Span<byte> page, uint pageId)
/// <summary>
/// Initialize a new dictionary page
/// </summary>
/// <param name="page">The page buffer to initialize.</param>
/// <param name="pageId">The page identifier.</param>
public static void Initialize(Span<byte> page, uint pageId)
{
// 1. Write Page Header
var header = new PageHeader
@@ -49,43 +49,40 @@ public struct DictionaryPage
header.WriteTo(page);
// 2. Initialize Counts
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), 0);
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), (ushort)page.Length);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), 0);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), (ushort)page.Length);
}
/// <summary>
/// Inserts a key-value pair into the page.
/// Returns false if there is not enough space.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="key">The dictionary key.</param>
/// <param name="value">The value mapped to the key.</param>
/// <returns><see langword="true"/> if the entry was inserted; otherwise, <see langword="false"/>.</returns>
public static bool Insert(Span<byte> page, string key, ushort value)
/// <summary>
/// Inserts a key-value pair into the page.
/// Returns false if there is not enough space.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="key">The dictionary key.</param>
/// <param name="value">The value mapped to the key.</param>
/// <returns><see langword="true" /> if the entry was inserted; otherwise, <see langword="false" />.</returns>
public static bool Insert(Span<byte> page, string key, ushort value)
{
var keyByteCount = Encoding.UTF8.GetByteCount(key);
int keyByteCount = Encoding.UTF8.GetByteCount(key);
if (keyByteCount > 255) throw new ArgumentException("Key length must be <= 255 bytes");
// Entry Size: KeyLen(1) + Key(N) + Value(2)
var entrySize = 1 + keyByteCount + 2;
var requiredSpace = entrySize + 2; // +2 for Offset entry
int entrySize = 1 + keyByteCount + 2;
int requiredSpace = entrySize + 2; // +2 for Offset entry
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
var freeSpaceEnd = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(FreeSpaceEndOffset));
var offsetsEnd = OffsetsStart + (count * 2);
var freeSpace = freeSpaceEnd - offsetsEnd;
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort freeSpaceEnd = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(FreeSpaceEndOffset));
if (freeSpace < requiredSpace)
{
return false; // Page Full
}
int offsetsEnd = OffsetsStart + count * 2;
int freeSpace = freeSpaceEnd - offsetsEnd;
if (freeSpace < requiredSpace) return false; // Page Full
// 1. Prepare Data
var insertionOffset = (ushort)(freeSpaceEnd - entrySize);
page[insertionOffset] = (byte)keyByteCount; // Write Key Length
Encoding.UTF8.GetBytes(key, page.Slice(insertionOffset + 1, keyByteCount)); // Write Key
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(insertionOffset + 1 + keyByteCount), value); // Write Value
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(insertionOffset + 1 + keyByteCount), value); // Write Value
// 2. Insert Offset into Sorted List
// Find insert Index using spans
@@ -95,57 +92,57 @@ public struct DictionaryPage
// Shift offsets if needed
if (insertIndex < count)
{
var src = page.Slice(OffsetsStart + (insertIndex * 2), (count - insertIndex) * 2);
var dest = page.Slice(OffsetsStart + ((insertIndex + 1) * 2));
var src = page.Slice(OffsetsStart + insertIndex * 2, (count - insertIndex) * 2);
var dest = page.Slice(OffsetsStart + (insertIndex + 1) * 2);
src.CopyTo(dest);
}
// Write new offset
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(OffsetsStart + (insertIndex * 2)), insertionOffset);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(OffsetsStart + insertIndex * 2), insertionOffset);
// 3. Update Metadata
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), (ushort)(count + 1));
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), insertionOffset);
// Update FreeBytes in header (approximate)
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), (ushort)(count + 1));
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), insertionOffset);
// Update FreeBytes in header (approximate)
var pageHeader = PageHeader.ReadFrom(page);
pageHeader.FreeBytes = (ushort)(insertionOffset - (OffsetsStart + ((count + 1) * 2)));
pageHeader.FreeBytes = (ushort)(insertionOffset - (OffsetsStart + (count + 1) * 2));
pageHeader.WriteTo(page);
return true;
}
/// <summary>
/// Tries to find a value for the given key in THIS page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="keyBytes">The UTF-8 encoded key bytes.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <returns><see langword="true"/> if the key was found; otherwise, <see langword="false"/>.</returns>
public static bool TryFind(ReadOnlySpan<byte> page, ReadOnlySpan<byte> keyBytes, out ushort value)
/// <summary>
/// Tries to find a value for the given key in THIS page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="keyBytes">The UTF-8 encoded key bytes.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <returns><see langword="true" /> if the key was found; otherwise, <see langword="false" />.</returns>
public static bool TryFind(ReadOnlySpan<byte> page, ReadOnlySpan<byte> keyBytes, out ushort value)
{
value = 0;
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
if (count == 0) return false;
// Binary Search
int low = 0;
var low = 0;
int high = count - 1;
while (low <= high)
{
int mid = low + (high - low) / 2;
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (mid * 2)));
// Read Key at Offset
var keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + mid * 2));
// Read Key at Offset
byte keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
int comparison = entryKeySpan.SequenceCompareTo(keyBytes);
if (comparison == 0)
{
value = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
value = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
return true;
}
@@ -158,126 +155,125 @@ public struct DictionaryPage
return false;
}
/// <summary>
/// Tries to find a value for the given key across a chain of DictionaryPages.
/// </summary>
/// <param name="storage">The storage engine used to read pages.</param>
/// <param name="startPageId">The first page in the dictionary chain.</param>
/// <param name="key">The key to search for.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns><see langword="true"/> if the key was found; otherwise, <see langword="false"/>.</returns>
public static bool TryFindGlobal(StorageEngine storage, uint startPageId, string key, out ushort value, ulong? transactionId = null)
/// <summary>
/// Tries to find a value for the given key across a chain of DictionaryPages.
/// </summary>
/// <param name="storage">The storage engine used to read pages.</param>
/// <param name="startPageId">The first page in the dictionary chain.</param>
/// <param name="key">The key to search for.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns><see langword="true" /> if the key was found; otherwise, <see langword="false" />.</returns>
public static bool TryFindGlobal(StorageEngine storage, uint startPageId, string key, out ushort value,
ulong? transactionId = null)
{
var keyByteCount = Encoding.UTF8.GetByteCount(key);
Span<byte> keyBytes = keyByteCount <= 256 ? stackalloc byte[keyByteCount] : new byte[keyByteCount];
int keyByteCount = Encoding.UTF8.GetByteCount(key);
var keyBytes = keyByteCount <= 256 ? stackalloc byte[keyByteCount] : new byte[keyByteCount];
Encoding.UTF8.GetBytes(key, keyBytes);
var pageId = startPageId;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(storage.PageSize);
uint pageId = startPageId;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(storage.PageSize);
try
{
while (pageId != 0)
{
// Read page
storage.ReadPage(pageId, transactionId, pageBuffer);
// TryFind in this page
if (TryFind(pageBuffer, keyBytes, out value))
{
return true;
}
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
{
// Read page
storage.ReadPage(pageId, transactionId, pageBuffer);
// TryFind in this page
if (TryFind(pageBuffer, keyBytes, out value)) return true;
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
pageId = header.NextPageId;
}
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
}
ArrayPool<byte>.Shared.Return(pageBuffer);
}
value = 0;
return false;
}
private static int FindInsertIndex(ReadOnlySpan<byte> page, int count, ReadOnlySpan<byte> keyBytes)
{
int low = 0;
var low = 0;
int high = count - 1;
while (low <= high)
{
int mid = low + (high - low) / 2;
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (mid * 2)));
var keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + mid * 2));
byte keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
int comparison = entryKeySpan.SequenceCompareTo(keyBytes);
if (comparison == 0) return mid;
if (comparison == 0) return mid;
if (comparison < 0)
low = mid + 1;
else
high = mid - 1;
}
return low;
}
/// <summary>
/// Gets all entries in the page (for debugging/dumping)
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>All key-value pairs in the page.</returns>
public static IEnumerable<(string Key, ushort Value)> GetAll(ReadOnlySpan<byte> page)
{
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
var list = new List<(string Key, ushort Value)>();
for (int i = 0; i < count; i++)
/// <summary>
/// Gets all entries in the page (for debugging/dumping)
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>All key-value pairs in the page.</returns>
public static IEnumerable<(string Key, ushort Value)> GetAll(ReadOnlySpan<byte> page)
{
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
var list = new List<(string Key, ushort Value)>();
for (var i = 0; i < count; i++)
{
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (i * 2)));
var keyLen = page[offset];
var keyStr = Encoding.UTF8.GetString(page.Slice(offset + 1, keyLen));
var val = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
list.Add((keyStr, val));
}
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + i * 2));
byte keyLen = page[offset];
string keyStr = Encoding.UTF8.GetString(page.Slice(offset + 1, keyLen));
ushort val = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
list.Add((keyStr, val));
}
return list;
}
/// <summary>
/// Retrieves all key-value pairs across a chain of DictionaryPages.
/// Used for rebuilding the in-memory cache.
/// </summary>
/// <param name="storage">The storage engine used to read pages.</param>
/// <param name="startPageId">The first page in the dictionary chain.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns>All key-value pairs across the page chain.</returns>
public static IEnumerable<(string Key, ushort Value)> FindAllGlobal(StorageEngine storage, uint startPageId, ulong? transactionId = null)
/// <summary>
/// Retrieves all key-value pairs across a chain of DictionaryPages.
/// Used for rebuilding the in-memory cache.
/// </summary>
/// <param name="storage">The storage engine used to read pages.</param>
/// <param name="startPageId">The first page in the dictionary chain.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns>All key-value pairs across the page chain.</returns>
public static IEnumerable<(string Key, ushort Value)> FindAllGlobal(StorageEngine storage, uint startPageId,
ulong? transactionId = null)
{
var pageId = startPageId;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(storage.PageSize);
uint pageId = startPageId;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(storage.PageSize);
try
{
while (pageId != 0)
{
// Read page
storage.ReadPage(pageId, transactionId, pageBuffer);
// Get all entries in this page
foreach (var entry in GetAll(pageBuffer))
{
yield return entry;
}
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
{
// Read page
storage.ReadPage(pageId, transactionId, pageBuffer);
// Get all entries in this page
foreach (var entry in GetAll(pageBuffer)) yield return entry;
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
pageId = header.NextPageId;
}
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
}
}

21
src/CBDD.Core/Storage/IIndexStorage.cs
View File

@@ -1,41 +1,46 @@
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Narrow storage port for index structures (page operations + allocation only).
/// Narrow storage port for index structures (page operations + allocation only).
/// </summary>
internal interface IIndexStorage
{
/// <summary>
/// Gets or sets the PageSize.
/// Gets or sets the PageSize.
/// </summary>
int PageSize { get; }
/// <summary>
/// Executes AllocatePage.
/// Executes AllocatePage.
/// </summary>
uint AllocatePage();
/// <summary>
/// Executes FreePage.
/// Executes FreePage.
/// </summary>
/// <param name="pageId">The page identifier.</param>
void FreePage(uint pageId);
/// <summary>
/// Executes ReadPage.
/// Executes ReadPage.
/// </summary>
/// <param name="pageId">The page identifier.</param>
/// <param name="transactionId">The optional transaction identifier.</param>
/// <param name="destination">The destination buffer.</param>
void ReadPage(uint pageId, ulong? transactionId, Span<byte> destination);
/// <summary>
/// Executes WritePage.
/// Executes WritePage.
/// </summary>
/// <param name="pageId">The page identifier.</param>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="data">The source page data.</param>
void WritePage(uint pageId, ulong transactionId, ReadOnlySpan<byte> data);
/// <summary>
/// Executes WritePageImmediate.
/// Executes WritePageImmediate.
/// </summary>
/// <param name="pageId">The page identifier.</param>
/// <param name="data">The source page data.</param>
void WritePageImmediate(uint pageId, ReadOnlySpan<byte> data);
}
}

43
src/CBDD.Core/Storage/IStorageEngine.cs
View File

@@ -8,111 +8,112 @@ using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Storage port used by collection/index orchestration to avoid concrete engine coupling.
/// Storage port used by collection/index orchestration to avoid concrete engine coupling.
/// </summary>
internal interface IStorageEngine : IIndexStorage, IDisposable
{
/// <summary>
/// Gets the current page count.
/// Gets the current page count.
/// </summary>
uint PageCount { get; }
/// <summary>
/// Gets the active change stream dispatcher.
/// Gets the active change stream dispatcher.
/// </summary>
ChangeStreamDispatcher? Cdc { get; }
/// <summary>
/// Gets compression options used by the storage engine.
/// Gets compression options used by the storage engine.
/// </summary>
CompressionOptions CompressionOptions { get; }
/// <summary>
/// Gets the compression service.
/// Gets the compression service.
/// </summary>
CompressionService CompressionService { get; }
/// <summary>
/// Gets compression telemetry for the storage engine.
/// Gets compression telemetry for the storage engine.
/// </summary>
CompressionTelemetry CompressionTelemetry { get; }
/// <summary>
/// Determines whether a page is locked.
/// Determines whether a page is locked.
/// </summary>
/// <param name="pageId">The page identifier to inspect.</param>
/// <param name="excludingTxId">A transaction identifier to exclude from lock checks.</param>
bool IsPageLocked(uint pageId, ulong excludingTxId);
/// <summary>
/// Registers the change stream dispatcher.
/// Registers the change stream dispatcher.
/// </summary>
/// <param name="cdc">The change stream dispatcher instance.</param>
void RegisterCdc(ChangeStreamDispatcher cdc);
/// <summary>
/// Begins a transaction.
/// Begins a transaction.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
Transaction BeginTransaction(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted);
/// <summary>
/// Begins a transaction asynchronously.
/// Begins a transaction asynchronously.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">A cancellation token.</param>
Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted, CancellationToken ct = default);
Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted,
CancellationToken ct = default);
/// <summary>
/// Gets collection metadata by name.
/// Gets collection metadata by name.
/// </summary>
/// <param name="name">The collection name.</param>
CollectionMetadata? GetCollectionMetadata(string name);
/// <summary>
/// Saves collection metadata.
/// Saves collection metadata.
/// </summary>
/// <param name="metadata">The metadata to persist.</param>
void SaveCollectionMetadata(CollectionMetadata metadata);
/// <summary>
/// Registers document mappers.
/// Registers document mappers.
/// </summary>
/// <param name="mappers">The mapper instances to register.</param>
void RegisterMappers(IEnumerable<IDocumentMapper> mappers);
/// <summary>
/// Gets schema chain entries for the specified root page.
/// Gets schema chain entries for the specified root page.
/// </summary>
/// <param name="rootPageId">The schema root page identifier.</param>
List<BsonSchema> GetSchemas(uint rootPageId);
/// <summary>
/// Appends a schema to the specified schema chain.
/// Appends a schema to the specified schema chain.
/// </summary>
/// <param name="rootPageId">The schema root page identifier.</param>
/// <param name="schema">The schema to append.</param>
uint AppendSchema(uint rootPageId, BsonSchema schema);
/// <summary>
/// Gets the key-to-token mapping.
/// Gets the key-to-token mapping.
/// </summary>
ConcurrentDictionary<string, ushort> GetKeyMap();
/// <summary>
/// Gets the token-to-key mapping.
/// Gets the token-to-key mapping.
/// </summary>
ConcurrentDictionary<ushort, string> GetKeyReverseMap();
/// <summary>
/// Gets or creates a dictionary token for the specified key.
/// Gets or creates a dictionary token for the specified key.
/// </summary>
/// <param name="key">The key value.</param>
ushort GetOrAddDictionaryEntry(string key);
/// <summary>
/// Registers key values in the dictionary mapping.
/// Registers key values in the dictionary mapping.
/// </summary>
/// <param name="keys">The keys to register.</param>
void RegisterKeys(IEnumerable<string> keys);
}
}

782
src/CBDD.Core/Storage/PageFile.cs
View File

File diff suppressed because it is too large Load Diff

76
src/CBDD.Core/Storage/PageHeader.cs
View File

@@ -3,57 +3,45 @@ using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Represents a page header in the database file.
/// Fixed 32-byte structure at the start of each page.
/// Implemented as struct for efficient memory layout.
/// Represents a page header in the database file.
/// Fixed 32-byte structure at the start of each page.
/// Implemented as struct for efficient memory layout.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 32)]
public struct PageHeader
{
/// <summary>Page ID (offset in pages from start of file)</summary>
[FieldOffset(0)]
public uint PageId;
[FieldOffset(0)] public uint PageId;
/// <summary>Type of this page</summary>
[FieldOffset(4)]
public PageType PageType;
[FieldOffset(4)] public PageType PageType;
/// <summary>Number of free bytes in this page</summary>
[FieldOffset(5)]
public ushort FreeBytes;
[FieldOffset(5)] public ushort FreeBytes;
/// <summary>ID of next page in linked list (0 if none). For Page 0 (Header), this points to the First Free Page.</summary>
[FieldOffset(7)]
public uint NextPageId;
[FieldOffset(7)] public uint NextPageId;
/// <summary>Transaction ID that last modified this page</summary>
[FieldOffset(11)]
public ulong TransactionId;
[FieldOffset(11)] public ulong TransactionId;
/// <summary>Checksum for data integrity (CRC32)</summary>
[FieldOffset(19)]
public uint Checksum;
[FieldOffset(19)] public uint Checksum;
/// <summary>Dictionary Root Page ID (Only used in Page 0 / File Header)</summary>
[FieldOffset(23)]
public uint DictionaryRootPageId;
[FieldOffset(23)] public uint DictionaryRootPageId;
[FieldOffset(27)]
private byte _reserved5;
[FieldOffset(28)]
private byte _reserved6;
[FieldOffset(29)]
private byte _reserved7;
[FieldOffset(30)]
private byte _reserved8;
[FieldOffset(31)]
private byte _reserved9;
[FieldOffset(27)] private byte _reserved5;
[FieldOffset(28)] private byte _reserved6;
[FieldOffset(29)] private byte _reserved7;
[FieldOffset(30)] private byte _reserved8;
[FieldOffset(31)] private byte _reserved9;
/// <summary>
/// Writes the header to a span
/// </summary>
/// <param name="destination">The destination span that receives the serialized header.</param>
public readonly void WriteTo(Span<byte> destination)
/// <summary>
/// Writes the header to a span
/// </summary>
/// <param name="destination">The destination span that receives the serialized header.</param>
public readonly void WriteTo(Span<byte> destination)
{
if (destination.Length < 32)
throw new ArgumentException("Destination must be at least 32 bytes");
@@ -61,15 +49,15 @@ public struct PageHeader
MemoryMarshal.Write(destination, in this);
}
/// <summary>
/// Reads a header from a span
/// </summary>
/// <param name="source">The source span containing a serialized header.</param>
public static PageHeader ReadFrom(ReadOnlySpan<byte> source)
/// <summary>
/// Reads a header from a span
/// </summary>
/// <param name="source">The source span containing a serialized header.</param>
public static PageHeader ReadFrom(ReadOnlySpan<byte> source)
{
if (source.Length < 32)
throw new ArgumentException("Source must be at least 32 bytes");
return MemoryMarshal.Read<PageHeader>(source);
}
}
}

28
src/CBDD.Core/Storage/PageType.cs
View File

@@ -1,28 +1,28 @@
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page types in the database file
/// Page types in the database file
/// </summary>
public enum PageType : byte
{
/// <summary>Empty/free page</summary>
Empty = 0,
Empty = 0,
/// <summary>File header page (page 0)</summary>
Header = 1,
Header = 1,
/// <summary>Collection metadata page</summary>
Collection = 2,
Collection = 2,
/// <summary>Data page containing documents</summary>
Data = 3,
Data = 3,
/// <summary>Index B+Tree node page</summary>
Index = 4,
Index = 4,
/// <summary>Free page list</summary>
FreeList = 5,
FreeList = 5,
/// <summary>Overflow page for large documents</summary>
Overflow = 6,
@@ -40,4 +40,4 @@ public enum PageType : byte
/// <summary>GEO Spatial index page</summary>
Spatial = 11
}
}

191
src/CBDD.Core/Storage/SlottedPage.cs
View File

@@ -1,50 +1,43 @@
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Header for slotted pages supporting multiple variable-size documents per page.
/// Fixed 24-byte structure at start of each data page.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 24)]
using System.Buffers.Binary;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Header for slotted pages supporting multiple variable-size documents per page.
/// Fixed 24-byte structure at start of each data page.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 24)]
public struct SlottedPageHeader
{
/// <summary>Page ID</summary>
[FieldOffset(0)]
public uint PageId;
/// <summary>Page ID</summary>
[FieldOffset(0)] public uint PageId;
/// <summary>Type of page (Data, Overflow, Index, Metadata)</summary>
[FieldOffset(4)]
public PageType PageType;
/// <summary>Type of page (Data, Overflow, Index, Metadata)</summary>
[FieldOffset(4)] public PageType PageType;
/// <summary>Number of slot entries in this page</summary>
[FieldOffset(8)]
public ushort SlotCount;
/// <summary>Number of slot entries in this page</summary>
[FieldOffset(8)] public ushort SlotCount;
/// <summary>Offset where free space starts (grows down with slots)</summary>
[FieldOffset(10)]
public ushort FreeSpaceStart;
/// <summary>Offset where free space starts (grows down with slots)</summary>
[FieldOffset(10)] public ushort FreeSpaceStart;
/// <summary>Offset where free space ends (grows up with data)</summary>
[FieldOffset(12)]
public ushort FreeSpaceEnd;
/// <summary>Offset where free space ends (grows up with data)</summary>
[FieldOffset(12)] public ushort FreeSpaceEnd;
/// <summary>Next overflow page ID (0 if none)</summary>
[FieldOffset(14)]
public uint NextOverflowPage;
/// <summary>Next overflow page ID (0 if none)</summary>
[FieldOffset(14)] public uint NextOverflowPage;
/// <summary>Transaction ID that last modified this page</summary>
[FieldOffset(18)]
public uint TransactionId;
/// <summary>Transaction ID that last modified this page</summary>
[FieldOffset(18)] public uint TransactionId;
/// <summary>Reserved for future use</summary>
[FieldOffset(22)]
public ushort Reserved;
/// <summary>Reserved for future use</summary>
[FieldOffset(22)] public ushort Reserved;
public const int Size = 24;
/// <summary>
/// Initializes a header with the current slotted-page format marker.
/// Initializes a header with the current slotted-page format marker.
/// </summary>
public SlottedPageHeader()
{
@@ -52,13 +45,13 @@ public struct SlottedPageHeader
Reserved = StorageFormatConstants.SlottedPageFormatMarker;
}
/// <summary>
/// Gets available free space in bytes
/// </summary>
/// <summary>
/// Gets available free space in bytes
/// </summary>
public readonly int AvailableFreeSpace => FreeSpaceEnd - FreeSpaceStart;
/// <summary>
/// Writes header to span
/// Writes header to span
/// </summary>
/// <param name="destination">The destination span that receives the serialized header.</param>
public readonly void WriteTo(Span<byte> destination)
@@ -66,11 +59,11 @@ public struct SlottedPageHeader
if (destination.Length < Size)
throw new ArgumentException($"Destination must be at least {Size} bytes");
MemoryMarshal.Write(destination, in this);
MemoryMarshal.Write(destination, in this);
}
/// <summary>
/// Reads header from span
/// Reads header from span
/// </summary>
/// <param name="source">The source span containing the serialized header.</param>
public static SlottedPageHeader ReadFrom(ReadOnlySpan<byte> source)
@@ -78,33 +71,30 @@ public struct SlottedPageHeader
if (source.Length < Size)
throw new ArgumentException($"Source must be at least {Size} bytes");
return MemoryMarshal.Read<SlottedPageHeader>(source);
}
}
/// <summary>
/// Slot entry pointing to a document within a page.
/// Fixed 8-byte structure in slot array.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 8)]
public struct SlotEntry
{
/// <summary>Offset to document data within page</summary>
[FieldOffset(0)]
public ushort Offset;
return MemoryMarshal.Read<SlottedPageHeader>(source);
}
}
/// <summary>Length of document data in bytes</summary>
[FieldOffset(2)]
public ushort Length;
/// <summary>
/// Slot entry pointing to a document within a page.
/// Fixed 8-byte structure in slot array.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 8)]
public struct SlotEntry
{
/// <summary>Offset to document data within page</summary>
[FieldOffset(0)] public ushort Offset;
/// <summary>Length of document data in bytes</summary>
[FieldOffset(2)] public ushort Length;
/// <summary>Slot flags (deleted, overflow, etc.)</summary>
[FieldOffset(4)] public SlotFlags Flags;
/// <summary>Slot flags (deleted, overflow, etc.)</summary>
[FieldOffset(4)]
public SlotFlags Flags;
public const int Size = 8;
/// <summary>
/// Writes slot entry to span
/// Writes slot entry to span
/// </summary>
/// <param name="destination">The destination span that receives the serialized slot entry.</param>
public readonly void WriteTo(Span<byte> destination)
@@ -112,11 +102,11 @@ public struct SlotEntry
if (destination.Length < Size)
throw new ArgumentException($"Destination must be at least {Size} bytes");
MemoryMarshal.Write(destination, in this);
MemoryMarshal.Write(destination, in this);
}
/// <summary>
/// Reads slot entry from span
/// Reads slot entry from span
/// </summary>
/// <param name="source">The source span containing the serialized slot entry.</param>
public static SlotEntry ReadFrom(ReadOnlySpan<byte> source)
@@ -124,46 +114,47 @@ public struct SlotEntry
if (source.Length < Size)
throw new ArgumentException($"Source must be at least {Size} bytes");
return MemoryMarshal.Read<SlotEntry>(source);
}
}
/// <summary>
/// Flags for slot entries
/// </summary>
[Flags]
public enum SlotFlags : uint
{
/// <summary>Slot is active and contains data</summary>
return MemoryMarshal.Read<SlotEntry>(source);
}
}
/// <summary>
/// Flags for slot entries
/// </summary>
[Flags]
public enum SlotFlags : uint
{
/// <summary>Slot is active and contains data</summary>
None = 0,
/// <summary>Slot is marked as deleted (can be reused)</summary>
/// <summary>Slot is marked as deleted (can be reused)</summary>
Deleted = 1 << 0,
/// <summary>Document continues in overflow pages</summary>
/// <summary>Document continues in overflow pages</summary>
HasOverflow = 1 << 1,
/// <summary>Document data is compressed</summary>
Compressed = 1 << 2,
}
/// <summary>
/// Location of a document within the database.
/// Maps ObjectId to specific page and slot.
/// </summary>
/// <summary>Document data is compressed</summary>
Compressed = 1 << 2
}
/// <summary>
/// Location of a document within the database.
/// Maps ObjectId to specific page and slot.
/// </summary>
public readonly struct DocumentLocation
{
/// <summary>
/// Gets the page identifier containing the document.
/// Gets the page identifier containing the document.
/// </summary>
public uint PageId { get; init; }
/// <summary>
/// Gets the slot index within the page.
/// Gets the slot index within the page.
/// </summary>
public ushort SlotIndex { get; init; }
/// <summary>
/// Initializes a new instance of the <see cref="DocumentLocation"/> struct.
/// Initializes a new instance of the <see cref="DocumentLocation" /> struct.
/// </summary>
/// <param name="pageId">The page identifier containing the document.</param>
/// <param name="slotIndex">The slot index within the page.</param>
@@ -174,7 +165,7 @@ public readonly struct DocumentLocation
}
/// <summary>
/// Serializes DocumentLocation to a byte span (6 bytes: 4 for PageId + 2 for SlotIndex)
/// Serializes DocumentLocation to a byte span (6 bytes: 4 for PageId + 2 for SlotIndex)
/// </summary>
/// <param name="destination">The destination span that receives the serialized value.</param>
public void WriteTo(Span<byte> destination)
@@ -182,12 +173,12 @@ public readonly struct DocumentLocation
if (destination.Length < 6)
throw new ArgumentException("Destination must be at least 6 bytes", nameof(destination));
System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(destination, PageId);
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(destination.Slice(4), SlotIndex);
BinaryPrimitives.WriteUInt32LittleEndian(destination, PageId);
BinaryPrimitives.WriteUInt16LittleEndian(destination.Slice(4), SlotIndex);
}
/// <summary>
/// Deserializes DocumentLocation from a byte span (6 bytes)
/// Deserializes DocumentLocation from a byte span (6 bytes)
/// </summary>
/// <param name="source">The source span containing the serialized value.</param>
public static DocumentLocation ReadFrom(ReadOnlySpan<byte> source)
@@ -195,14 +186,14 @@ public readonly struct DocumentLocation
if (source.Length < 6)
throw new ArgumentException("Source must be at least 6 bytes", nameof(source));
var pageId = System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(source);
var slotIndex = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(source.Slice(4));
uint pageId = BinaryPrimitives.ReadUInt32LittleEndian(source);
ushort slotIndex = BinaryPrimitives.ReadUInt16LittleEndian(source.Slice(4));
return new DocumentLocation(pageId, slotIndex);
return new DocumentLocation(pageId, slotIndex);
}
/// <summary>
/// Size in bytes when serialized
/// </summary>
public const int SerializedSize = 6;
}
/// <summary>
/// Size in bytes when serialized
/// </summary>
public const int SerializedSize = 6;
}

193
src/CBDD.Core/Storage/SpatialPage.cs
View File

@@ -1,12 +1,11 @@
using System.Buffers.Binary;
using System.Runtime.InteropServices;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page for storing R-Tree nodes for Geospatial Indexing.
/// Page for storing R-Tree nodes for Geospatial Indexing.
/// </summary>
internal struct SpatialPage
{
@@ -29,14 +28,14 @@ internal struct SpatialPage
public const int EntrySize = 38; // 32 (GeoBox) + 6 (Pointer)
/// <summary>
/// Initializes a spatial page.
/// </summary>
/// <param name="page">The page buffer to initialize.</param>
/// <param name="pageId">The page identifier.</param>
/// <param name="isLeaf">Whether this page is a leaf node.</param>
/// <param name="level">The tree level for this page.</param>
public static void Initialize(Span<byte> page, uint pageId, bool isLeaf, byte level)
/// <summary>
/// Initializes a spatial page.
/// </summary>
/// <param name="page">The page buffer to initialize.</param>
/// <param name="pageId">The page identifier.</param>
/// <param name="isLeaf">Whether this page is a leaf node.</param>
/// <param name="level">The tree level for this page.</param>
public static void Initialize(Span<byte> page, uint pageId, bool isLeaf, byte level)
{
var header = new PageHeader
{
@@ -54,65 +53,86 @@ internal struct SpatialPage
BinaryPrimitives.WriteUInt32LittleEndian(page.Slice(ParentPageIdOffset), 0);
}
/// <summary>
/// Gets a value indicating whether the page is a leaf node.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns><see langword="true"/> if the page is a leaf node; otherwise, <see langword="false"/>.</returns>
public static bool GetIsLeaf(ReadOnlySpan<byte> page) => page[IsLeafOffset] == 1;
/// <summary>
/// Gets the tree level stored in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The level value.</returns>
public static byte GetLevel(ReadOnlySpan<byte> page) => page[LevelOffset];
/// <summary>
/// Gets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The number of entries.</returns>
public static ushort GetEntryCount(ReadOnlySpan<byte> page) => BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(EntryCountOffset));
/// <summary>
/// Sets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="count">The entry count to set.</param>
public static void SetEntryCount(Span<byte> page, ushort count) => BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(EntryCountOffset), count);
/// <summary>
/// Gets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The parent page identifier.</returns>
public static uint GetParentPageId(ReadOnlySpan<byte> page) => BinaryPrimitives.ReadUInt32LittleEndian(page.Slice(ParentPageIdOffset));
/// <summary>
/// Sets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="parentId">The parent page identifier.</param>
public static void SetParentPageId(Span<byte> page, uint parentId) => BinaryPrimitives.WriteUInt32LittleEndian(page.Slice(ParentPageIdOffset), parentId);
/// <summary>
/// Gets the maximum number of entries that can fit in a page.
/// </summary>
/// <param name="pageSize">The page size in bytes.</param>
/// <returns>The maximum number of entries.</returns>
public static int GetMaxEntries(int pageSize) => (pageSize - DataOffset) / EntrySize;
/// <summary>
/// Writes an entry at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="index">The entry index.</param>
/// <param name="mbr">The minimum bounding rectangle for the entry.</param>
/// <param name="pointer">The document location pointer.</param>
public static void WriteEntry(Span<byte> page, int index, GeoBox mbr, DocumentLocation pointer)
/// <summary>
/// Gets a value indicating whether the page is a leaf node.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns><see langword="true" /> if the page is a leaf node; otherwise, <see langword="false" />.</returns>
public static bool GetIsLeaf(ReadOnlySpan<byte> page)
{
int offset = DataOffset + (index * EntrySize);
return page[IsLeafOffset] == 1;
}
/// <summary>
/// Gets the tree level stored in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The level value.</returns>
public static byte GetLevel(ReadOnlySpan<byte> page)
{
return page[LevelOffset];
}
/// <summary>
/// Gets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The number of entries.</returns>
public static ushort GetEntryCount(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(EntryCountOffset));
}
/// <summary>
/// Sets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="count">The entry count to set.</param>
public static void SetEntryCount(Span<byte> page, ushort count)
{
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(EntryCountOffset), count);
}
/// <summary>
/// Gets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The parent page identifier.</returns>
public static uint GetParentPageId(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadUInt32LittleEndian(page.Slice(ParentPageIdOffset));
}
/// <summary>
/// Sets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="parentId">The parent page identifier.</param>
public static void SetParentPageId(Span<byte> page, uint parentId)
{
BinaryPrimitives.WriteUInt32LittleEndian(page.Slice(ParentPageIdOffset), parentId);
}
/// <summary>
/// Gets the maximum number of entries that can fit in a page.
/// </summary>
/// <param name="pageSize">The page size in bytes.</param>
/// <returns>The maximum number of entries.</returns>
public static int GetMaxEntries(int pageSize)
{
return (pageSize - DataOffset) / EntrySize;
}
/// <summary>
/// Writes an entry at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="index">The entry index.</param>
/// <param name="mbr">The minimum bounding rectangle for the entry.</param>
/// <param name="pointer">The document location pointer.</param>
public static void WriteEntry(Span<byte> page, int index, GeoBox mbr, DocumentLocation pointer)
{
int offset = DataOffset + index * EntrySize;
var entrySpan = page.Slice(offset, EntrySize);
// Write MBR (4 doubles)
@@ -126,16 +146,16 @@ internal struct SpatialPage
pointer.WriteTo(entrySpan.Slice(32, 6));
}
/// <summary>
/// Reads an entry at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="index">The entry index.</param>
/// <param name="mbr">When this method returns, contains the entry MBR.</param>
/// <param name="pointer">When this method returns, contains the entry document location.</param>
public static void ReadEntry(ReadOnlySpan<byte> page, int index, out GeoBox mbr, out DocumentLocation pointer)
/// <summary>
/// Reads an entry at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="index">The entry index.</param>
/// <param name="mbr">When this method returns, contains the entry MBR.</param>
/// <param name="pointer">When this method returns, contains the entry document location.</param>
public static void ReadEntry(ReadOnlySpan<byte> page, int index, out GeoBox mbr, out DocumentLocation pointer)
{
int offset = DataOffset + (index * EntrySize);
int offset = DataOffset + index * EntrySize;
var entrySpan = page.Slice(offset, EntrySize);
var doubles = MemoryMarshal.Cast<byte, double>(entrySpan.Slice(0, 32));
@@ -143,23 +163,24 @@ internal struct SpatialPage
pointer = DocumentLocation.ReadFrom(entrySpan.Slice(32, 6));
}
/// <summary>
/// Calculates the combined MBR of all entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The combined MBR, or <see cref="GeoBox.Empty"/> when the page has no entries.</returns>
public static GeoBox CalculateMBR(ReadOnlySpan<byte> page)
/// <summary>
/// Calculates the combined MBR of all entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The combined MBR, or <see cref="GeoBox.Empty" /> when the page has no entries.</returns>
public static GeoBox CalculateMBR(ReadOnlySpan<byte> page)
{
ushort count = GetEntryCount(page);
if (count == 0) return GeoBox.Empty;
GeoBox result = GeoBox.Empty;
for (int i = 0; i < count; i++)
var result = GeoBox.Empty;
for (var i = 0; i < count; i++)
{
ReadEntry(page, i, out var mbr, out _);
if (i == 0) result = mbr;
else result = result.ExpandTo(mbr);
}
return result;
}
}
}

298
src/CBDD.Core/Storage/StorageEngine.Collections.cs
View File

@@ -1,31 +1,27 @@
using System;
using System.Collections.Generic;
using System.IO;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Collections;
namespace ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public class CollectionMetadata
{
/// <summary>
/// Gets or sets the collection name.
/// Gets or sets the collection name.
/// </summary>
public string Name { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the root page identifier of the primary index.
/// Gets or sets the root page identifier of the primary index.
/// </summary>
public uint PrimaryRootPageId { get; set; }
/// <summary>
/// Gets or sets the root page identifier of the schema chain.
/// Gets or sets the root page identifier of the schema chain.
/// </summary>
public uint SchemaRootPageId { get; set; }
/// <summary>
/// Gets the collection index metadata list.
/// Gets the collection index metadata list.
/// </summary>
public List<IndexMetadata> Indexes { get; } = new();
}
@@ -33,45 +29,45 @@ public class CollectionMetadata
public class IndexMetadata
{
/// <summary>
/// Gets or sets the index name.
/// Gets or sets the index name.
/// </summary>
public string Name { get; set; } = string.Empty;
/// <summary>
/// Gets or sets a value indicating whether this index enforces uniqueness.
/// Gets or sets a value indicating whether this index enforces uniqueness.
/// </summary>
public bool IsUnique { get; set; }
/// <summary>
/// Gets or sets the index type.
/// Gets or sets the index type.
/// </summary>
public IndexType Type { get; set; }
/// <summary>
/// Gets or sets indexed property paths.
/// Gets or sets indexed property paths.
/// </summary>
public string[] PropertyPaths { get; set; } = Array.Empty<string>();
/// <summary>
/// Gets or sets vector dimensions for vector indexes.
/// Gets or sets vector dimensions for vector indexes.
/// </summary>
public int Dimensions { get; set; }
/// <summary>
/// Gets or sets the vector similarity metric for vector indexes.
/// Gets or sets the vector similarity metric for vector indexes.
/// </summary>
public VectorMetric Metric { get; set; }
/// <summary>
/// Gets or sets the root page identifier of the index structure.
/// Gets or sets the root page identifier of the index structure.
/// </summary>
public uint RootPageId { get; set; }
}
public sealed partial class StorageEngine
{
public sealed partial class StorageEngine
{
/// <summary>
/// Gets collection metadata by name.
/// Gets collection metadata by name.
/// </summary>
/// <param name="name">The collection name.</param>
/// <returns>The collection metadata if found; otherwise, null.</returns>
@@ -82,7 +78,121 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Returns all collection metadata entries currently registered in page 1.
/// Saves collection metadata to the metadata page.
/// </summary>
/// <param name="metadata">The metadata to save.</param>
public void SaveCollectionMetadata(CollectionMetadata metadata)
{
using var stream = new MemoryStream();
using var writer = new BinaryWriter(stream);
writer.Write(metadata.Name);
writer.Write(metadata.PrimaryRootPageId);
writer.Write(metadata.SchemaRootPageId);
writer.Write(metadata.Indexes.Count);
foreach (var idx in metadata.Indexes)
{
writer.Write(idx.Name);
writer.Write(idx.IsUnique);
writer.Write((byte)idx.Type);
writer.Write(idx.RootPageId);
writer.Write(idx.PropertyPaths.Length);
foreach (string path in idx.PropertyPaths) writer.Write(path);
if (idx.Type == IndexType.Vector)
{
writer.Write(idx.Dimensions);
writer.Write((byte)idx.Metric);
}
}
byte[] newData = stream.ToArray();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
int existingSlotIndex = -1;
for (ushort i = 0; i < header.SlotCount; i++)
{
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0) continue;
try
{
using var ms = new MemoryStream(buffer, slot.Offset, slot.Length, false);
using var reader = new BinaryReader(ms);
string name = reader.ReadString();
if (string.Equals(name, metadata.Name, StringComparison.OrdinalIgnoreCase))
{
existingSlotIndex = i;
break;
}
}
catch
{
}
}
if (existingSlotIndex >= 0)
{
int slotOffset = SlottedPageHeader.Size + existingSlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
slot.Flags |= SlotFlags.Deleted;
slot.WriteTo(buffer.AsSpan(slotOffset));
}
if (header.AvailableFreeSpace < newData.Length + SlotEntry.Size)
// Compact logic omitted as per current architecture
throw new InvalidOperationException(
"Not enough space in Metadata Page (Page 1) to save collection metadata.");
int docOffset = header.FreeSpaceEnd - newData.Length;
newData.CopyTo(buffer.AsSpan(docOffset));
ushort slotIndex;
if (existingSlotIndex >= 0)
{
slotIndex = (ushort)existingSlotIndex;
}
else
{
slotIndex = header.SlotCount;
header.SlotCount++;
}
int newSlotEntryOffset = SlottedPageHeader.Size + slotIndex * SlotEntry.Size;
var newSlot = new SlotEntry
{
Offset = (ushort)docOffset,
Length = (ushort)newData.Length,
Flags = SlotFlags.None
};
newSlot.WriteTo(buffer.AsSpan(newSlotEntryOffset));
header.FreeSpaceEnd = (ushort)docOffset;
if (existingSlotIndex == -1)
header.FreeSpaceStart = (ushort)(SlottedPageHeader.Size + header.SlotCount * SlotEntry.Size);
header.WriteTo(buffer);
WritePageImmediate(1, buffer);
}
/// <summary>
/// Registers all BSON keys used by a set of mappers into the global dictionary.
/// </summary>
/// <param name="mappers">The mappers whose keys should be registered.</param>
public void RegisterMappers(IEnumerable<IDocumentMapper> mappers)
{
var allKeys = mappers.SelectMany(m => m.UsedKeys).Distinct();
RegisterKeys(allKeys);
}
/// <summary>
/// Returns all collection metadata entries currently registered in page 1.
/// </summary>
public IReadOnlyList<CollectionMetadata> GetAllCollectionMetadata()
{
@@ -96,7 +206,7 @@ public sealed partial class StorageEngine
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0)
continue;
@@ -104,122 +214,12 @@ public sealed partial class StorageEngine
if (slot.Offset < SlottedPageHeader.Size || slot.Offset + slot.Length > buffer.Length)
continue;
if (TryDeserializeCollectionMetadata(buffer.AsSpan(slot.Offset, slot.Length), out var metadata) && metadata != null)
{
result.Add(metadata);
}
if (TryDeserializeCollectionMetadata(buffer.AsSpan(slot.Offset, slot.Length), out var metadata) &&
metadata != null) result.Add(metadata);
}
return result;
}
/// <summary>
/// Saves collection metadata to the metadata page.
/// </summary>
/// <param name="metadata">The metadata to save.</param>
public void SaveCollectionMetadata(CollectionMetadata metadata)
{
using var stream = new MemoryStream();
using var writer = new BinaryWriter(stream);
writer.Write(metadata.Name);
writer.Write(metadata.PrimaryRootPageId);
writer.Write(metadata.SchemaRootPageId);
writer.Write(metadata.Indexes.Count);
foreach (var idx in metadata.Indexes)
{
writer.Write(idx.Name);
writer.Write(idx.IsUnique);
writer.Write((byte)idx.Type);
writer.Write(idx.RootPageId);
writer.Write(idx.PropertyPaths.Length);
foreach (var path in idx.PropertyPaths)
{
writer.Write(path);
}
if (idx.Type == IndexType.Vector)
{
writer.Write(idx.Dimensions);
writer.Write((byte)idx.Metric);
}
}
var newData = stream.ToArray();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
int existingSlotIndex = -1;
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0) continue;
try
{
using var ms = new MemoryStream(buffer, slot.Offset, slot.Length, false);
using var reader = new BinaryReader(ms);
var name = reader.ReadString();
if (string.Equals(name, metadata.Name, StringComparison.OrdinalIgnoreCase))
{
existingSlotIndex = i;
break;
}
}
catch { }
}
if (existingSlotIndex >= 0)
{
var slotOffset = SlottedPageHeader.Size + (existingSlotIndex * SlotEntry.Size);
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
slot.Flags |= SlotFlags.Deleted;
slot.WriteTo(buffer.AsSpan(slotOffset));
}
if (header.AvailableFreeSpace < newData.Length + SlotEntry.Size)
{
// Compact logic omitted as per current architecture
throw new InvalidOperationException("Not enough space in Metadata Page (Page 1) to save collection metadata.");
}
int docOffset = header.FreeSpaceEnd - newData.Length;
newData.CopyTo(buffer.AsSpan(docOffset));
ushort slotIndex;
if (existingSlotIndex >= 0)
{
slotIndex = (ushort)existingSlotIndex;
}
else
{
slotIndex = header.SlotCount;
header.SlotCount++;
}
var newSlotEntryOffset = SlottedPageHeader.Size + (slotIndex * SlotEntry.Size);
var newSlot = new SlotEntry
{
Offset = (ushort)docOffset,
Length = (ushort)newData.Length,
Flags = SlotFlags.None
};
newSlot.WriteTo(buffer.AsSpan(newSlotEntryOffset));
header.FreeSpaceEnd = (ushort)docOffset;
if (existingSlotIndex == -1)
{
header.FreeSpaceStart = (ushort)(SlottedPageHeader.Size + (header.SlotCount * SlotEntry.Size));
}
header.WriteTo(buffer);
WritePageImmediate(1, buffer);
}
private static bool TryDeserializeCollectionMetadata(ReadOnlySpan<byte> rawBytes, out CollectionMetadata? metadata)
{
@@ -230,16 +230,16 @@ public sealed partial class StorageEngine
using var ms = new MemoryStream(rawBytes.ToArray());
using var reader = new BinaryReader(ms);
var collName = reader.ReadString();
string collName = reader.ReadString();
var parsed = new CollectionMetadata { Name = collName };
parsed.PrimaryRootPageId = reader.ReadUInt32();
parsed.SchemaRootPageId = reader.ReadUInt32();
var indexCount = reader.ReadInt32();
int indexCount = reader.ReadInt32();
if (indexCount < 0)
return false;
for (int j = 0; j < indexCount; j++)
for (var j = 0; j < indexCount; j++)
{
var idx = new IndexMetadata
{
@@ -249,12 +249,12 @@ public sealed partial class StorageEngine
RootPageId = reader.ReadUInt32()
};
var pathCount = reader.ReadInt32();
int pathCount = reader.ReadInt32();
if (pathCount < 0)
return false;
idx.PropertyPaths = new string[pathCount];
for (int k = 0; k < pathCount; k++)
for (var k = 0; k < pathCount; k++)
idx.PropertyPaths[k] = reader.ReadString();
if (idx.Type == IndexType.Vector)
@@ -274,14 +274,4 @@ public sealed partial class StorageEngine
return false;
}
}
/// <summary>
/// Registers all BSON keys used by a set of mappers into the global dictionary.
/// </summary>
/// <param name="mappers">The mappers whose keys should be registered.</param>
public void RegisterMappers(IEnumerable<IDocumentMapper> mappers)
{
var allKeys = mappers.SelectMany(m => m.UsedKeys).Distinct();
RegisterKeys(allKeys);
}
}
}

157
src/CBDD.Core/Storage/StorageEngine.Diagnostics.cs
View File

@@ -5,172 +5,173 @@ using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Aggregated page counts grouped by page type.
/// Aggregated page counts grouped by page type.
/// </summary>
public sealed class PageTypeUsageEntry
{
/// <summary>
/// Gets the page type.
/// Gets the page type.
/// </summary>
public PageType PageType { get; init; }
/// <summary>
/// Gets the number of pages of this type.
/// Gets the number of pages of this type.
/// </summary>
public int PageCount { get; init; }
}
/// <summary>
/// Per-collection page usage summary.
/// Per-collection page usage summary.
/// </summary>
public sealed class CollectionPageUsageEntry
{
/// <summary>
/// Gets the collection name.
/// Gets the collection name.
/// </summary>
public string CollectionName { get; init; } = string.Empty;
/// <summary>
/// Gets the total number of distinct pages referenced by the collection.
/// Gets the total number of distinct pages referenced by the collection.
/// </summary>
public int TotalDistinctPages { get; init; }
/// <summary>
/// Gets the number of data pages.
/// Gets the number of data pages.
/// </summary>
public int DataPages { get; init; }
/// <summary>
/// Gets the number of overflow pages.
/// Gets the number of overflow pages.
/// </summary>
public int OverflowPages { get; init; }
/// <summary>
/// Gets the number of index pages.
/// Gets the number of index pages.
/// </summary>
public int IndexPages { get; init; }
/// <summary>
/// Gets the number of other page types.
/// Gets the number of other page types.
/// </summary>
public int OtherPages { get; init; }
}
/// <summary>
/// Per-collection compression ratio summary.
/// Per-collection compression ratio summary.
/// </summary>
public sealed class CollectionCompressionRatioEntry
{
/// <summary>
/// Gets the collection name.
/// Gets the collection name.
/// </summary>
public string CollectionName { get; init; } = string.Empty;
/// <summary>
/// Gets the number of documents.
/// Gets the number of documents.
/// </summary>
public long DocumentCount { get; init; }
/// <summary>
/// Gets the number of compressed documents.
/// Gets the number of compressed documents.
/// </summary>
public long CompressedDocumentCount { get; init; }
/// <summary>
/// Gets the total uncompressed byte count.
/// Gets the total uncompressed byte count.
/// </summary>
public long BytesBeforeCompression { get; init; }
/// <summary>
/// Gets the total stored byte count.
/// Gets the total stored byte count.
/// </summary>
public long BytesAfterCompression { get; init; }
/// <summary>
/// Gets the compression ratio.
/// Gets the compression ratio.
/// </summary>
public double CompressionRatio => BytesAfterCompression <= 0 ? 1.0 : (double)BytesBeforeCompression / BytesAfterCompression;
public double CompressionRatio =>
BytesAfterCompression <= 0 ? 1.0 : (double)BytesBeforeCompression / BytesAfterCompression;
}
/// <summary>
/// Summary of free-list and reclaimable tail information.
/// Summary of free-list and reclaimable tail information.
/// </summary>
public sealed class FreeListSummary
{
/// <summary>
/// Gets the total page count.
/// Gets the total page count.
/// </summary>
public uint PageCount { get; init; }
/// <summary>
/// Gets the free page count.
/// Gets the free page count.
/// </summary>
public int FreePageCount { get; init; }
/// <summary>
/// Gets the total free bytes.
/// Gets the total free bytes.
/// </summary>
public long FreeBytes { get; init; }
/// <summary>
/// Gets the fragmentation percentage.
/// Gets the fragmentation percentage.
/// </summary>
public double FragmentationPercent { get; init; }
/// <summary>
/// Gets the number of reclaimable pages at the file tail.
/// Gets the number of reclaimable pages at the file tail.
/// </summary>
public uint TailReclaimablePages { get; init; }
}
/// <summary>
/// Single page entry in fragmentation reporting.
/// Single page entry in fragmentation reporting.
/// </summary>
public sealed class FragmentationPageEntry
{
/// <summary>
/// Gets the page identifier.
/// Gets the page identifier.
/// </summary>
public uint PageId { get; init; }
/// <summary>
/// Gets the page type.
/// Gets the page type.
/// </summary>
public PageType PageType { get; init; }
/// <summary>
/// Gets a value indicating whether this page is free.
/// Gets a value indicating whether this page is free.
/// </summary>
public bool IsFreePage { get; init; }
/// <summary>
/// Gets the free bytes on the page.
/// Gets the free bytes on the page.
/// </summary>
public int FreeBytes { get; init; }
}
/// <summary>
/// Detailed fragmentation map and totals.
/// Detailed fragmentation map and totals.
/// </summary>
public sealed class FragmentationMapReport
{
/// <summary>
/// Gets the page entries.
/// Gets the page entries.
/// </summary>
public IReadOnlyList<FragmentationPageEntry> Pages { get; init; } = Array.Empty<FragmentationPageEntry>();
/// <summary>
/// Gets the total free bytes across all pages.
/// Gets the total free bytes across all pages.
/// </summary>
public long TotalFreeBytes { get; init; }
/// <summary>
/// Gets the fragmentation percentage.
/// Gets the fragmentation percentage.
/// </summary>
public double FragmentationPercent { get; init; }
/// <summary>
/// Gets the number of reclaimable pages at the file tail.
/// Gets the number of reclaimable pages at the file tail.
/// </summary>
public uint TailReclaimablePages { get; init; }
}
@@ -178,11 +179,11 @@ public sealed class FragmentationMapReport
public sealed partial class StorageEngine
{
/// <summary>
/// Gets page usage grouped by page type.
/// Gets page usage grouped by page type.
/// </summary>
public IReadOnlyList<PageTypeUsageEntry> GetPageUsageByPageType()
{
var pageCount = _pageFile.NextPageId;
uint pageCount = _pageFile.NextPageId;
var buffer = new byte[_pageFile.PageSize];
var counts = new Dictionary<PageType, int>();
@@ -190,7 +191,7 @@ public sealed partial class StorageEngine
{
_pageFile.ReadPage(pageId, buffer);
var pageType = PageHeader.ReadFrom(buffer).PageType;
counts[pageType] = counts.TryGetValue(pageType, out var count) ? count + 1 : 1;
counts[pageType] = counts.TryGetValue(pageType, out int count) ? count + 1 : 1;
}
return counts
@@ -204,7 +205,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets per-collection page usage by resolving primary-index locations and related index roots.
/// Gets per-collection page usage by resolving primary-index locations and related index roots.
/// </summary>
public IReadOnlyList<CollectionPageUsageEntry> GetPageUsageByCollection()
{
@@ -221,27 +222,23 @@ public sealed partial class StorageEngine
pageIds.Add(metadata.SchemaRootPageId);
foreach (var indexMetadata in metadata.Indexes)
{
if (indexMetadata.RootPageId != 0)
pageIds.Add(indexMetadata.RootPageId);
}
foreach (var location in EnumeratePrimaryLocations(metadata))
{
pageIds.Add(location.PageId);
if (TryReadFirstOverflowPage(location, out var firstOverflowPage))
{
if (TryReadFirstOverflowPage(location, out uint firstOverflowPage))
AddOverflowChainPages(pageIds, firstOverflowPage);
}
}
int data = 0;
int overflow = 0;
int indexPages = 0;
int other = 0;
var data = 0;
var overflow = 0;
var indexPages = 0;
var other = 0;
var pageBuffer = new byte[_pageFile.PageSize];
foreach (var pageId in pageIds)
foreach (uint pageId in pageIds)
{
if (pageId >= _pageFile.NextPageId)
continue;
@@ -250,21 +247,13 @@ public sealed partial class StorageEngine
var pageType = PageHeader.ReadFrom(pageBuffer).PageType;
if (pageType == PageType.Data)
{
data++;
}
else if (pageType == PageType.Overflow)
{
overflow++;
}
else if (pageType == PageType.Index || pageType == PageType.Vector || pageType == PageType.Spatial)
{
indexPages++;
}
else
{
other++;
}
}
results.Add(new CollectionPageUsageEntry
@@ -282,7 +271,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets per-collection logical-vs-stored compression ratios.
/// Gets per-collection logical-vs-stored compression ratios.
/// </summary>
public IReadOnlyList<CollectionCompressionRatioEntry> GetCompressionRatioByCollection()
{
@@ -298,7 +287,8 @@ public sealed partial class StorageEngine
foreach (var location in EnumeratePrimaryLocations(metadata))
{
if (!TryReadSlotPayloadStats(location, out var isCompressed, out var originalBytes, out var storedBytes))
if (!TryReadSlotPayloadStats(location, out bool isCompressed, out int originalBytes,
out int storedBytes))
continue;
docs++;
@@ -323,7 +313,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets free-list summary for diagnostics.
/// Gets free-list summary for diagnostics.
/// </summary>
public FreeListSummary GetFreeListSummary()
{
@@ -339,12 +329,12 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets detailed page-level fragmentation diagnostics.
/// Gets detailed page-level fragmentation diagnostics.
/// </summary>
public FragmentationMapReport GetFragmentationMap()
{
var freePageSet = new HashSet<uint>(_pageFile.EnumerateFreePages(includeEmptyPages: true));
var pageCount = _pageFile.NextPageId;
var freePageSet = new HashSet<uint>(_pageFile.EnumerateFreePages());
uint pageCount = _pageFile.NextPageId;
var buffer = new byte[_pageFile.PageSize];
var pages = new List<FragmentationPageEntry>((int)pageCount);
@@ -354,17 +344,12 @@ public sealed partial class StorageEngine
{
_pageFile.ReadPage(pageId, buffer);
var pageHeader = PageHeader.ReadFrom(buffer);
var isFreePage = freePageSet.Contains(pageId);
bool isFreePage = freePageSet.Contains(pageId);
int freeBytes = 0;
var freeBytes = 0;
if (isFreePage)
{
freeBytes = _pageFile.PageSize;
}
else if (TryReadSlottedFreeSpace(buffer, out var slottedFreeBytes))
{
freeBytes = slottedFreeBytes;
}
else if (TryReadSlottedFreeSpace(buffer, out int slottedFreeBytes)) freeBytes = slottedFreeBytes;
totalFreeBytes += freeBytes;
@@ -378,7 +363,7 @@ public sealed partial class StorageEngine
}
uint tailReclaimablePages = 0;
for (var i = pageCount; i > 2; i--)
for (uint i = pageCount; i > 2; i--)
{
if (!freePageSet.Contains(i - 1))
break;
@@ -386,12 +371,12 @@ public sealed partial class StorageEngine
tailReclaimablePages++;
}
var fileBytes = Math.Max(1L, _pageFile.FileLengthBytes);
long fileBytes = Math.Max(1L, _pageFile.FileLengthBytes);
return new FragmentationMapReport
{
Pages = pages,
TotalFreeBytes = totalFreeBytes,
FragmentationPercent = (totalFreeBytes * 100d) / fileBytes,
FragmentationPercent = totalFreeBytes * 100d / fileBytes,
TailReclaimablePages = tailReclaimablePages
};
}
@@ -403,10 +388,8 @@ public sealed partial class StorageEngine
var index = new BTreeIndex(this, IndexOptions.CreateUnique("_id"), metadata.PrimaryRootPageId);
foreach (var entry in index.Range(IndexKey.MinKey, IndexKey.MaxKey, IndexDirection.Forward, transactionId: 0))
{
foreach (var entry in index.Range(IndexKey.MinKey, IndexKey.MaxKey, IndexDirection.Forward, 0))
yield return entry.Location;
}
}
private bool TryReadFirstOverflowPage(in DocumentLocation location, out uint firstOverflowPage)
@@ -419,7 +402,7 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
@@ -441,7 +424,7 @@ public sealed partial class StorageEngine
var buffer = new byte[_pageFile.PageSize];
var visited = new HashSet<uint>();
var current = firstOverflowPage;
uint current = firstOverflowPage;
while (current != 0 && current < _pageFile.NextPageId && visited.Add(current))
{
@@ -472,12 +455,12 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
var hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
isCompressed = (slot.Flags & SlotFlags.Compressed) != 0;
if (!hasOverflow)
@@ -492,7 +475,8 @@ public sealed partial class StorageEngine
if (slot.Length < CompressedPayloadHeader.Size)
return false;
var compressedHeader = CompressedPayloadHeader.ReadFrom(pageBuffer.AsSpan(slot.Offset, CompressedPayloadHeader.Size));
var compressedHeader =
CompressedPayloadHeader.ReadFrom(pageBuffer.AsSpan(slot.Offset, CompressedPayloadHeader.Size));
originalBytes = compressedHeader.OriginalLength;
return true;
}
@@ -501,7 +485,7 @@ public sealed partial class StorageEngine
return false;
var primaryPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
var totalStoredBytes = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
int totalStoredBytes = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
if (totalStoredBytes < 0)
return false;
@@ -522,8 +506,8 @@ public sealed partial class StorageEngine
else
{
storedPrefix.CopyTo(headerBuffer);
var copied = storedPrefix.Length;
var nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
int copied = storedPrefix.Length;
uint nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
var overflowBuffer = new byte[_pageFile.PageSize];
while (copied < CompressedPayloadHeader.Size && nextOverflow != 0 && nextOverflow < _pageFile.NextPageId)
@@ -533,7 +517,8 @@ public sealed partial class StorageEngine
if (overflowHeader.PageType != PageType.Overflow)
return false;
var available = Math.Min(CompressedPayloadHeader.Size - copied, _pageFile.PageSize - SlottedPageHeader.Size);
int available = Math.Min(CompressedPayloadHeader.Size - copied,
_pageFile.PageSize - SlottedPageHeader.Size);
overflowBuffer.AsSpan(SlottedPageHeader.Size, available).CopyTo(headerBuffer.Slice(copied));
copied += available;
nextOverflow = overflowHeader.NextOverflowPage;
@@ -547,4 +532,4 @@ public sealed partial class StorageEngine
originalBytes = headerFromPayload.OriginalLength;
return true;
}
}
}

297
src/CBDD.Core/Storage/StorageEngine.Dictionary.cs
View File

@@ -1,29 +1,104 @@
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
private readonly ConcurrentDictionary<string, ushort> _dictionaryCache = new(StringComparer.OrdinalIgnoreCase);
private readonly ConcurrentDictionary<ushort, string> _dictionaryReverseCache = new();
private uint _dictionaryRootPageId;
private ushort _nextDictionaryId;
using System.Collections.Concurrent;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
private readonly ConcurrentDictionary<string, ushort> _dictionaryCache = new(StringComparer.OrdinalIgnoreCase);
// Lock for dictionary modifications (simple lock for now, could be RW lock)
private readonly object _dictionaryLock = new();
private void InitializeDictionary()
{
// 1. Read File Header (Page 0) to get Dictionary Root
var headerBuffer = new byte[PageSize];
ReadPage(0, null, headerBuffer);
private readonly object _dictionaryLock = new();
private readonly ConcurrentDictionary<ushort, string> _dictionaryReverseCache = new();
private uint _dictionaryRootPageId;
private ushort _nextDictionaryId;
/// <summary>
/// Gets the key-to-id dictionary cache.
/// </summary>
/// <returns>The key-to-id map.</returns>
public ConcurrentDictionary<string, ushort> GetKeyMap()
{
return _dictionaryCache;
}
/// <summary>
/// Gets the id-to-key dictionary cache.
/// </summary>
/// <returns>The id-to-key map.</returns>
public ConcurrentDictionary<ushort, string> GetKeyReverseMap()
{
return _dictionaryReverseCache;
}
/// <summary>
/// Gets the ID for a dictionary key, creating it if it doesn't exist.
/// Thread-safe.
/// </summary>
/// <param name="key">The dictionary key.</param>
/// <returns>The dictionary identifier for the key.</returns>
public ushort GetOrAddDictionaryEntry(string key)
{
key = key.ToLowerInvariant();
if (_dictionaryCache.TryGetValue(key, out ushort id)) return id;
lock (_dictionaryLock)
{
// Double checked locking
if (_dictionaryCache.TryGetValue(key, out id)) return id;
// Try to find in storage (in case cache is incomplete or another process?)
// Note: FindAllGlobal loaded everything, so strict cache miss means it's not in DB.
// BUT if we support concurrent writers (multiple processed), we should re-check DB.
// Current CBDD seems to be single-process exclusive lock (FileShare.None).
// So in-memory cache is authoritative after load.
// Generate New ID
ushort nextId = _nextDictionaryId;
if (nextId == 0) nextId = DictionaryPage.ReservedValuesEnd + 1; // Should be init, but safety
// Insert into Page
// usage of default(ulong) or null transaction?
// Dictionary updates should ideally be transactional or immediate?
// "Immediate" for now to simplify, as dictionary is cross-collection.
// If we use transaction, we need to pass it in. For now, immediate write.
// We need to support "Insert Global" which handles overflow.
// DictionaryPage.Insert only handles single page.
// We need logic here to traverse chain and find space.
if (InsertDictionaryEntryGlobal(key, nextId))
{
_dictionaryCache[key] = nextId;
_dictionaryReverseCache[nextId] = key;
_nextDictionaryId++;
return nextId;
}
throw new InvalidOperationException("Failed to insert dictionary entry (Storage Full?)");
}
}
/// <summary>
/// Registers a set of keys in the global dictionary.
/// Ensures all keys are assigned an ID and persisted.
/// </summary>
/// <param name="keys">The keys to register.</param>
public void RegisterKeys(IEnumerable<string> keys)
{
foreach (string key in keys) GetOrAddDictionaryEntry(key.ToLowerInvariant());
}
private void InitializeDictionary()
{
// 1. Read File Header (Page 0) to get Dictionary Root
var headerBuffer = new byte[PageSize];
ReadPage(0, null, headerBuffer);
var header = PageHeader.ReadFrom(headerBuffer);
if (header.DictionaryRootPageId == 0)
{
// Initialize new Dictionary
lock (_dictionaryLock)
if (header.DictionaryRootPageId == 0)
{
// Initialize new Dictionary
lock (_dictionaryLock)
{
// Double check
ReadPage(0, null, headerBuffer);
@@ -48,172 +123,92 @@ public sealed partial class StorageEngine
else
{
_dictionaryRootPageId = header.DictionaryRootPageId;
}
}
}
else
{
}
}
}
else
{
_dictionaryRootPageId = header.DictionaryRootPageId;
// Warm cache
ushort maxId = DictionaryPage.ReservedValuesEnd;
foreach (var (key, val) in DictionaryPage.FindAllGlobal(this, _dictionaryRootPageId))
{
var lowerKey = key.ToLowerInvariant();
_dictionaryCache[lowerKey] = val;
_dictionaryReverseCache[val] = lowerKey;
if (val > maxId) maxId = val;
}
_nextDictionaryId = (ushort)(maxId + 1);
}
// Pre-register internal keys used for Schema persistence
ushort maxId = DictionaryPage.ReservedValuesEnd;
foreach ((string key, ushort val) in DictionaryPage.FindAllGlobal(this, _dictionaryRootPageId))
{
string lowerKey = key.ToLowerInvariant();
_dictionaryCache[lowerKey] = val;
_dictionaryReverseCache[val] = lowerKey;
if (val > maxId) maxId = val;
}
_nextDictionaryId = (ushort)(maxId + 1);
}
// Pre-register internal keys used for Schema persistence
RegisterKeys(new[] { "_id", "t", "_v", "f", "n", "b", "s", "a" });
// Pre-register common array indices to avoid mapping during high-frequency writes
var indices = new List<string>(101);
for (int i = 0; i <= 100; i++) indices.Add(i.ToString());
RegisterKeys(indices);
}
/// <summary>
/// Gets the key-to-id dictionary cache.
/// </summary>
/// <returns>The key-to-id map.</returns>
public ConcurrentDictionary<string, ushort> GetKeyMap() => _dictionaryCache;
/// <summary>
/// Gets the id-to-key dictionary cache.
/// </summary>
/// <returns>The id-to-key map.</returns>
public ConcurrentDictionary<ushort, string> GetKeyReverseMap() => _dictionaryReverseCache;
/// <summary>
/// Gets the ID for a dictionary key, creating it if it doesn't exist.
/// Thread-safe.
/// </summary>
/// <param name="key">The dictionary key.</param>
/// <returns>The dictionary identifier for the key.</returns>
public ushort GetOrAddDictionaryEntry(string key)
{
key = key.ToLowerInvariant();
if (_dictionaryCache.TryGetValue(key, out var id))
{
return id;
}
lock (_dictionaryLock)
{
// Double checked locking
if (_dictionaryCache.TryGetValue(key, out id))
{
return id;
}
// Try to find in storage (in case cache is incomplete or another process?)
// Note: FindAllGlobal loaded everything, so strict cache miss means it's not in DB.
// BUT if we support concurrent writers (multiple processed), we should re-check DB.
// Current CBDD seems to be single-process exclusive lock (FileShare.None).
// So in-memory cache is authoritative after load.
// Generate New ID
ushort nextId = _nextDictionaryId;
if (nextId == 0) nextId = DictionaryPage.ReservedValuesEnd + 1; // Should be init, but safety
// Insert into Page
// usage of default(ulong) or null transaction?
// Dictionary updates should ideally be transactional or immediate?
// "Immediate" for now to simplify, as dictionary is cross-collection.
// If we use transaction, we need to pass it in. For now, immediate write.
// We need to support "Insert Global" which handles overflow.
// DictionaryPage.Insert only handles single page.
// We need logic here to traverse chain and find space.
if (InsertDictionaryEntryGlobal(key, nextId))
{
_dictionaryCache[key] = nextId;
_dictionaryReverseCache[nextId] = key;
_nextDictionaryId++;
return nextId;
}
else
{
throw new InvalidOperationException("Failed to insert dictionary entry (Storage Full?)");
}
}
var indices = new List<string>(101);
for (var i = 0; i <= 100; i++) indices.Add(i.ToString());
RegisterKeys(indices);
}
/// <summary>
/// Gets the dictionary key for an identifier.
/// Gets the dictionary key for an identifier.
/// </summary>
/// <param name="id">The dictionary identifier.</param>
/// <returns>The dictionary key if found; otherwise, <see langword="null"/>.</returns>
/// <returns>The dictionary key if found; otherwise, <see langword="null" />.</returns>
public string? GetDictionaryKey(ushort id)
{
if (_dictionaryReverseCache.TryGetValue(id, out var key))
if (_dictionaryReverseCache.TryGetValue(id, out string? key))
return key;
return null;
return null;
}
private bool InsertDictionaryEntryGlobal(string key, ushort value)
{
var pageId = _dictionaryRootPageId;
private bool InsertDictionaryEntryGlobal(string key, ushort value)
{
uint pageId = _dictionaryRootPageId;
var pageBuffer = new byte[PageSize];
while (true)
{
while (true)
{
ReadPage(pageId, null, pageBuffer);
// Try Insert
if (DictionaryPage.Insert(pageBuffer, key, value))
{
// Success - Write Back
WritePageImmediate(pageId, pageBuffer);
return true;
if (DictionaryPage.Insert(pageBuffer, key, value))
{
// Success - Write Back
WritePageImmediate(pageId, pageBuffer);
return true;
}
// Page Full - Check Next Page
var header = PageHeader.ReadFrom(pageBuffer);
if (header.NextPageId != 0)
{
pageId = header.NextPageId;
continue;
var header = PageHeader.ReadFrom(pageBuffer);
if (header.NextPageId != 0)
{
pageId = header.NextPageId;
continue;
}
// No Next Page - Allocate New
var newPageId = AllocatePage();
var newPageBuffer = new byte[PageSize];
uint newPageId = AllocatePage();
var newPageBuffer = new byte[PageSize];
DictionaryPage.Initialize(newPageBuffer, newPageId);
// Should likely insert into NEW page immediately to save I/O?
// Or just link and loop?
// Let's Insert into new page logic here to avoid re-reading.
if (!DictionaryPage.Insert(newPageBuffer, key, value))
if (!DictionaryPage.Insert(newPageBuffer, key, value))
return false; // Should not happen on empty page unless key is huge > page
// Write New Page
WritePageImmediate(newPageId, newPageBuffer);
// Update Previous Page Link
header.NextPageId = newPageId;
header.WriteTo(pageBuffer);
header.NextPageId = newPageId;
header.WriteTo(pageBuffer);
WritePageImmediate(pageId, pageBuffer);
return true;
}
}
/// <summary>
/// Registers a set of keys in the global dictionary.
/// Ensures all keys are assigned an ID and persisted.
/// </summary>
/// <param name="keys">The keys to register.</param>
public void RegisterKeys(IEnumerable<string> keys)
{
foreach (var key in keys)
{
GetOrAddDictionaryEntry(key.ToLowerInvariant());
}
}
}
return true;
}
}
}

33
src/CBDD.Core/Storage/StorageEngine.Format.cs
View File

@@ -15,31 +15,32 @@ internal readonly struct StorageFormatMetadata
internal const int WireSize = 16;
/// <summary>
/// Gets a value indicating whether format metadata is present.
/// Gets a value indicating whether format metadata is present.
/// </summary>
public bool IsPresent { get; }
/// <summary>
/// Gets the storage format version.
/// Gets the storage format version.
/// </summary>
public byte Version { get; }
/// <summary>
/// Gets enabled storage feature flags.
/// Gets enabled storage feature flags.
/// </summary>
public StorageFeatureFlags FeatureFlags { get; }
/// <summary>
/// Gets the default compression codec.
/// Gets the default compression codec.
/// </summary>
public CompressionCodec DefaultCodec { get; }
/// <summary>
/// Gets a value indicating whether compression capability is enabled.
/// Gets a value indicating whether compression capability is enabled.
/// </summary>
public bool CompressionCapabilityEnabled => (FeatureFlags & StorageFeatureFlags.CompressionCapability) != 0;
private StorageFormatMetadata(bool isPresent, byte version, StorageFeatureFlags featureFlags, CompressionCodec defaultCodec)
private StorageFormatMetadata(bool isPresent, byte version, StorageFeatureFlags featureFlags,
CompressionCodec defaultCodec)
{
IsPresent = isPresent;
Version = version;
@@ -48,18 +49,19 @@ internal readonly struct StorageFormatMetadata
}
/// <summary>
/// Creates metadata representing a modern format-aware file.
/// Creates metadata representing a modern format-aware file.
/// </summary>
/// <param name="version">The storage format version.</param>
/// <param name="featureFlags">Enabled feature flags.</param>
/// <param name="defaultCodec">The default compression codec.</param>
public static StorageFormatMetadata Present(byte version, StorageFeatureFlags featureFlags, CompressionCodec defaultCodec)
public static StorageFormatMetadata Present(byte version, StorageFeatureFlags featureFlags,
CompressionCodec defaultCodec)
{
return new StorageFormatMetadata(true, version, featureFlags, defaultCodec);
}
/// <summary>
/// Creates metadata representing a legacy file without format metadata.
/// Creates metadata representing a legacy file without format metadata.
/// </summary>
/// <param name="defaultCodec">The default compression codec.</param>
public static StorageFormatMetadata Legacy(CompressionCodec defaultCodec)
@@ -88,12 +90,13 @@ public sealed partial class StorageEngine
return metadata;
if (!_pageFile.WasCreated)
return StorageFormatMetadata.Legacy(_compressionOptions.Codec);
return StorageFormatMetadata.Legacy(CompressionOptions.Codec);
var featureFlags = _compressionOptions.EnableCompression
var featureFlags = CompressionOptions.EnableCompression
? StorageFeatureFlags.CompressionCapability
: StorageFeatureFlags.None;
var initialMetadata = StorageFormatMetadata.Present(CurrentStorageFormatVersion, featureFlags, _compressionOptions.Codec);
var initialMetadata =
StorageFormatMetadata.Present(CurrentStorageFormatVersion, featureFlags, CompressionOptions.Codec);
WriteStorageFormatMetadata(initialMetadata);
return initialMetadata;
}
@@ -104,11 +107,11 @@ public sealed partial class StorageEngine
if (source.Length < StorageFormatMetadata.WireSize)
return false;
var magic = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(0, 4));
uint magic = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(0, 4));
if (magic != StorageFormatMagic)
return false;
var version = source[4];
byte version = source[4];
var featureFlags = (StorageFeatureFlags)source[5];
var codec = (CompressionCodec)source[6];
if (!Enum.IsDefined(codec))
@@ -128,4 +131,4 @@ public sealed partial class StorageEngine
buffer[6] = (byte)metadata.DefaultCodec;
_pageFile.WritePageZeroExtension(StorageHeaderExtensionOffset, buffer);
}
}
}

1125
src/CBDD.Core/Storage/StorageEngine.Maintenance.cs
View File

File diff suppressed because it is too large Load Diff

8
src/CBDD.Core/Storage/StorageEngine.Memory.cs
View File

@@ -1,11 +1,9 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Allocates a new page.
/// Allocates a new page.
/// </summary>
/// <returns>Page ID of the allocated page</returns>
public uint AllocatePage()
@@ -14,11 +12,11 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Frees a page.
/// Frees a page.
/// </summary>
/// <param name="pageId">Page to free</param>
public void FreePage(uint pageId)
{
_pageFile.FreePage(pageId);
}
}
}

124
src/CBDD.Core/Storage/StorageEngine.Migration.cs
View File

@@ -5,98 +5,98 @@ using ZB.MOM.WW.CBDD.Core.Compression;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Options controlling compression migration.
/// Options controlling compression migration.
/// </summary>
public sealed class CompressionMigrationOptions
{
/// <summary>
/// Enables dry-run estimation without mutating database contents.
/// Enables dry-run estimation without mutating database contents.
/// </summary>
public bool DryRun { get; init; } = true;
/// <summary>
/// Target codec for migrated payloads.
/// Target codec for migrated payloads.
/// </summary>
public CompressionCodec Codec { get; init; } = CompressionCodec.Brotli;
/// <summary>
/// Target compression level.
/// Target compression level.
/// </summary>
public CompressionLevel Level { get; init; } = CompressionLevel.Fastest;
/// <summary>
/// Minimum logical payload size required before compression is attempted.
/// Minimum logical payload size required before compression is attempted.
/// </summary>
public int MinSizeBytes { get; init; } = 1024;
/// <summary>
/// Minimum savings percent required to keep compressed output.
/// Minimum savings percent required to keep compressed output.
/// </summary>
public int MinSavingsPercent { get; init; } = 10;
/// <summary>
/// Optional include-only collection list (case-insensitive).
/// Optional include-only collection list (case-insensitive).
/// </summary>
public IReadOnlyList<string>? IncludeCollections { get; init; }
/// <summary>
/// Optional exclusion collection list (case-insensitive).
/// Optional exclusion collection list (case-insensitive).
/// </summary>
public IReadOnlyList<string>? ExcludeCollections { get; init; }
}
/// <summary>
/// Result of a compression migration run.
/// Result of a compression migration run.
/// </summary>
public sealed class CompressionMigrationResult
{
/// <summary>
/// Gets a value indicating whether this run was executed in dry-run mode.
/// Gets a value indicating whether this run was executed in dry-run mode.
/// </summary>
public bool DryRun { get; init; }
/// <summary>
/// Gets the target codec used for migration output.
/// Gets the target codec used for migration output.
/// </summary>
public CompressionCodec Codec { get; init; }
/// <summary>
/// Gets the target compression level used for migration output.
/// Gets the target compression level used for migration output.
/// </summary>
public CompressionLevel Level { get; init; }
/// <summary>
/// Gets the number of collections processed.
/// Gets the number of collections processed.
/// </summary>
public int CollectionsProcessed { get; init; }
/// <summary>
/// Gets the number of documents scanned.
/// Gets the number of documents scanned.
/// </summary>
public long DocumentsScanned { get; init; }
/// <summary>
/// Gets the number of documents rewritten.
/// Gets the number of documents rewritten.
/// </summary>
public long DocumentsRewritten { get; init; }
/// <summary>
/// Gets the number of documents skipped.
/// Gets the number of documents skipped.
/// </summary>
public long DocumentsSkipped { get; init; }
/// <summary>
/// Gets the total logical bytes observed before migration decisions.
/// Gets the total logical bytes observed before migration decisions.
/// </summary>
public long BytesBefore { get; init; }
/// <summary>
/// Gets the estimated total stored bytes after migration.
/// Gets the estimated total stored bytes after migration.
/// </summary>
public long BytesEstimatedAfter { get; init; }
/// <summary>
/// Gets the actual total stored bytes after migration when not in dry-run mode.
/// Gets the actual total stored bytes after migration when not in dry-run mode.
/// </summary>
public long BytesActualAfter { get; init; }
}
@@ -104,7 +104,7 @@ public sealed class CompressionMigrationResult
public sealed partial class StorageEngine
{
/// <summary>
/// Estimates or applies a one-time compression migration.
/// Estimates or applies a one-time compression migration.
/// </summary>
/// <param name="options">Optional compression migration options.</param>
public CompressionMigrationResult MigrateCompression(CompressionMigrationOptions? options = null)
@@ -113,11 +113,12 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Estimates or applies a one-time compression migration.
/// Estimates or applies a one-time compression migration.
/// </summary>
/// <param name="options">Optional compression migration options.</param>
/// <param name="ct">A token used to cancel the operation.</param>
public async Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null, CancellationToken ct = default)
public async Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null,
CancellationToken ct = default)
{
var normalized = NormalizeMigrationOptions(options);
@@ -147,13 +148,13 @@ public sealed partial class StorageEngine
{
ct.ThrowIfCancellationRequested();
if (!TryReadStoredPayload(location, out var storedPayload, out var isCompressed))
if (!TryReadStoredPayload(location, out byte[] storedPayload, out bool isCompressed))
{
docsSkipped++;
continue;
}
if (!TryGetLogicalPayload(storedPayload, isCompressed, out var logicalPayload))
if (!TryGetLogicalPayload(storedPayload, isCompressed, out byte[] logicalPayload))
{
docsSkipped++;
continue;
@@ -162,15 +163,14 @@ public sealed partial class StorageEngine
docsScanned++;
bytesBefore += logicalPayload.Length;
var targetStored = BuildTargetStoredPayload(logicalPayload, normalized, out var targetCompressed);
byte[] targetStored =
BuildTargetStoredPayload(logicalPayload, normalized, out bool targetCompressed);
bytesEstimatedAfter += targetStored.Length;
if (normalized.DryRun)
{
continue;
}
if (normalized.DryRun) continue;
if (!TryRewriteStoredPayloadAtLocation(location, targetStored, targetCompressed, out var actualStoredBytes))
if (!TryRewriteStoredPayloadAtLocation(location, targetStored, targetCompressed,
out int actualStoredBytes))
{
docsSkipped++;
continue;
@@ -184,9 +184,9 @@ public sealed partial class StorageEngine
if (!normalized.DryRun)
{
var metadata = StorageFormatMetadata.Present(
version: 1,
featureFlags: StorageFeatureFlags.CompressionCapability,
defaultCodec: normalized.Codec);
1,
StorageFeatureFlags.CompressionCapability,
normalized.Codec);
WriteStorageFormatMetadata(metadata);
_pageFile.Flush();
}
@@ -221,7 +221,8 @@ public sealed partial class StorageEngine
var normalized = options ?? new CompressionMigrationOptions();
if (!Enum.IsDefined(normalized.Codec) || normalized.Codec == CompressionCodec.None)
throw new ArgumentOutOfRangeException(nameof(options), "Migration codec must be a supported non-None codec.");
throw new ArgumentOutOfRangeException(nameof(options),
"Migration codec must be a supported non-None codec.");
if (normalized.MinSizeBytes < 0)
throw new ArgumentOutOfRangeException(nameof(options), "MinSizeBytes must be non-negative.");
@@ -250,7 +251,8 @@ public sealed partial class StorageEngine
.ToList();
}
private byte[] BuildTargetStoredPayload(ReadOnlySpan<byte> logicalPayload, CompressionMigrationOptions options, out bool compressed)
private byte[] BuildTargetStoredPayload(ReadOnlySpan<byte> logicalPayload, CompressionMigrationOptions options,
out bool compressed)
{
compressed = false;
@@ -259,10 +261,10 @@ public sealed partial class StorageEngine
try
{
var compressedPayload = _compressionService.Compress(logicalPayload, options.Codec, options.Level);
var storedLength = CompressedPayloadHeader.Size + compressedPayload.Length;
var savings = logicalPayload.Length - storedLength;
var savingsPercent = logicalPayload.Length == 0 ? 0 : (int)((savings * 100L) / logicalPayload.Length);
byte[] compressedPayload = CompressionService.Compress(logicalPayload, options.Codec, options.Level);
int storedLength = CompressedPayloadHeader.Size + compressedPayload.Length;
int savings = logicalPayload.Length - storedLength;
int savingsPercent = logicalPayload.Length == 0 ? 0 : (int)(savings * 100L / logicalPayload.Length);
if (savings <= 0 || savingsPercent < options.MinSavingsPercent)
return logicalPayload.ToArray();
@@ -308,11 +310,11 @@ public sealed partial class StorageEngine
try
{
logicalPayload = _compressionService.Decompress(
logicalPayload = CompressionService.Decompress(
compressedPayload,
header.Codec,
header.OriginalLength,
Math.Max(header.OriginalLength, _compressionOptions.MaxDecompressedSizeBytes));
Math.Max(header.OriginalLength, CompressionOptions.MaxDecompressedSizeBytes));
return true;
}
catch
@@ -336,13 +338,13 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
isCompressed = (slot.Flags & SlotFlags.Compressed) != 0;
var hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
if (!hasOverflow)
{
@@ -354,14 +356,14 @@ public sealed partial class StorageEngine
return false;
var primaryPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
var totalStoredLength = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
var nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
int totalStoredLength = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
uint nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
if (totalStoredLength < 0)
return false;
var output = new byte[totalStoredLength];
var primaryChunk = primaryPayload.Slice(8);
var copied = Math.Min(primaryChunk.Length, output.Length);
int copied = Math.Min(primaryChunk.Length, output.Length);
primaryChunk.Slice(0, copied).CopyTo(output);
var overflowBuffer = new byte[_pageFile.PageSize];
@@ -372,7 +374,7 @@ public sealed partial class StorageEngine
if (overflowHeader.PageType != PageType.Overflow)
return false;
var chunk = Math.Min(output.Length - copied, _pageFile.PageSize - SlottedPageHeader.Size);
int chunk = Math.Min(output.Length - copied, _pageFile.PageSize - SlottedPageHeader.Size);
overflowBuffer.AsSpan(SlottedPageHeader.Size, chunk).CopyTo(output.AsSpan(copied));
copied += chunk;
nextOverflow = overflowHeader.NextOverflowPage;
@@ -403,12 +405,12 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= pageHeader.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
var oldHasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool oldHasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
uint oldOverflowHead = 0;
if (oldHasOverflow)
{
@@ -442,12 +444,12 @@ public sealed partial class StorageEngine
if (slot.Length < 8)
return false;
var primaryChunkSize = slot.Length - 8;
int primaryChunkSize = slot.Length - 8;
if (primaryChunkSize < 0)
return false;
var remainder = newStoredPayload.Slice(primaryChunkSize);
var newOverflowHead = BuildOverflowChainForMigration(remainder);
uint newOverflowHead = BuildOverflowChainForMigration(remainder);
var slotPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
slotPayload.Clear();
@@ -475,22 +477,22 @@ public sealed partial class StorageEngine
if (overflowPayload.IsEmpty)
return 0;
var chunkSize = _pageFile.PageSize - SlottedPageHeader.Size;
int chunkSize = _pageFile.PageSize - SlottedPageHeader.Size;
uint nextOverflowPageId = 0;
var tailSize = overflowPayload.Length % chunkSize;
var fullPages = overflowPayload.Length / chunkSize;
int tailSize = overflowPayload.Length % chunkSize;
int fullPages = overflowPayload.Length / chunkSize;
if (tailSize > 0)
{
var tailOffset = fullPages * chunkSize;
int tailOffset = fullPages * chunkSize;
var tailSlice = overflowPayload.Slice(tailOffset, tailSize);
nextOverflowPageId = AllocateOverflowPageForMigration(tailSlice, nextOverflowPageId);
}
for (var i = fullPages - 1; i >= 0; i--)
for (int i = fullPages - 1; i >= 0; i--)
{
var chunkOffset = i * chunkSize;
int chunkOffset = i * chunkSize;
var chunk = overflowPayload.Slice(chunkOffset, chunkSize);
nextOverflowPageId = AllocateOverflowPageForMigration(chunk, nextOverflowPageId);
}
@@ -500,7 +502,7 @@ public sealed partial class StorageEngine
private uint AllocateOverflowPageForMigration(ReadOnlySpan<byte> payloadChunk, uint nextOverflowPageId)
{
var pageId = _pageFile.AllocatePage();
uint pageId = _pageFile.AllocatePage();
var buffer = new byte[_pageFile.PageSize];
var header = new SlottedPageHeader
@@ -524,15 +526,15 @@ public sealed partial class StorageEngine
{
var buffer = new byte[_pageFile.PageSize];
var visited = new HashSet<uint>();
var current = firstOverflowPage;
uint current = firstOverflowPage;
while (current != 0 && current < _pageFile.NextPageId && visited.Add(current))
{
_pageFile.ReadPage(current, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
var next = header.NextOverflowPage;
uint next = header.NextOverflowPage;
_pageFile.FreePage(current);
current = next;
}
}
}
}

64
src/CBDD.Core/Storage/StorageEngine.Pages.cs
View File

@@ -1,14 +1,14 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
using System.Collections.Concurrent;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Reads a page with transaction isolation.
/// 1. Check WAL cache for uncommitted writes (Read Your Own Writes)
/// 2. Check WAL index for committed writes (lazy replay)
/// 3. Read from PageFile (committed baseline)
/// Reads a page with transaction isolation.
/// 1. Check WAL cache for uncommitted writes (Read Your Own Writes)
/// 2. Check WAL index for committed writes (lazy replay)
/// 3. Read from PageFile (committed baseline)
/// </summary>
/// <param name="pageId">Page to read</param>
/// <param name="transactionId">Optional transaction ID for isolation</param>
@@ -17,32 +17,32 @@ public sealed partial class StorageEngine
{
// 1. Check transaction-local WAL cache (Read Your Own Writes)
// transactionId=0 or null means "no active transaction, read committed only"
if (transactionId.HasValue &&
if (transactionId.HasValue &&
transactionId.Value != 0 &&
_walCache.TryGetValue(transactionId.Value, out var txnPages) &&
txnPages.TryGetValue(pageId, out var uncommittedData))
txnPages.TryGetValue(pageId, out byte[]? uncommittedData))
{
var length = Math.Min(uncommittedData.Length, destination.Length);
int length = Math.Min(uncommittedData.Length, destination.Length);
uncommittedData.AsSpan(0, length).CopyTo(destination);
return;
}
// 2. Check WAL index (committed but not checkpointed)
if (_walIndex.TryGetValue(pageId, out var committedData))
}
// 2. Check WAL index (committed but not checkpointed)
if (_walIndex.TryGetValue(pageId, out byte[]? committedData))
{
var length = Math.Min(committedData.Length, destination.Length);
int length = Math.Min(committedData.Length, destination.Length);
committedData.AsSpan(0, length).CopyTo(destination);
return;
}
// 3. Read committed baseline from PageFile
}
// 3. Read committed baseline from PageFile
_pageFile.ReadPage(pageId, destination);
}
/// <summary>
/// Writes a page within a transaction.
/// Data goes to WAL cache immediately and becomes visible to that transaction only.
/// Will be written to WAL on commit.
/// Writes a page within a transaction.
/// Data goes to WAL cache immediately and becomes visible to that transaction only.
/// Will be written to WAL on commit.
/// </summary>
/// <param name="pageId">Page to write</param>
/// <param name="transactionId">Transaction ID owning this write</param>
@@ -50,20 +50,20 @@ public sealed partial class StorageEngine
public void WritePage(uint pageId, ulong transactionId, ReadOnlySpan<byte> data)
{
if (transactionId == 0)
throw new InvalidOperationException("Cannot write without a transaction (transactionId=0 is reserved)");
// Get or create transaction-local cache
var txnPages = _walCache.GetOrAdd(transactionId,
_ => new System.Collections.Concurrent.ConcurrentDictionary<uint, byte[]>());
// Store defensive copy
var copy = data.ToArray();
throw new InvalidOperationException("Cannot write without a transaction (transactionId=0 is reserved)");
// Get or create transaction-local cache
var txnPages = _walCache.GetOrAdd(transactionId,
_ => new ConcurrentDictionary<uint, byte[]>());
// Store defensive copy
byte[] copy = data.ToArray();
txnPages[pageId] = copy;
}
/// <summary>
/// Writes a page immediately to disk (non-transactional).
/// Used for initialization and metadata updates outside of transactions.
/// Writes a page immediately to disk (non-transactional).
/// Used for initialization and metadata updates outside of transactions.
/// </summary>
/// <param name="pageId">Page to write</param>
/// <param name="data">Page data</param>
@@ -73,8 +73,8 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets the number of pages currently allocated in the page file.
/// Useful for full database scans.
/// Gets the number of pages currently allocated in the page file.
/// Useful for full database scans.
/// </summary>
public uint PageCount => _pageFile.NextPageId;
}
}

133
src/CBDD.Core/Storage/StorageEngine.Recovery.cs
View File

@@ -1,36 +1,36 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Gets the current size of the WAL file.
/// </summary>
public long GetWalSize()
{
return _wal.GetCurrentSize();
}
/// <summary>
/// Truncates the WAL file.
/// Should only be called after a successful checkpoint.
/// </summary>
public void TruncateWal()
{
_wal.Truncate();
}
/// <summary>
/// Flushes the WAL to disk.
/// </summary>
public void FlushWal()
{
_wal.Flush();
/// <summary>
/// Gets the current size of the WAL file.
/// </summary>
public long GetWalSize()
{
return _wal.GetCurrentSize();
}
/// <summary>
/// Performs a truncate checkpoint by default.
/// Truncates the WAL file.
/// Should only be called after a successful checkpoint.
/// </summary>
public void TruncateWal()
{
_wal.Truncate();
}
/// <summary>
/// Flushes the WAL to disk.
/// </summary>
public void FlushWal()
{
_wal.Flush();
}
/// <summary>
/// Performs a truncate checkpoint by default.
/// </summary>
public void Checkpoint()
{
@@ -38,7 +38,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Performs a checkpoint using the requested mode.
/// Performs a checkpoint using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <returns>The checkpoint execution result.</returns>
@@ -50,7 +50,7 @@ public sealed partial class StorageEngine
lockAcquired = _commitLock.Wait(0);
if (!lockAcquired)
{
var walSize = _wal.GetCurrentSize();
long walSize = _wal.GetCurrentSize();
return new CheckpointResult(mode, false, 0, walSize, walSize, false, false);
}
}
@@ -66,19 +66,18 @@ public sealed partial class StorageEngine
}
finally
{
if (lockAcquired)
{
_commitLock.Release();
}
if (lockAcquired) _commitLock.Release();
}
}
private void CheckpointInternal()
=> _ = CheckpointInternal(CheckpointMode.Truncate);
{
_ = CheckpointInternal(CheckpointMode.Truncate);
}
private CheckpointResult CheckpointInternal(CheckpointMode mode)
{
var walBytesBefore = _wal.GetCurrentSize();
long walBytesBefore = _wal.GetCurrentSize();
var appliedPages = 0;
var truncated = false;
var restarted = false;
@@ -91,10 +90,7 @@ public sealed partial class StorageEngine
}
// 2. Flush PageFile to ensure durability.
if (appliedPages > 0)
{
_pageFile.Flush();
}
if (appliedPages > 0) _pageFile.Flush();
// 3. Clear in-memory WAL index (now persisted).
_walIndex.Clear();
@@ -109,6 +105,7 @@ public sealed partial class StorageEngine
_wal.WriteCheckpointRecord();
_wal.Flush();
}
break;
case CheckpointMode.Truncate:
if (walBytesBefore > 0)
@@ -116,6 +113,7 @@ public sealed partial class StorageEngine
_wal.Truncate();
truncated = true;
}
break;
case CheckpointMode.Restart:
_wal.Restart();
@@ -126,12 +124,12 @@ public sealed partial class StorageEngine
throw new ArgumentOutOfRangeException(nameof(mode), mode, "Unsupported checkpoint mode.");
}
var walBytesAfter = _wal.GetCurrentSize();
long walBytesAfter = _wal.GetCurrentSize();
return new CheckpointResult(mode, true, appliedPages, walBytesBefore, walBytesAfter, truncated, restarted);
}
/// <summary>
/// Performs a truncate checkpoint asynchronously by default.
/// Performs a truncate checkpoint asynchronously by default.
/// </summary>
/// <param name="ct">The cancellation token.</param>
public async Task CheckpointAsync(CancellationToken ct = default)
@@ -140,7 +138,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Performs a checkpoint asynchronously using the requested mode.
/// Performs a checkpoint asynchronously using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <param name="ct">The cancellation token.</param>
@@ -153,7 +151,7 @@ public sealed partial class StorageEngine
lockAcquired = await _commitLock.WaitAsync(0, ct);
if (!lockAcquired)
{
var walSize = _wal.GetCurrentSize();
long walSize = _wal.GetCurrentSize();
return new CheckpointResult(mode, false, 0, walSize, walSize, false, false);
}
}
@@ -170,16 +168,13 @@ public sealed partial class StorageEngine
}
finally
{
if (lockAcquired)
{
_commitLock.Release();
}
if (lockAcquired) _commitLock.Release();
}
}
/// <summary>
/// Recovers from crash by replaying WAL.
/// Applies committed transactions to PageFile in deterministic WAL order, then truncates WAL.
/// Recovers from crash by replaying WAL.
/// Applies committed transactions to PageFile in deterministic WAL order, then truncates WAL.
/// </summary>
public void Recover()
{
@@ -189,35 +184,28 @@ public sealed partial class StorageEngine
// 1. Read WAL and locate the latest checkpoint boundary.
var records = _wal.ReadAll();
var startIndex = 0;
for (var i = records.Count - 1; i >= 0; i--)
{
for (int i = records.Count - 1; i >= 0; i--)
if (records[i].Type == WalRecordType.Checkpoint)
{
startIndex = i + 1;
break;
}
}
// 2. Replay WAL in source order with deterministic commit application.
var pendingWrites = new Dictionary<ulong, List<(uint pageId, byte[] data)>>();
var appliedAny = false;
for (var i = startIndex; i < records.Count; i++)
for (int i = startIndex; i < records.Count; i++)
{
var record = records[i];
switch (record.Type)
{
case WalRecordType.Begin:
if (!pendingWrites.ContainsKey(record.TransactionId))
{
pendingWrites[record.TransactionId] = new List<(uint, byte[])>();
}
break;
case WalRecordType.Write:
if (record.AfterImage == null)
{
break;
}
if (record.AfterImage == null) break;
if (!pendingWrites.TryGetValue(record.TransactionId, out var writes))
{
@@ -228,12 +216,9 @@ public sealed partial class StorageEngine
writes.Add((record.PageId, record.AfterImage));
break;
case WalRecordType.Commit:
if (!pendingWrites.TryGetValue(record.TransactionId, out var committedWrites))
{
break;
}
if (!pendingWrites.TryGetValue(record.TransactionId, out var committedWrites)) break;
foreach (var (pageId, data) in committedWrites)
foreach ((uint pageId, byte[] data) in committedWrites)
{
_pageFile.WritePage(pageId, data);
appliedAny = true;
@@ -251,23 +236,17 @@ public sealed partial class StorageEngine
}
// 3. Flush PageFile to ensure durability.
if (appliedAny)
{
_pageFile.Flush();
}
if (appliedAny) _pageFile.Flush();
// 4. Clear in-memory WAL index (redundant since we just recovered).
_walIndex.Clear();
// 5. Truncate WAL (all changes now in PageFile).
if (_wal.GetCurrentSize() > 0)
{
_wal.Truncate();
}
if (_wal.GetCurrentSize() > 0) _wal.Truncate();
}
finally
{
_commitLock.Release();
}
}
}
_commitLock.Release();
}
}
}

66
src/CBDD.Core/Storage/StorageEngine.Schema.cs
View File

@@ -1,30 +1,28 @@
using System;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Bson.Schema;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Reads all schemas from the schema page chain.
/// </summary>
/// <param name="rootPageId">The root page identifier of the schema chain.</param>
/// <returns>The list of schemas in chain order.</returns>
public List<BsonSchema> GetSchemas(uint rootPageId)
public sealed partial class StorageEngine
{
/// <summary>
/// Reads all schemas from the schema page chain.
/// </summary>
/// <param name="rootPageId">The root page identifier of the schema chain.</param>
/// <returns>The list of schemas in chain order.</returns>
public List<BsonSchema> GetSchemas(uint rootPageId)
{
var schemas = new List<BsonSchema>();
if (rootPageId == 0) return schemas;
var pageId = rootPageId;
uint pageId = rootPageId;
var buffer = new byte[PageSize];
while (pageId != 0)
{
ReadPage(pageId, null, buffer);
var header = PageHeader.ReadFrom(buffer);
var header = PageHeader.ReadFrom(buffer);
if (header.PageType != PageType.Schema) break;
int used = PageSize - 32 - header.FreeBytes;
@@ -33,7 +31,7 @@ public sealed partial class StorageEngine
var reader = new BsonSpanReader(buffer.AsSpan(32, used), GetKeyReverseMap());
while (reader.Remaining >= 4)
{
var docSize = reader.PeekInt32();
int docSize = reader.PeekInt32();
if (docSize <= 0 || docSize > reader.Remaining) break;
var schema = BsonSchema.FromBson(ref reader);
@@ -47,27 +45,27 @@ public sealed partial class StorageEngine
return schemas;
}
/// <summary>
/// Appends a new schema version. Returns the root page ID (which might be new if it was 0 initially).
/// </summary>
/// <param name="rootPageId">The root page identifier of the schema chain.</param>
/// <param name="schema">The schema to append.</param>
public uint AppendSchema(uint rootPageId, BsonSchema schema)
/// <summary>
/// Appends a new schema version. Returns the root page ID (which might be new if it was 0 initially).
/// </summary>
/// <param name="rootPageId">The root page identifier of the schema chain.</param>
/// <param name="schema">The schema to append.</param>
public uint AppendSchema(uint rootPageId, BsonSchema schema)
{
var buffer = new byte[PageSize];
// Serialize schema to temporary buffer to calculate size
var buffer = new byte[PageSize];
// Serialize schema to temporary buffer to calculate size
var tempBuffer = new byte[PageSize];
var tempWriter = new BsonSpanWriter(tempBuffer, GetKeyMap());
schema.ToBson(ref tempWriter);
var schemaSize = tempWriter.Position;
int schemaSize = tempWriter.Position;
if (rootPageId == 0)
{
rootPageId = AllocatePage();
InitializeSchemaPage(buffer, rootPageId);
tempBuffer.AsSpan(0, schemaSize).CopyTo(buffer.AsSpan(32));
tempBuffer.AsSpan(0, schemaSize).CopyTo(buffer.AsSpan(32));
var header = PageHeader.ReadFrom(buffer);
header.FreeBytes = (ushort)(PageSize - 32 - schemaSize);
header.WriteTo(buffer);
@@ -91,13 +89,13 @@ public sealed partial class StorageEngine
// Buffer now contains the last page
var lastHeader = PageHeader.ReadFrom(buffer);
int currentUsed = PageSize - 32 - lastHeader.FreeBytes;
int lastOffset = 32 + currentUsed;
int lastOffset = 32 + currentUsed;
if (lastHeader.FreeBytes >= schemaSize)
{
// Fits in current page
tempBuffer.AsSpan(0, schemaSize).CopyTo(buffer.AsSpan(lastOffset));
tempBuffer.AsSpan(0, schemaSize).CopyTo(buffer.AsSpan(lastOffset));
lastHeader.FreeBytes -= (ushort)schemaSize;
lastHeader.WriteTo(buffer);
@@ -106,14 +104,14 @@ public sealed partial class StorageEngine
else
{
// Allocate new page
var newPageId = AllocatePage();
uint newPageId = AllocatePage();
lastHeader.NextPageId = newPageId;
lastHeader.WriteTo(buffer);
WritePageImmediate(lastPageId, buffer);
InitializeSchemaPage(buffer, newPageId);
tempBuffer.AsSpan(0, schemaSize).CopyTo(buffer.AsSpan(32));
tempBuffer.AsSpan(0, schemaSize).CopyTo(buffer.AsSpan(32));
var newHeader = PageHeader.ReadFrom(buffer);
newHeader.FreeBytes = (ushort)(PageSize - 32 - schemaSize);
newHeader.WriteTo(buffer);
@@ -145,4 +143,4 @@ public sealed partial class StorageEngine
var doc = reader.RemainingBytes();
doc.CopyTo(page.Slice(32));
}
}
}

437
src/CBDD.Core/Storage/StorageEngine.Transactions.cs
View File

@@ -1,194 +1,75 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
#region Transaction Management
/// <summary>
/// Gets the number of active transactions (diagnostics).
/// </summary>
public int ActiveTransactionCount => _walCache.Count;
/// <summary>
/// Begins a new transaction.
/// Prepares a transaction: writes all changes to WAL but doesn't commit yet.
/// Part of 2-Phase Commit protocol.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <returns>The started transaction.</returns>
public Transaction BeginTransaction(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
/// <param name="transactionId">Transaction ID</param>
/// <param name="writeSet">All writes to record in WAL</param>
/// <returns>True if preparation succeeded</returns>
public bool PrepareTransaction(ulong transactionId)
{
_commitLock.Wait();
try
{
var txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Begins a new transaction asynchronously.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The started transaction.</returns>
public async Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
var txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
public void CommitTransaction(Transaction transaction)
{
_commitLock.Wait();
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
// In a fuller 2PC, this would be separate. Here we do it as part of commit.
if (!PrepareTransaction(transaction.TransactionId))
throw new IOException("Failed to write transaction to WAL");
_wal.WriteBeginRecord(transactionId);
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
CommitTransactionCore(transaction.TransactionId);
foreach (var walEntry in _walCache[transactionId])
_wal.WriteDataRecord(transactionId, walEntry.Key, walEntry.Value);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
_wal.Flush(); // Ensure WAL is persisted
return true;
}
finally
catch
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction asynchronously.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
/// <param name="ct">The cancellation token.</param>
public async Task CommitTransactionAsync(Transaction transaction, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
if (!await PrepareTransactionAsync(transaction.TransactionId, ct))
throw new IOException("Failed to write transaction to WAL");
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
await CommitTransactionCoreAsync(transaction.TransactionId, ct);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to roll back.</param>
public void RollbackTransaction(Transaction transaction)
{
RollbackTransaction(transaction.TransactionId);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
// Rollback doesn't usually require async logic unless logging abort record is async,
// but for consistency we might consider it. For now, sync is fine as it's not the happy path bottleneck.
#endregion
/// <summary>
/// Prepares a transaction: writes all changes to WAL but doesn't commit yet.
/// Part of 2-Phase Commit protocol.
/// </summary>
/// <param name="transactionId">Transaction ID</param>
/// <param name="writeSet">All writes to record in WAL</param>
/// <returns>True if preparation succeeded</returns>
public bool PrepareTransaction(ulong transactionId)
{
try
{
_wal.WriteBeginRecord(transactionId);
foreach (var walEntry in _walCache[transactionId])
{
_wal.WriteDataRecord(transactionId, walEntry.Key, walEntry.Value);
}
_wal.Flush(); // Ensure WAL is persisted
return true;
}
catch
{
// TODO: Log error?
// TODO: Log error?
return false;
}
}
/// <summary>
/// Prepares a transaction asynchronously by writing pending changes to the WAL.
/// Prepares a transaction asynchronously by writing pending changes to the WAL.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns><see langword="true"/> if preparation succeeds; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if preparation succeeds; otherwise, <see langword="false" />.</returns>
public async Task<bool> PrepareTransactionAsync(ulong transactionId, CancellationToken ct = default)
{
try
{
await _wal.WriteBeginRecordAsync(transactionId, ct);
if (_walCache.TryGetValue(transactionId, out var changes))
{
foreach (var walEntry in changes)
{
await _wal.WriteDataRecordAsync(transactionId, walEntry.Key, walEntry.Value, ct);
}
}
await _wal.WriteBeginRecordAsync(transactionId, ct);
if (_walCache.TryGetValue(transactionId, out var changes))
foreach (var walEntry in changes)
await _wal.WriteDataRecordAsync(transactionId, walEntry.Key, walEntry.Value, ct);
await _wal.FlushAsync(ct); // Ensure WAL is persisted
return true;
}
catch
{
return false;
}
}
await _wal.FlushAsync(ct); // Ensure WAL is persisted
return true;
}
catch
{
return false;
}
}
/// <summary>
/// Commits a transaction:
/// 1. Writes all changes to WAL (for durability)
/// 2. Writes commit record
/// 3. Flushes WAL to disk
/// 4. Moves pages from cache to WAL index (for future reads)
/// 5. Clears WAL cache
/// </summary>
/// <param name="transactionId">Transaction to commit</param>
/// <param name="writeSet">All writes performed in this transaction (unused, kept for compatibility)</param>
/// Commits a transaction:
/// 1. Writes all changes to WAL (for durability)
/// 2. Writes commit record
/// 3. Flushes WAL to disk
/// 4. Moves pages from cache to WAL index (for future reads)
/// 5. Clears WAL cache
/// </summary>
/// <param name="transactionId">Transaction to commit</param>
/// <param name="writeSet">All writes performed in this transaction (unused, kept for compatibility)</param>
public void CommitTransaction(ulong transactionId)
{
_commitLock.Wait();
@@ -216,10 +97,7 @@ public sealed partial class StorageEngine
// 1. Write all changes to WAL (from cache, not writeSet!)
_wal.WriteBeginRecord(transactionId);
foreach (var (pageId, data) in pages)
{
_wal.WriteDataRecord(transactionId, pageId, data);
}
foreach ((uint pageId, byte[] data) in pages) _wal.WriteDataRecord(transactionId, pageId, data);
// 2. Write commit record and flush
_wal.WriteCommitRecord(transactionId);
@@ -227,20 +105,14 @@ public sealed partial class StorageEngine
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
foreach (var kvp in pages) _walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
CheckpointInternal();
}
if (_wal.GetCurrentSize() > MaxWalSize) CheckpointInternal();
}
/// <summary>
/// Commits a prepared transaction asynchronously by identifier.
/// Commits a prepared transaction asynchronously by identifier.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
@@ -271,10 +143,7 @@ public sealed partial class StorageEngine
// 1. Write all changes to WAL (from cache, not writeSet!)
await _wal.WriteBeginRecordAsync(transactionId, ct);
foreach (var (pageId, data) in pages)
{
await _wal.WriteDataRecordAsync(transactionId, pageId, data, ct);
}
foreach ((uint pageId, byte[] data) in pages) await _wal.WriteDataRecordAsync(transactionId, pageId, data, ct);
// 2. Write commit record and flush
await _wal.WriteCommitRecordAsync(transactionId, ct);
@@ -282,75 +151,177 @@ public sealed partial class StorageEngine
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
foreach (var kvp in pages) _walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
// Checkpoint might be sync or async. For now sync inside the lock is "safe" but blocking.
// Ideally this should be async too.
CheckpointInternal();
}
}
/// <summary>
/// Marks a transaction as committed after WAL writes.
/// Used for 2PC: after Prepare() writes to WAL, this finalizes the commit.
/// </summary>
/// <param name="transactionId">Transaction to mark committed</param>
public void MarkTransactionCommitted(ulong transactionId)
{
_commitLock.Wait();
try
{
_wal.WriteCommitRecord(transactionId);
/// <summary>
/// Marks a transaction as committed after WAL writes.
/// Used for 2PC: after Prepare() writes to WAL, this finalizes the commit.
/// </summary>
/// <param name="transactionId">Transaction to mark committed</param>
public void MarkTransactionCommitted(ulong transactionId)
{
_commitLock.Wait();
try
{
_wal.WriteCommitRecord(transactionId);
_wal.Flush();
// Move from cache to WAL index
if (_walCache.TryRemove(transactionId, out var pages))
{
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
}
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
CheckpointInternal();
}
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back a transaction: discards all uncommitted changes.
/// </summary>
/// <param name="transactionId">Transaction to rollback</param>
public void RollbackTransaction(ulong transactionId)
{
_walCache.TryRemove(transactionId, out _);
if (_walCache.TryRemove(transactionId, out var pages))
foreach (var kvp in pages)
_walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize) CheckpointInternal();
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back a transaction: discards all uncommitted changes.
/// </summary>
/// <param name="transactionId">Transaction to rollback</param>
public void RollbackTransaction(ulong transactionId)
{
_walCache.TryRemove(transactionId, out _);
_wal.WriteAbortRecord(transactionId);
}
/// <summary>
/// Writes an abort record for the specified transaction.
/// Writes an abort record for the specified transaction.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
internal void WriteAbortRecord(ulong transactionId)
{
_wal.WriteAbortRecord(transactionId);
}
/// <summary>
/// Gets the number of active transactions (diagnostics).
/// </summary>
public int ActiveTransactionCount => _walCache.Count;
}
#region Transaction Management
/// <summary>
/// Begins a new transaction.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <returns>The started transaction.</returns>
public Transaction BeginTransaction(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
{
_commitLock.Wait();
try
{
ulong txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Begins a new transaction asynchronously.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The started transaction.</returns>
public async Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted,
CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
ulong txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
public void CommitTransaction(Transaction transaction)
{
_commitLock.Wait();
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
// In a fuller 2PC, this would be separate. Here we do it as part of commit.
if (!PrepareTransaction(transaction.TransactionId))
throw new IOException("Failed to write transaction to WAL");
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
CommitTransactionCore(transaction.TransactionId);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction asynchronously.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
/// <param name="ct">The cancellation token.</param>
public async Task CommitTransactionAsync(Transaction transaction, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
if (!await PrepareTransactionAsync(transaction.TransactionId, ct))
throw new IOException("Failed to write transaction to WAL");
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
await CommitTransactionCoreAsync(transaction.TransactionId, ct);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to roll back.</param>
public void RollbackTransaction(Transaction transaction)
{
RollbackTransaction(transaction.TransactionId);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
// Rollback doesn't usually require async logic unless logging abort record is async,
// but for consistency we might consider it. For now, sync is fine as it's not the happy path bottleneck.
#endregion
}

181
src/CBDD.Core/Storage/StorageEngine.cs
View File

@@ -1,29 +1,30 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Compression;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Central storage engine managing page-based storage with WAL for durability.
///
/// Architecture (WAL-based like SQLite/PostgreSQL):
/// - PageFile: Committed baseline (persistent on disk)
/// - WAL Cache: Uncommitted transaction writes (in-memory)
/// - Read: PageFile + WAL cache overlay (for Read Your Own Writes)
/// - Commit: Flush to WAL, clear cache
/// - Checkpoint: Merge WAL ? PageFile periodically
/// Central storage engine managing page-based storage with WAL for durability.
/// Architecture (WAL-based like SQLite/PostgreSQL):
/// - PageFile: Committed baseline (persistent on disk)
/// - WAL Cache: Uncommitted transaction writes (in-memory)
/// - Read: PageFile + WAL cache overlay (for Read Your Own Writes)
/// - Commit: Flush to WAL, clear cache
/// - Checkpoint: Merge WAL ? PageFile periodically
/// </summary>
public sealed partial class StorageEngine : IStorageEngine, IDisposable
{
private const long MaxWalSize = 4 * 1024 * 1024; // 4MB
// Transaction Management
private readonly ConcurrentDictionary<ulong, Transaction> _activeTransactions;
// Global lock for commit/checkpoint synchronization
private readonly SemaphoreSlim _commitLock = new(1, 1);
private readonly PageFile _pageFile;
private readonly WriteAheadLog _wal;
private readonly CompressionOptions _compressionOptions;
private readonly CompressionService _compressionService;
private readonly CompressionTelemetry _compressionTelemetry;
private readonly StorageFormatMetadata _storageFormatMetadata;
private readonly MaintenanceOptions _maintenanceOptions;
private CDC.ChangeStreamDispatcher? _cdc;
// WAL cache: TransactionId → (PageId → PageData)
// Stores uncommitted writes for "Read Your Own Writes" isolation
@@ -32,18 +33,10 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
// WAL index cache: PageId → PageData (from latest committed transaction)
// Lazily populated on first read after commit
private readonly ConcurrentDictionary<uint, byte[]> _walIndex;
// Global lock for commit/checkpoint synchronization
private readonly SemaphoreSlim _commitLock = new(1, 1);
// Transaction Management
private readonly ConcurrentDictionary<ulong, Transaction> _activeTransactions;
private ulong _nextTransactionId;
private const long MaxWalSize = 4 * 1024 * 1024; // 4MB
/// <summary>
/// Initializes a new instance of the <see cref="StorageEngine"/> class.
/// Initializes a new instance of the <see cref="StorageEngine" /> class.
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="config">The page file configuration.</param>
@@ -55,13 +48,13 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
CompressionOptions? compressionOptions = null,
MaintenanceOptions? maintenanceOptions = null)
{
_compressionOptions = CompressionOptions.Normalize(compressionOptions);
_compressionService = new CompressionService();
_compressionTelemetry = new CompressionTelemetry();
_maintenanceOptions = maintenanceOptions ?? new MaintenanceOptions();
CompressionOptions = CompressionOptions.Normalize(compressionOptions);
CompressionService = new CompressionService();
CompressionTelemetry = new CompressionTelemetry();
MaintenanceOptions = maintenanceOptions ?? new MaintenanceOptions();
// Auto-derive WAL path
var walPath = Path.ChangeExtension(databasePath, ".wal");
string walPath = Path.ChangeExtension(databasePath, ".wal");
// Initialize storage infrastructure
_pageFile = new PageFile(databasePath, config);
@@ -72,14 +65,11 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
_walIndex = new ConcurrentDictionary<uint, byte[]>();
_activeTransactions = new ConcurrentDictionary<ulong, Transaction>();
_nextTransactionId = 1;
_storageFormatMetadata = InitializeStorageFormatMetadata();
StorageFormatMetadata = InitializeStorageFormatMetadata();
// Recover from WAL if exists (crash recovery or resume after close)
// This replays any committed transactions not yet checkpointed
if (_wal.GetCurrentSize() > 0)
{
Recover();
}
if (_wal.GetCurrentSize() > 0) Recover();
_ = ResumeCompactionIfNeeded();
@@ -92,58 +82,59 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
}
/// <summary>
/// Page size for this storage engine
/// Compression options for this engine instance.
/// </summary>
public CompressionOptions CompressionOptions { get; }
/// <summary>
/// Compression codec service for payload roundtrip operations.
/// </summary>
public CompressionService CompressionService { get; }
/// <summary>
/// Compression telemetry counters for this engine instance.
/// </summary>
public CompressionTelemetry CompressionTelemetry { get; }
/// <summary>
/// Gets storage format metadata associated with the current database.
/// </summary>
internal StorageFormatMetadata StorageFormatMetadata { get; }
/// <summary>
/// Gets the registered change stream dispatcher, if available.
/// </summary>
internal ChangeStreamDispatcher? Cdc { get; private set; }
/// <summary>
/// Page size for this storage engine
/// </summary>
public int PageSize => _pageFile.PageSize;
/// <summary>
/// Compression options for this engine instance.
/// </summary>
public CompressionOptions CompressionOptions => _compressionOptions;
/// <summary>
/// Compression codec service for payload roundtrip operations.
/// </summary>
public CompressionService CompressionService => _compressionService;
/// <summary>
/// Compression telemetry counters for this engine instance.
/// </summary>
public CompressionTelemetry CompressionTelemetry => _compressionTelemetry;
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats() => _compressionTelemetry.GetSnapshot();
/// <summary>
/// Gets storage format metadata associated with the current database.
/// </summary>
internal StorageFormatMetadata StorageFormatMetadata => _storageFormatMetadata;
/// <summary>
/// Checks if a page is currently being modified by another active transaction.
/// This is used to implement pessimistic locking for page allocation/selection.
/// Checks if a page is currently being modified by another active transaction.
/// This is used to implement pessimistic locking for page allocation/selection.
/// </summary>
/// <param name="pageId">The page identifier to check.</param>
/// <param name="excludingTxId">The transaction identifier to exclude from the check.</param>
/// <returns><see langword="true"/> if another transaction holds the page; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if another transaction holds the page; otherwise, <see langword="false" />.</returns>
public bool IsPageLocked(uint pageId, ulong excludingTxId)
{
foreach (var kvp in _walCache)
{
var txId = kvp.Key;
ulong txId = kvp.Key;
if (txId == excludingTxId) continue;
var txnPages = kvp.Value;
if (txnPages.ContainsKey(pageId))
return true;
}
return false;
}
/// <summary>
/// Disposes the storage engine and closes WAL.
/// Disposes the storage engine and closes WAL.
/// </summary>
public void Dispose()
{
@@ -151,13 +142,15 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
if (_activeTransactions != null)
{
foreach (var txn in _activeTransactions.Values)
{
try
{
RollbackTransaction(txn.TransactionId);
}
catch { /* Ignore errors during dispose */ }
}
catch
{
/* Ignore errors during dispose */
}
_activeTransactions.Clear();
}
@@ -168,32 +161,38 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
_commitLock?.Dispose();
}
/// <summary>
/// Registers the change stream dispatcher used for CDC notifications.
/// </summary>
/// <param name="cdc">The change stream dispatcher instance.</param>
internal void RegisterCdc(CDC.ChangeStreamDispatcher cdc)
/// <inheritdoc />
void IStorageEngine.RegisterCdc(ChangeStreamDispatcher cdc)
{
_cdc = cdc;
RegisterCdc(cdc);
}
/// <inheritdoc />
ChangeStreamDispatcher? IStorageEngine.Cdc => Cdc;
/// <inheritdoc />
CompressionOptions IStorageEngine.CompressionOptions => CompressionOptions;
/// <inheritdoc />
CompressionService IStorageEngine.CompressionService => CompressionService;
/// <inheritdoc />
CompressionTelemetry IStorageEngine.CompressionTelemetry => CompressionTelemetry;
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats()
{
return CompressionTelemetry.GetSnapshot();
}
/// <summary>
/// Gets the registered change stream dispatcher, if available.
/// Registers the change stream dispatcher used for CDC notifications.
/// </summary>
internal CDC.ChangeStreamDispatcher? Cdc => _cdc;
/// <inheritdoc />
void IStorageEngine.RegisterCdc(CDC.ChangeStreamDispatcher cdc) => RegisterCdc(cdc);
/// <inheritdoc />
CDC.ChangeStreamDispatcher? IStorageEngine.Cdc => _cdc;
/// <inheritdoc />
CompressionOptions IStorageEngine.CompressionOptions => _compressionOptions;
/// <inheritdoc />
CompressionService IStorageEngine.CompressionService => _compressionService;
/// <inheritdoc />
CompressionTelemetry IStorageEngine.CompressionTelemetry => _compressionTelemetry;
}
/// <param name="cdc">The change stream dispatcher instance.</param>
internal void RegisterCdc(ChangeStreamDispatcher cdc)
{
Cdc = cdc;
}
}

192
src/CBDD.Core/Storage/VectorPage.cs
View File

@@ -1,40 +1,40 @@
using System.Runtime.InteropServices;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page for storing HNSW Vector Index nodes.
/// Each page stores a fixed number of nodes based on vector dimensions and M.
/// </summary>
public struct VectorPage
{
// Layout:
// [PageHeader (32)]
// [Dimensions (4)]
// [MaxM (4)]
// [NodeSize (4)]
// [NodeCount (4)]
// [Nodes Data (Contiguous)...]
private const int DimensionsOffset = 32;
private const int MaxMOffset = 36;
private const int NodeSizeOffset = 40;
private const int NodeCountOffset = 44;
private const int DataOffset = 48;
using System.Buffers.Binary;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page for storing HNSW Vector Index nodes.
/// Each page stores a fixed number of nodes based on vector dimensions and M.
/// </summary>
public struct VectorPage
{
// Layout:
// [PageHeader (32)]
// [Dimensions (4)]
// [MaxM (4)]
// [NodeSize (4)]
// [NodeCount (4)]
// [Nodes Data (Contiguous)...]
private const int DimensionsOffset = 32;
private const int MaxMOffset = 36;
private const int NodeSizeOffset = 40;
private const int NodeCountOffset = 44;
private const int DataOffset = 48;
/// <summary>
/// Increments the node count stored in the vector page header.
/// Increments the node count stored in the vector page header.
/// </summary>
/// <param name="page">The page buffer.</param>
public static void IncrementNodeCount(Span<byte> page)
{
int count = GetNodeCount(page);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), count + 1);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), count + 1);
}
/// <summary>
/// Initializes a vector page with header metadata and sizing information.
/// Initializes a vector page with header metadata and sizing information.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="pageId">The page identifier.</param>
@@ -43,54 +43,60 @@ public struct VectorPage
public static void Initialize(Span<byte> page, uint pageId, int dimensions, int maxM)
{
var header = new PageHeader
{
PageId = pageId,
PageType = PageType.Vector,
FreeBytes = (ushort)(page.Length - DataOffset),
NextPageId = 0,
TransactionId = 0
};
header.WriteTo(page);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(DimensionsOffset), dimensions);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(MaxMOffset), maxM);
{
PageId = pageId,
PageType = PageType.Vector,
FreeBytes = (ushort)(page.Length - DataOffset),
NextPageId = 0,
TransactionId = 0
};
header.WriteTo(page);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(DimensionsOffset), dimensions);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(MaxMOffset), maxM);
// Node Size Calculation:
// Location (6) + MaxLevel (1) + Vector (dim * 4) + Links (maxM * 10 * 6) -- estimating 10 levels for simplicity
// Better: Node size is variable? No, let's keep it fixed per index configuration to avoid fragmentation.
// HNSW standard: level 0 has 2*M links, levels > 0 have M links.
// Max level is typically < 16. Let's reserve space for 16 levels.
int nodeSize = 6 + 1 + (dimensions * 4) + (maxM * (2 + 15) * 6);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeSizeOffset), nodeSize);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), 0);
int nodeSize = 6 + 1 + dimensions * 4 + maxM * (2 + 15) * 6;
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeSizeOffset), nodeSize);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), 0);
}
/// <summary>
/// Gets the number of nodes currently stored in the page.
/// Gets the number of nodes currently stored in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The node count.</returns>
public static int GetNodeCount(ReadOnlySpan<byte> page) =>
System.Buffers.Binary.BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeCountOffset));
public static int GetNodeCount(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeCountOffset));
}
/// <summary>
/// Gets the configured node size for the page.
/// Gets the configured node size for the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The node size in bytes.</returns>
public static int GetNodeSize(ReadOnlySpan<byte> page) =>
System.Buffers.Binary.BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeSizeOffset));
public static int GetNodeSize(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeSizeOffset));
}
/// <summary>
/// Gets the maximum number of nodes that can fit in the page.
/// Gets the maximum number of nodes that can fit in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The maximum node count.</returns>
public static int GetMaxNodes(ReadOnlySpan<byte> page) =>
(page.Length - DataOffset) / GetNodeSize(page);
public static int GetMaxNodes(ReadOnlySpan<byte> page)
{
return (page.Length - DataOffset) / GetNodeSize(page);
}
/// <summary>
/// Writes a node to the page at the specified index.
/// Writes a node to the page at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="nodeIndex">The zero-based node index.</param>
@@ -98,50 +104,52 @@ public struct VectorPage
/// <param name="maxLevel">The maximum graph level for the node.</param>
/// <param name="vector">The vector values to store.</param>
/// <param name="dimensions">The vector dimensionality.</param>
public static void WriteNode(Span<byte> page, int nodeIndex, DocumentLocation loc, int maxLevel, ReadOnlySpan<float> vector, int dimensions)
public static void WriteNode(Span<byte> page, int nodeIndex, DocumentLocation loc, int maxLevel,
ReadOnlySpan<float> vector, int dimensions)
{
int nodeSize = GetNodeSize(page);
int offset = DataOffset + (nodeIndex * nodeSize);
var nodeSpan = page.Slice(offset, nodeSize);
// 1. Document Location
loc.WriteTo(nodeSpan.Slice(0, 6));
// 2. Max Level
nodeSpan[6] = (byte)maxLevel;
// 3. Vector
var vectorSpan = MemoryMarshal.Cast<byte, float>(nodeSpan.Slice(7, dimensions * 4));
vector.CopyTo(vectorSpan);
// 4. Links (initialize with 0/empty)
// Links follow the vector. Level 0: 2*M links, Level 1..15: M links.
int offset = DataOffset + nodeIndex * nodeSize;
var nodeSpan = page.Slice(offset, nodeSize);
// 1. Document Location
loc.WriteTo(nodeSpan.Slice(0, 6));
// 2. Max Level
nodeSpan[6] = (byte)maxLevel;
// 3. Vector
var vectorSpan = MemoryMarshal.Cast<byte, float>(nodeSpan.Slice(7, dimensions * 4));
vector.CopyTo(vectorSpan);
// 4. Links (initialize with 0/empty)
// Links follow the vector. Level 0: 2*M links, Level 1..15: M links.
// For now, just ensure it's cleared or handled by the indexer.
}
/// <summary>
/// Reads node metadata and vector data from the page.
/// Reads node metadata and vector data from the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="nodeIndex">The zero-based node index.</param>
/// <param name="loc">When this method returns, contains the node document location.</param>
/// <param name="maxLevel">When this method returns, contains the node max level.</param>
/// <param name="vector">The destination span for vector values.</param>
public static void ReadNodeData(ReadOnlySpan<byte> page, int nodeIndex, out DocumentLocation loc, out int maxLevel, Span<float> vector)
public static void ReadNodeData(ReadOnlySpan<byte> page, int nodeIndex, out DocumentLocation loc, out int maxLevel,
Span<float> vector)
{
int nodeSize = GetNodeSize(page);
int offset = DataOffset + (nodeIndex * nodeSize);
var nodeSpan = page.Slice(offset, nodeSize);
loc = DocumentLocation.ReadFrom(nodeSpan.Slice(0, 6));
int offset = DataOffset + nodeIndex * nodeSize;
var nodeSpan = page.Slice(offset, nodeSize);
loc = DocumentLocation.ReadFrom(nodeSpan.Slice(0, 6));
maxLevel = nodeSpan[6];
var vectorSource = MemoryMarshal.Cast<byte, float>(nodeSpan.Slice(7, vector.Length * 4));
var vectorSource = MemoryMarshal.Cast<byte, float>(nodeSpan.Slice(7, vector.Length * 4));
vectorSource.CopyTo(vector);
}
/// <summary>
/// Gets the span that stores links for a node at a specific level.
/// Gets the span that stores links for a node at a specific level.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="nodeIndex">The zero-based node index.</param>
@@ -152,23 +160,19 @@ public struct VectorPage
public static Span<byte> GetLinksSpan(Span<byte> page, int nodeIndex, int level, int dimensions, int maxM)
{
int nodeSize = GetNodeSize(page);
int nodeOffset = DataOffset + (nodeIndex * nodeSize);
int nodeOffset = DataOffset + nodeIndex * nodeSize;
// Link offset: Location(6) + MaxLevel(1) + Vector(dim*4)
int linkBaseOffset = nodeOffset + 7 + (dimensions * 4);
int linkBaseOffset = nodeOffset + 7 + dimensions * 4;
int levelOffset;
if (level == 0)
{
levelOffset = 0;
}
else
{
// Level 0 has 2*M links
levelOffset = (2 * maxM * 6) + ((level - 1) * maxM * 6);
}
int count = (level == 0) ? (2 * maxM) : maxM;
return page.Slice(linkBaseOffset + levelOffset, count * 6);
}
}
int levelOffset;
if (level == 0)
levelOffset = 0;
else
// Level 0 has 2*M links
levelOffset = 2 * maxM * 6 + (level - 1) * maxM * 6;
int count = level == 0 ? 2 * maxM : maxM;
return page.Slice(linkBaseOffset + levelOffset, count * 6);
}
}

30
src/CBDD.Core/Transactions/CheckpointMode.cs
View File

@@ -1,39 +1,39 @@
namespace ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Defines checkpoint modes for WAL (Write-Ahead Log) checkpointing.
/// Similar to SQLite's checkpoint strategies.
/// Defines checkpoint modes for WAL (Write-Ahead Log) checkpointing.
/// Similar to SQLite's checkpoint strategies.
/// </summary>
public enum CheckpointMode
{
/// <summary>
/// Passive checkpoint: non-blocking, best-effort transfer from WAL to database.
/// If the checkpoint lock is busy, the operation is skipped.
/// WAL content is preserved and a checkpoint marker is appended when work is applied.
/// Passive checkpoint: non-blocking, best-effort transfer from WAL to database.
/// If the checkpoint lock is busy, the operation is skipped.
/// WAL content is preserved and a checkpoint marker is appended when work is applied.
/// </summary>
Passive = 0,
/// <summary>
/// Full checkpoint: waits for the checkpoint lock, transfers committed pages to
/// the page file, and preserves WAL content by appending a checkpoint marker.
/// Full checkpoint: waits for the checkpoint lock, transfers committed pages to
/// the page file, and preserves WAL content by appending a checkpoint marker.
/// </summary>
Full = 1,
/// <summary>
/// Truncate checkpoint: same as <see cref="Full"/> but truncates WAL after
/// successfully applying committed pages. Use this to reclaim disk space.
/// Truncate checkpoint: same as <see cref="Full" /> but truncates WAL after
/// successfully applying committed pages. Use this to reclaim disk space.
/// </summary>
Truncate = 2,
/// <summary>
/// Restart checkpoint: same as <see cref="Truncate"/> and then reinitializes
/// the WAL stream for a fresh writer session.
/// Restart checkpoint: same as <see cref="Truncate" /> and then reinitializes
/// the WAL stream for a fresh writer session.
/// </summary>
Restart = 3
}
/// <summary>
/// Result of a checkpoint execution.
/// Result of a checkpoint execution.
/// </summary>
/// <param name="Mode">Requested checkpoint mode.</param>
/// <param name="Executed">True when checkpoint logic ran; false when skipped (for passive mode contention).</param>
@@ -49,4 +49,4 @@ public readonly record struct CheckpointResult(
long WalBytesBefore,
long WalBytesAfter,
bool Truncated,
bool Restarted);
bool Restarted);

46
src/CBDD.Core/Transactions/ITransaction.cs
View File

@@ -1,66 +1,62 @@
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Public interface for database transactions.
/// Allows user-controlled transaction boundaries for batch operations.
/// Public interface for database transactions.
/// Allows user-controlled transaction boundaries for batch operations.
/// </summary>
/// <example>
/// using (var txn = collection.BeginTransaction())
/// {
/// using (var txn = collection.BeginTransaction())
/// {
/// collection.Insert(entity1, txn);
/// collection.Insert(entity2, txn);
/// txn.Commit();
/// }
/// }
/// </example>
public interface ITransaction : IDisposable
{
/// <summary>
/// Unique transaction identifier
/// Unique transaction identifier
/// </summary>
ulong TransactionId { get; }
/// <summary>
/// Current state of the transaction
/// Current state of the transaction
/// </summary>
TransactionState State { get; }
/// <summary>
/// Commits the transaction, making all changes permanent.
/// Must be called before Dispose() to persist changes.
/// Commits the transaction, making all changes permanent.
/// Must be called before Dispose() to persist changes.
/// </summary>
void Commit();
/// <summary>
/// Asynchronously commits the transaction, making all changes permanent.
/// </summary>
/// <param name="ct">The cancellation token.</param>
Task CommitAsync(CancellationToken ct = default);
/// <summary>
/// Asynchronously commits the transaction, making all changes permanent.
/// </summary>
/// <param name="ct">The cancellation token.</param>
Task CommitAsync(CancellationToken ct = default);
/// <summary>
/// Rolls back the transaction, discarding all changes.
/// Called automatically on Dispose() if Commit() was not called.
/// Rolls back the transaction, discarding all changes.
/// Called automatically on Dispose() if Commit() was not called.
/// </summary>
void Rollback();
/// <summary>
/// Adds a write operation to the current batch or transaction.
/// Adds a write operation to the current batch or transaction.
/// </summary>
/// <param name="operation">The write operation to add. Cannot be null.</param>
void AddWrite(WriteOperation operation);
/// <summary>
/// Prepares the object for use by performing any necessary initialization or setup.
/// Prepares the object for use by performing any necessary initialization or setup.
/// </summary>
/// <returns>true if the preparation was successful; otherwise, false.</returns>
bool Prepare();
/// <summary>
/// Event triggered when the transaction acts rollback.
/// Useful for restoring in-memory state (like ID maps).
/// Event triggered when the transaction acts rollback.
/// Useful for restoring in-memory state (like ID maps).
/// </summary>
event Action? OnRollback;
}
}

32
src/CBDD.Core/Transactions/ITransactionHolder.cs
View File

@@ -1,26 +1,32 @@
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Defines a contract for managing and providing access to the current transaction context.
/// Defines a contract for managing and providing access to the current transaction context.
/// </summary>
/// <remarks>Implementations of this interface are responsible for tracking the current transaction and starting a
/// new one if none exists. This is typically used in scenarios where transactional consistency is required across
/// multiple operations.</remarks>
/// <remarks>
/// Implementations of this interface are responsible for tracking the current transaction and starting a
/// new one if none exists. This is typically used in scenarios where transactional consistency is required across
/// multiple operations.
/// </remarks>
public interface ITransactionHolder
{
/// <summary>
/// Gets the current transaction if one exists; otherwise, starts a new transaction.
/// Gets the current transaction if one exists; otherwise, starts a new transaction.
/// </summary>
/// <remarks>Use this method to ensure that a transaction context is available for the current operation.
/// If a transaction is already in progress, it is returned; otherwise, a new transaction is started and returned.
/// The caller is responsible for managing the transaction's lifetime as appropriate.</remarks>
/// <returns>An <see cref="ITransaction"/> representing the current transaction, or a new transaction if none is active.</returns>
/// <remarks>
/// Use this method to ensure that a transaction context is available for the current operation.
/// If a transaction is already in progress, it is returned; otherwise, a new transaction is started and returned.
/// The caller is responsible for managing the transaction's lifetime as appropriate.
/// </remarks>
/// <returns>An <see cref="ITransaction" /> representing the current transaction, or a new transaction if none is active.</returns>
ITransaction GetCurrentTransactionOrStart();
/// <summary>
/// Gets the current transaction if one exists; otherwise, starts a new transaction asynchronously.
/// Gets the current transaction if one exists; otherwise, starts a new transaction asynchronously.
/// </summary>
/// <returns>A task that represents the asynchronous operation. The task result contains an <see cref="ITransaction"/>
/// representing the current or newly started transaction.</returns>
/// <returns>
/// A task that represents the asynchronous operation. The task result contains an <see cref="ITransaction" />
/// representing the current or newly started transaction.
/// </returns>
Task<ITransaction> GetCurrentTransactionOrStartAsync();
}
}

338
src/CBDD.Core/Transactions/Transaction.cs
View File

@@ -1,257 +1,241 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Represents a transaction with ACID properties.
/// Uses MVCC (Multi-Version Concurrency Control) for isolation.
/// Represents a transaction with ACID properties.
/// Uses MVCC (Multi-Version Concurrency Control) for isolation.
/// </summary>
public sealed class Transaction : ITransaction
{
private readonly ulong _transactionId;
private readonly IsolationLevel _isolationLevel;
private readonly DateTime _startTime;
private readonly List<InternalChangeEvent> _pendingChanges = new();
private readonly StorageEngine _storage;
private readonly List<CDC.InternalChangeEvent> _pendingChanges = new();
private TransactionState _state;
private bool _disposed;
/// <summary>
/// Initializes a new transaction.
/// </summary>
/// <param name="transactionId">The unique transaction identifier.</param>
/// <param name="storage">The storage engine used by this transaction.</param>
/// <param name="isolationLevel">The transaction isolation level.</param>
public Transaction(ulong transactionId,
StorageEngine storage,
IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
/// <summary>
/// Initializes a new transaction.
/// </summary>
/// <param name="transactionId">The unique transaction identifier.</param>
/// <param name="storage">The storage engine used by this transaction.</param>
/// <param name="isolationLevel">The transaction isolation level.</param>
public Transaction(ulong transactionId,
StorageEngine storage,
IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
{
_transactionId = transactionId;
TransactionId = transactionId;
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_isolationLevel = isolationLevel;
_startTime = DateTime.UtcNow;
_state = TransactionState.Active;
IsolationLevel = isolationLevel;
StartTime = DateTime.UtcNow;
State = TransactionState.Active;
}
/// <summary>
/// Adds a pending CDC change to be published after commit.
/// </summary>
/// <param name="change">The change event to buffer.</param>
internal void AddChange(CDC.InternalChangeEvent change)
{
_pendingChanges.Add(change);
}
/// <summary>
/// Gets the unique transaction identifier.
/// </summary>
public ulong TransactionId => _transactionId;
/// <summary>
/// Gets the current transaction state.
/// </summary>
public TransactionState State => _state;
/// <summary>
/// Gets the configured transaction isolation level.
/// </summary>
public IsolationLevel IsolationLevel => _isolationLevel;
/// <summary>
/// Gets the UTC start time of the transaction.
/// </summary>
public DateTime StartTime => _startTime;
/// <summary>
/// Gets the configured transaction isolation level.
/// </summary>
public IsolationLevel IsolationLevel { get; }
/// <summary>
/// Adds a write operation to the transaction's write set.
/// NOTE: Makes a defensive copy of the data to ensure memory safety.
/// This allocation is necessary because the caller may return the buffer to a pool.
/// </summary>
/// <param name="operation">The write operation to add.</param>
public void AddWrite(WriteOperation operation)
/// Gets the UTC start time of the transaction.
/// </summary>
public DateTime StartTime { get; }
/// <summary>
/// Gets the unique transaction identifier.
/// </summary>
public ulong TransactionId { get; }
/// <summary>
/// Gets the current transaction state.
/// </summary>
public TransactionState State { get; private set; }
/// <summary>
/// Adds a write operation to the transaction's write set.
/// NOTE: Makes a defensive copy of the data to ensure memory safety.
/// This allocation is necessary because the caller may return the buffer to a pool.
/// </summary>
/// <param name="operation">The write operation to add.</param>
public void AddWrite(WriteOperation operation)
{
if (_state != TransactionState.Active)
throw new InvalidOperationException($"Cannot add writes to transaction in state {_state}");
if (State != TransactionState.Active)
throw new InvalidOperationException($"Cannot add writes to transaction in state {State}");
// Defensive copy: necessary to prevent use-after-return if caller uses pooled buffers
byte[] ownedCopy = operation.NewValue.ToArray();
// StorageEngine gestisce tutte le scritture transazionali
_storage.WritePage(operation.PageId, _transactionId, ownedCopy);
_storage.WritePage(operation.PageId, TransactionId, ownedCopy);
}
/// <summary>
/// Prepares the transaction for commit (2PC first phase)
/// Prepares the transaction for commit (2PC first phase)
/// </summary>
public bool Prepare()
{
if (_state != TransactionState.Active)
if (State != TransactionState.Active)
return false;
_state = TransactionState.Preparing;
// StorageEngine handles WAL writes
return _storage.PrepareTransaction(_transactionId);
State = TransactionState.Preparing;
// StorageEngine handles WAL writes
return _storage.PrepareTransaction(TransactionId);
}
/// <summary>
/// Commits the transaction.
/// Writes to WAL for durability and moves data to committed buffer.
/// Pages remain in memory until CheckpointManager writes them to disk.
/// Commits the transaction.
/// Writes to WAL for durability and moves data to committed buffer.
/// Pages remain in memory until CheckpointManager writes them to disk.
/// </summary>
public void Commit()
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
// StorageEngine handles WAL writes and buffer management
_storage.CommitTransaction(_transactionId);
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
_state = TransactionState.Committed;
// StorageEngine handles WAL writes and buffer management
_storage.CommitTransaction(TransactionId);
State = TransactionState.Committed;
// Publish CDC events after successful commit
if (_pendingChanges.Count > 0 && _storage.Cdc != null)
{
foreach (var change in _pendingChanges)
{
_storage.Cdc.Publish(change);
}
}
}
/// <summary>
/// Asynchronously commits the transaction.
/// </summary>
/// <param name="ct">A cancellation token.</param>
public async Task CommitAsync(CancellationToken ct = default)
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
// StorageEngine handles WAL writes and buffer management
await _storage.CommitTransactionAsync(_transactionId, ct);
_state = TransactionState.Committed;
// Publish CDC events after successful commit
if (_pendingChanges.Count > 0 && _storage.Cdc != null)
{
foreach (var change in _pendingChanges)
{
_storage.Cdc.Publish(change);
}
}
}
/// <summary>
/// Marks the transaction as committed without writing to PageFile.
/// Used by TransactionManager with lazy checkpointing.
/// Asynchronously commits the transaction.
/// </summary>
internal void MarkCommitted()
/// <param name="ct">A cancellation token.</param>
public async Task CommitAsync(CancellationToken ct = default)
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine marks transaction as committed and moves to committed buffer
_storage.MarkTransactionCommitted(_transactionId);
_state = TransactionState.Committed;
// StorageEngine handles WAL writes and buffer management
await _storage.CommitTransactionAsync(TransactionId, ct);
State = TransactionState.Committed;
// Publish CDC events after successful commit
if (_pendingChanges.Count > 0 && _storage.Cdc != null)
foreach (var change in _pendingChanges)
_storage.Cdc.Publish(change);
}
/// <summary>
/// Rolls back the transaction (discards all writes)
/// Rolls back the transaction (discards all writes)
/// </summary>
public event Action? OnRollback;
/// <summary>
/// Rolls back the transaction and discards pending writes.
/// </summary>
public void Rollback()
/// <summary>
/// Rolls back the transaction and discards pending writes.
/// </summary>
public void Rollback()
{
if (_state == TransactionState.Committed)
if (State == TransactionState.Committed)
throw new InvalidOperationException("Cannot rollback committed transaction");
_pendingChanges.Clear();
_storage.RollbackTransaction(_transactionId);
_state = TransactionState.Aborted;
_storage.RollbackTransaction(TransactionId);
State = TransactionState.Aborted;
OnRollback?.Invoke();
}
/// <summary>
/// Releases transaction resources and rolls back if still active.
/// </summary>
public void Dispose()
/// <summary>
/// Releases transaction resources and rolls back if still active.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
if (_state == TransactionState.Active || _state == TransactionState.Preparing)
{
if (State == TransactionState.Active || State == TransactionState.Preparing)
// Auto-rollback if not committed
Rollback();
}
_disposed = true;
GC.SuppressFinalize(this);
}
/// <summary>
/// Adds a pending CDC change to be published after commit.
/// </summary>
/// <param name="change">The change event to buffer.</param>
internal void AddChange(InternalChangeEvent change)
{
_pendingChanges.Add(change);
}
/// <summary>
/// Marks the transaction as committed without writing to PageFile.
/// Used by TransactionManager with lazy checkpointing.
/// </summary>
internal void MarkCommitted()
{
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine marks transaction as committed and moves to committed buffer
_storage.MarkTransactionCommitted(TransactionId);
State = TransactionState.Committed;
}
}
/// <summary>
/// Represents a write operation in a transaction.
/// Optimized to avoid allocations by using ReadOnlyMemory instead of byte[].
/// Represents a write operation in a transaction.
/// Optimized to avoid allocations by using ReadOnlyMemory instead of byte[].
/// </summary>
public struct WriteOperation
{
/// <summary>
/// Gets or sets the identifier of the affected document.
/// </summary>
public ObjectId DocumentId { get; set; }
/// <summary>
/// Gets or sets the new serialized value.
/// </summary>
public ReadOnlyMemory<byte> NewValue { get; set; }
/// <summary>
/// Gets or sets the target page identifier.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Gets or sets the operation type.
/// </summary>
public OperationType Type { get; set; }
/// <summary>
/// Initializes a new write operation.
/// </summary>
/// <param name="documentId">The identifier of the affected document.</param>
/// <param name="newValue">The new serialized value.</param>
/// <param name="pageId">The target page identifier.</param>
/// <param name="type">The operation type.</param>
public WriteOperation(ObjectId documentId, ReadOnlyMemory<byte> newValue, uint pageId, OperationType type)
{
DocumentId = documentId;
public struct WriteOperation
{
/// <summary>
/// Gets or sets the identifier of the affected document.
/// </summary>
public ObjectId DocumentId { get; set; }
/// <summary>
/// Gets or sets the new serialized value.
/// </summary>
public ReadOnlyMemory<byte> NewValue { get; set; }
/// <summary>
/// Gets or sets the target page identifier.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Gets or sets the operation type.
/// </summary>
public OperationType Type { get; set; }
/// <summary>
/// Initializes a new write operation.
/// </summary>
/// <param name="documentId">The identifier of the affected document.</param>
/// <param name="newValue">The new serialized value.</param>
/// <param name="pageId">The target page identifier.</param>
/// <param name="type">The operation type.</param>
public WriteOperation(ObjectId documentId, ReadOnlyMemory<byte> newValue, uint pageId, OperationType type)
{
DocumentId = documentId;
NewValue = newValue;
PageId = pageId;
Type = type;
}
// Backward compatibility constructor
/// <summary>
/// Initializes a new write operation from a byte array payload.
/// </summary>
/// <param name="documentId">The identifier of the affected document.</param>
/// <param name="newValue">The new serialized value.</param>
/// <param name="pageId">The target page identifier.</param>
/// <param name="type">The operation type.</param>
public WriteOperation(ObjectId documentId, byte[] newValue, uint pageId, OperationType type)
{
DocumentId = documentId;
}
// Backward compatibility constructor
/// <summary>
/// Initializes a new write operation from a byte array payload.
/// </summary>
/// <param name="documentId">The identifier of the affected document.</param>
/// <param name="newValue">The new serialized value.</param>
/// <param name="pageId">The target page identifier.</param>
/// <param name="type">The operation type.</param>
public WriteOperation(ObjectId documentId, byte[] newValue, uint pageId, OperationType type)
{
DocumentId = documentId;
NewValue = newValue;
PageId = pageId;
Type = type;
@@ -259,7 +243,7 @@ public struct WriteOperation
}
/// <summary>
/// Type of write operation
/// Type of write operation
/// </summary>
public enum OperationType : byte
{
@@ -267,4 +251,4 @@ public enum OperationType : byte
Update = 2,
Delete = 3,
AllocatePage = 4
}
}

30
src/CBDD.Core/Transactions/TransactionState.cs
View File

@@ -1,37 +1,37 @@
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Transaction states
/// Transaction states
/// </summary>
public enum TransactionState : byte
{
/// <summary>Transaction is active and can accept operations</summary>
Active = 1,
Active = 1,
/// <summary>Transaction is preparing to commit</summary>
Preparing = 2,
Preparing = 2,
/// <summary>Transaction committed successfully</summary>
Committed = 3,
Committed = 3,
/// <summary>Transaction was rolled back</summary>
Aborted = 4
}
/// <summary>
/// Transaction isolation levels
/// Transaction isolation levels
/// </summary>
public enum IsolationLevel : byte
{
/// <summary>Read uncommitted data</summary>
ReadUncommitted = 0,
ReadUncommitted = 0,
/// <summary>Read only committed data (default)</summary>
ReadCommitted = 1,
ReadCommitted = 1,
/// <summary>Repeatable reads</summary>
RepeatableRead = 2,
RepeatableRead = 2,
/// <summary>Serializable (full isolation)</summary>
Serializable = 3
}
}

1192
src/CBDD.Core/Transactions/WriteAheadLog.cs
View File

File diff suppressed because it is too large Load Diff

62
src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj
View File

@@ -1,38 +1,38 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Core</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Core</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PackageId>ZB.MOM.WW.CBDD.Core</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine Core Library for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Core</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Core</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<ItemGroup>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj" />
</ItemGroup>
<PackageId>ZB.MOM.WW.CBDD.Core</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine Core Library for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
<_Parameter1>ZB.MOM.WW.CBDD.Tests</_Parameter1>
</AssemblyAttribute>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj"/>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup>
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
<_Parameter1>ZB.MOM.WW.CBDD.Tests</_Parameter1>
</AssemblyAttribute>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

474
src/CBDD.SourceGenerators/Analysis/EntityAnalyzer.cs
View File

@@ -1,261 +1,257 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
namespace ZB.MOM.WW.CBDD.SourceGenerators
{
public static class EntityAnalyzer
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
namespace ZB.MOM.WW.CBDD.SourceGenerators;
public static class EntityAnalyzer
{
/// <summary>
/// Analyzes an entity symbol and builds source-generation metadata.
/// </summary>
/// <param name="entityType">The entity type symbol to analyze.</param>
/// <param name="semanticModel">The semantic model for symbol and syntax analysis.</param>
/// <returns>The analyzed entity metadata.</returns>
public static EntityInfo Analyze(INamedTypeSymbol entityType, SemanticModel semanticModel)
{
/// <summary>
/// Analyzes an entity symbol and builds source-generation metadata.
/// </summary>
/// <param name="entityType">The entity type symbol to analyze.</param>
/// <param name="semanticModel">The semantic model for symbol and syntax analysis.</param>
/// <returns>The analyzed entity metadata.</returns>
public static EntityInfo Analyze(INamedTypeSymbol entityType, SemanticModel semanticModel)
var entityInfo = new EntityInfo
{
var entityInfo = new EntityInfo
{
Name = entityType.Name,
Namespace = entityType.ContainingNamespace.ToDisplayString(),
FullTypeName = SyntaxHelper.GetFullName(entityType),
CollectionName = entityType.Name.ToLowerInvariant() + "s"
};
var tableAttr = AttributeHelper.GetAttribute(entityType, "Table");
if (tableAttr != null)
{
var tableName = tableAttr.ConstructorArguments.Length > 0 ? tableAttr.ConstructorArguments[0].Value?.ToString() : null;
var schema = AttributeHelper.GetNamedArgumentValue(tableAttr, "Schema");
Name = entityType.Name,
Namespace = entityType.ContainingNamespace.ToDisplayString(),
FullTypeName = SyntaxHelper.GetFullName(entityType),
CollectionName = entityType.Name.ToLowerInvariant() + "s"
};
var collectionName = !string.IsNullOrEmpty(tableName) ? tableName! : entityInfo.Name;
if (!string.IsNullOrEmpty(schema))
{
collectionName = $"{schema}.{collectionName}";
}
entityInfo.CollectionName = collectionName;
}
var tableAttr = AttributeHelper.GetAttribute(entityType, "Table");
if (tableAttr != null)
{
string? tableName = tableAttr.ConstructorArguments.Length > 0
? tableAttr.ConstructorArguments[0].Value?.ToString()
: null;
string? schema = AttributeHelper.GetNamedArgumentValue(tableAttr, "Schema");
// Analyze properties of the root entity
AnalyzeProperties(entityType, entityInfo.Properties);
// Check if entity needs reflection-based deserialization
// Include properties with private setters or init-only setters (which can't be set outside initializers)
entityInfo.HasPrivateSetters = entityInfo.Properties.Any(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter);
string collectionName = !string.IsNullOrEmpty(tableName) ? tableName! : entityInfo.Name;
if (!string.IsNullOrEmpty(schema)) collectionName = $"{schema}.{collectionName}";
entityInfo.CollectionName = collectionName;
}
// Check if entity has public parameterless constructor
var hasPublicParameterlessConstructor = entityType.Constructors
.Any(c => c.DeclaredAccessibility == Accessibility.Public && c.Parameters.Length == 0);
entityInfo.HasPrivateOrNoConstructor = !hasPublicParameterlessConstructor;
// Analyze nested types recursively
// We use a dictionary for nested types to ensure uniqueness by name
var analyzedTypes = new HashSet<string>();
AnalyzeNestedTypesRecursive(entityInfo.Properties, entityInfo.NestedTypes, semanticModel, analyzedTypes, 1, 3);
// Analyze properties of the root entity
AnalyzeProperties(entityType, entityInfo.Properties);
// Determine ID property
// entityInfo.IdProperty is computed from Properties.FirstOrDefault(p => p.IsKey)
// Check if entity needs reflection-based deserialization
// Include properties with private setters or init-only setters (which can't be set outside initializers)
entityInfo.HasPrivateSetters =
entityInfo.Properties.Any(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter);
if (entityInfo.IdProperty == null)
{
// Fallback to convention: property named "Id"
var idProp = entityInfo.Properties.FirstOrDefault(p => p.Name == "Id");
if (idProp != null)
{
idProp.IsKey = true;
}
}
// Check if entity has public parameterless constructor
bool hasPublicParameterlessConstructor = entityType.Constructors
.Any(c => c.DeclaredAccessibility == Accessibility.Public && c.Parameters.Length == 0);
entityInfo.HasPrivateOrNoConstructor = !hasPublicParameterlessConstructor;
// Check for AutoId (int/long keys)
if (entityInfo.IdProperty != null)
{
var idType = entityInfo.IdProperty.TypeName.TrimEnd('?');
if (idType == "int" || idType == "Int32" || idType == "long" || idType == "Int64")
{
entityInfo.AutoId = true;
}
}
// Analyze nested types recursively
// We use a dictionary for nested types to ensure uniqueness by name
var analyzedTypes = new HashSet<string>();
AnalyzeNestedTypesRecursive(entityInfo.Properties, entityInfo.NestedTypes, semanticModel, analyzedTypes, 1, 3);
return entityInfo;
}
private static void AnalyzeProperties(INamedTypeSymbol typeSymbol, List<PropertyInfo> properties)
{
// Collect properties from the entire inheritance hierarchy
var seenProperties = new HashSet<string>();
var currentType = typeSymbol;
// Determine ID property
// entityInfo.IdProperty is computed from Properties.FirstOrDefault(p => p.IsKey)
while (currentType != null && currentType.SpecialType != SpecialType.System_Object)
{
var sourceProps = currentType.GetMembers()
.OfType<IPropertySymbol>()
.Where(p => p.DeclaredAccessibility == Accessibility.Public && !p.IsStatic);
if (entityInfo.IdProperty == null)
{
// Fallback to convention: property named "Id"
var idProp = entityInfo.Properties.FirstOrDefault(p => p.Name == "Id");
if (idProp != null) idProp.IsKey = true;
}
foreach (var prop in sourceProps)
{
// Skip if already seen (overridden property in derived class takes precedence)
if (!seenProperties.Add(prop.Name))
continue;
// Check for AutoId (int/long keys)
if (entityInfo.IdProperty != null)
{
string idType = entityInfo.IdProperty.TypeName.TrimEnd('?');
if (idType == "int" || idType == "Int32" || idType == "long" || idType == "Int64") entityInfo.AutoId = true;
}
if (AttributeHelper.ShouldIgnore(prop))
continue;
return entityInfo;
}
// Skip computed getter-only properties (no setter, no backing field)
bool isReadOnlyGetter = prop.SetMethod == null && !SyntaxHelper.HasBackingField(prop);
if (isReadOnlyGetter)
continue;
private static void AnalyzeProperties(INamedTypeSymbol typeSymbol, List<PropertyInfo> properties)
{
// Collect properties from the entire inheritance hierarchy
var seenProperties = new HashSet<string>();
var currentType = typeSymbol;
var columnAttr = AttributeHelper.GetAttribute(prop, "Column");
var bsonFieldName = AttributeHelper.GetAttributeStringValue(prop, "BsonProperty") ??
while (currentType != null && currentType.SpecialType != SpecialType.System_Object)
{
var sourceProps = currentType.GetMembers()
.OfType<IPropertySymbol>()
.Where(p => p.DeclaredAccessibility == Accessibility.Public && !p.IsStatic);
foreach (var prop in sourceProps)
{
// Skip if already seen (overridden property in derived class takes precedence)
if (!seenProperties.Add(prop.Name))
continue;
if (AttributeHelper.ShouldIgnore(prop))
continue;
// Skip computed getter-only properties (no setter, no backing field)
bool isReadOnlyGetter = prop.SetMethod == null && !SyntaxHelper.HasBackingField(prop);
if (isReadOnlyGetter)
continue;
var columnAttr = AttributeHelper.GetAttribute(prop, "Column");
string? bsonFieldName = AttributeHelper.GetAttributeStringValue(prop, "BsonProperty") ??
AttributeHelper.GetAttributeStringValue(prop, "JsonPropertyName");
if (bsonFieldName == null && columnAttr != null)
{
bsonFieldName = columnAttr.ConstructorArguments.Length > 0 ? columnAttr.ConstructorArguments[0].Value?.ToString() : null;
}
if (bsonFieldName == null && columnAttr != null)
bsonFieldName = columnAttr.ConstructorArguments.Length > 0
? columnAttr.ConstructorArguments[0].Value?.ToString()
: null;
var propInfo = new PropertyInfo
{
Name = prop.Name,
TypeName = SyntaxHelper.GetTypeName(prop.Type),
BsonFieldName = bsonFieldName ?? prop.Name.ToLowerInvariant(),
ColumnTypeName = columnAttr != null ? AttributeHelper.GetNamedArgumentValue(columnAttr, "TypeName") : null,
IsNullable = SyntaxHelper.IsNullableType(prop.Type),
IsKey = AttributeHelper.IsKey(prop),
IsRequired = AttributeHelper.HasAttribute(prop, "Required"),
HasPublicSetter = prop.SetMethod?.DeclaredAccessibility == Accessibility.Public,
HasInitOnlySetter = prop.SetMethod?.IsInitOnly == true,
HasAnySetter = prop.SetMethod != null,
IsReadOnlyGetter = isReadOnlyGetter,
BackingFieldName = (prop.SetMethod?.DeclaredAccessibility != Accessibility.Public)
? $"<{prop.Name}>k__BackingField"
: null
};
// MaxLength / MinLength
propInfo.MaxLength = AttributeHelper.GetAttributeIntValue(prop, "MaxLength");
propInfo.MinLength = AttributeHelper.GetAttributeIntValue(prop, "MinLength");
var stringLengthAttr = AttributeHelper.GetAttribute(prop, "StringLength");
if (stringLengthAttr != null)
{
if (stringLengthAttr.ConstructorArguments.Length > 0 && stringLengthAttr.ConstructorArguments[0].Value is int max)
propInfo.MaxLength = max;
var minLenStr = AttributeHelper.GetNamedArgumentValue(stringLengthAttr, "MinimumLength");
if (int.TryParse(minLenStr, out var min))
propInfo.MinLength = min;
}
// Range
var rangeAttr = AttributeHelper.GetAttribute(prop, "Range");
if (rangeAttr != null && rangeAttr.ConstructorArguments.Length >= 2)
{
if (rangeAttr.ConstructorArguments[0].Value is double dmin) propInfo.RangeMin = dmin;
else if (rangeAttr.ConstructorArguments[0].Value is int imin) propInfo.RangeMin = (double)imin;
if (rangeAttr.ConstructorArguments[1].Value is double dmax) propInfo.RangeMax = dmax;
else if (rangeAttr.ConstructorArguments[1].Value is int imax) propInfo.RangeMax = (double)imax;
}
if (SyntaxHelper.IsCollectionType(prop.Type, out var itemType))
{
propInfo.IsCollection = true;
propInfo.IsArray = prop.Type is IArrayTypeSymbol;
// Determine concrete collection type name
propInfo.CollectionConcreteTypeName = SyntaxHelper.GetTypeName(prop.Type);
if (itemType != null)
{
propInfo.CollectionItemType = SyntaxHelper.GetTypeName(itemType);
// Check if collection item is nested object
if (SyntaxHelper.IsNestedObjectType(itemType))
{
propInfo.IsCollectionItemNested = true;
propInfo.NestedTypeName = itemType.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)itemType);
}
}
}
// Check for Nested Object
else if (SyntaxHelper.IsNestedObjectType(prop.Type))
{
propInfo.IsNestedObject = true;
propInfo.NestedTypeName = prop.Type.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)prop.Type);
}
properties.Add(propInfo);
}
currentType = currentType.BaseType;
}
}
private static void AnalyzeNestedTypesRecursive(
List<PropertyInfo> properties,
Dictionary<string, NestedTypeInfo> targetNestedTypes,
SemanticModel semanticModel,
HashSet<string> analyzedTypes,
int currentDepth,
int maxDepth)
{
if (currentDepth > maxDepth) return;
// Identify properties that reference nested types (either directly or via collection)
var nestedProps = properties
.Where(p => (p.IsNestedObject || p.IsCollectionItemNested) && !string.IsNullOrEmpty(p.NestedTypeFullName))
.ToList();
foreach (var prop in nestedProps)
{
var fullTypeName = prop.NestedTypeFullName!;
var simpleName = prop.NestedTypeName!;
// Avoid cycles
if (analyzedTypes.Contains(fullTypeName)) continue;
// If already in target list, skip
if (targetNestedTypes.ContainsKey(fullTypeName)) continue;
// Try to find the symbol
INamedTypeSymbol? nestedTypeSymbol = null;
// Try by full name
nestedTypeSymbol = semanticModel.Compilation.GetTypeByMetadataName(fullTypeName);
// If not found, try to resolve via semantic model (might be in the same compilation)
if (nestedTypeSymbol == null)
var propInfo = new PropertyInfo
{
// This is more complex, but usually fullTypeName from ToDisplayString() is traceable.
// For now, let's assume GetTypeByMetadataName works for fully qualified names.
}
Name = prop.Name,
TypeName = SyntaxHelper.GetTypeName(prop.Type),
BsonFieldName = bsonFieldName ?? prop.Name.ToLowerInvariant(),
ColumnTypeName = columnAttr != null
? AttributeHelper.GetNamedArgumentValue(columnAttr, "TypeName")
: null,
IsNullable = SyntaxHelper.IsNullableType(prop.Type),
IsKey = AttributeHelper.IsKey(prop),
IsRequired = AttributeHelper.HasAttribute(prop, "Required"),
if (nestedTypeSymbol == null) continue;
analyzedTypes.Add(fullTypeName);
var nestedInfo = new NestedTypeInfo
{
Name = simpleName,
Namespace = nestedTypeSymbol.ContainingNamespace.ToDisplayString(),
FullTypeName = fullTypeName,
Depth = currentDepth
HasPublicSetter = prop.SetMethod?.DeclaredAccessibility == Accessibility.Public,
HasInitOnlySetter = prop.SetMethod?.IsInitOnly == true,
HasAnySetter = prop.SetMethod != null,
IsReadOnlyGetter = isReadOnlyGetter,
BackingFieldName = prop.SetMethod?.DeclaredAccessibility != Accessibility.Public
? $"<{prop.Name}>k__BackingField"
: null
};
// Analyze properties of this nested type
AnalyzeProperties(nestedTypeSymbol, nestedInfo.Properties);
targetNestedTypes[fullTypeName] = nestedInfo;
// MaxLength / MinLength
propInfo.MaxLength = AttributeHelper.GetAttributeIntValue(prop, "MaxLength");
propInfo.MinLength = AttributeHelper.GetAttributeIntValue(prop, "MinLength");
// Recurse
AnalyzeNestedTypesRecursive(nestedInfo.Properties, nestedInfo.NestedTypes, semanticModel, analyzedTypes, currentDepth + 1, maxDepth);
}
}
}
}
var stringLengthAttr = AttributeHelper.GetAttribute(prop, "StringLength");
if (stringLengthAttr != null)
{
if (stringLengthAttr.ConstructorArguments.Length > 0 &&
stringLengthAttr.ConstructorArguments[0].Value is int max)
propInfo.MaxLength = max;
string? minLenStr = AttributeHelper.GetNamedArgumentValue(stringLengthAttr, "MinimumLength");
if (int.TryParse(minLenStr, out int min))
propInfo.MinLength = min;
}
// Range
var rangeAttr = AttributeHelper.GetAttribute(prop, "Range");
if (rangeAttr != null && rangeAttr.ConstructorArguments.Length >= 2)
{
if (rangeAttr.ConstructorArguments[0].Value is double dmin) propInfo.RangeMin = dmin;
else if (rangeAttr.ConstructorArguments[0].Value is int imin) propInfo.RangeMin = (double)imin;
if (rangeAttr.ConstructorArguments[1].Value is double dmax) propInfo.RangeMax = dmax;
else if (rangeAttr.ConstructorArguments[1].Value is int imax) propInfo.RangeMax = (double)imax;
}
if (SyntaxHelper.IsCollectionType(prop.Type, out var itemType))
{
propInfo.IsCollection = true;
propInfo.IsArray = prop.Type is IArrayTypeSymbol;
// Determine concrete collection type name
propInfo.CollectionConcreteTypeName = SyntaxHelper.GetTypeName(prop.Type);
if (itemType != null)
{
propInfo.CollectionItemType = SyntaxHelper.GetTypeName(itemType);
// Check if collection item is nested object
if (SyntaxHelper.IsNestedObjectType(itemType))
{
propInfo.IsCollectionItemNested = true;
propInfo.NestedTypeName = itemType.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)itemType);
}
}
}
// Check for Nested Object
else if (SyntaxHelper.IsNestedObjectType(prop.Type))
{
propInfo.IsNestedObject = true;
propInfo.NestedTypeName = prop.Type.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)prop.Type);
}
properties.Add(propInfo);
}
currentType = currentType.BaseType;
}
}
private static void AnalyzeNestedTypesRecursive(
List<PropertyInfo> properties,
Dictionary<string, NestedTypeInfo> targetNestedTypes,
SemanticModel semanticModel,
HashSet<string> analyzedTypes,
int currentDepth,
int maxDepth)
{
if (currentDepth > maxDepth) return;
// Identify properties that reference nested types (either directly or via collection)
var nestedProps = properties
.Where(p => (p.IsNestedObject || p.IsCollectionItemNested) && !string.IsNullOrEmpty(p.NestedTypeFullName))
.ToList();
foreach (var prop in nestedProps)
{
string fullTypeName = prop.NestedTypeFullName!;
string simpleName = prop.NestedTypeName!;
// Avoid cycles
if (analyzedTypes.Contains(fullTypeName)) continue;
// If already in target list, skip
if (targetNestedTypes.ContainsKey(fullTypeName)) continue;
// Try to find the symbol
INamedTypeSymbol? nestedTypeSymbol = null;
// Try by full name
nestedTypeSymbol = semanticModel.Compilation.GetTypeByMetadataName(fullTypeName);
// If not found, try to resolve via semantic model (might be in the same compilation)
if (nestedTypeSymbol == null)
{
// This is more complex, but usually fullTypeName from ToDisplayString() is traceable.
// For now, let's assume GetTypeByMetadataName works for fully qualified names.
}
if (nestedTypeSymbol == null) continue;
analyzedTypes.Add(fullTypeName);
var nestedInfo = new NestedTypeInfo
{
Name = simpleName,
Namespace = nestedTypeSymbol.ContainingNamespace.ToDisplayString(),
FullTypeName = fullTypeName,
Depth = currentDepth
};
// Analyze properties of this nested type
AnalyzeProperties(nestedTypeSymbol, nestedInfo.Properties);
targetNestedTypes[fullTypeName] = nestedInfo;
// Recurse
AnalyzeNestedTypesRecursive(nestedInfo.Properties, nestedInfo.NestedTypes, semanticModel, analyzedTypes,
currentDepth + 1, maxDepth);
}
}
}

1737
src/CBDD.SourceGenerators/Generators/CodeGenerator.cs
View File

File diff suppressed because it is too large Load Diff

759
src/CBDD.SourceGenerators/Generators/MapperGenerator.cs
View File

@@ -1,403 +1,401 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
namespace ZB.MOM.WW.CBDD.SourceGenerators
{
public class DbContextInfo
{
/// <summary>
/// Gets or sets the simple class name of the DbContext.
/// </summary>
public string ClassName { get; set; } = "";
/// <summary>
/// Gets the fully qualified class name of the DbContext.
/// </summary>
public string FullClassName => string.IsNullOrEmpty(Namespace) ? ClassName : $"{Namespace}.{ClassName}";
/// <summary>
/// Gets or sets the namespace that contains the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the source file path where the DbContext was found.
/// </summary>
public string FilePath { get; set; } = "";
/// <summary>
/// Gets or sets a value indicating whether the DbContext is nested.
/// </summary>
public bool IsNested { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext is partial.
/// </summary>
public bool IsPartial { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext inherits from another DbContext.
/// </summary>
public bool HasBaseDbContext { get; set; } // True if inherits from another DbContext (not DocumentDbContext directly)
/// <summary>
/// Gets or sets the entities discovered for this DbContext.
/// </summary>
public List<EntityInfo> Entities { get; set; } = new List<EntityInfo>();
/// <summary>
/// Gets or sets the collected nested types keyed by full type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; set; } = new Dictionary<string, NestedTypeInfo>();
}
namespace ZB.MOM.WW.CBDD.SourceGenerators;
[Generator]
public class MapperGenerator : IIncrementalGenerator
{
/// <summary>
/// Initializes the mapper source generator pipeline.
/// </summary>
/// <param name="context">The incremental generator initialization context.</param>
public void Initialize(IncrementalGeneratorInitializationContext context)
{
// Find all classes that inherit from DocumentDbContext
var dbContextClasses = context.SyntaxProvider
.CreateSyntaxProvider(
predicate: static (node, _) => IsPotentialDbContext(node),
transform: static (ctx, _) => GetDbContextInfo(ctx))
.Where(static context => context is not null)
.Collect()
.SelectMany(static (contexts, _) => contexts.GroupBy(c => c!.FullClassName).Select(g => g.First())!);
// Generate code for each DbContext
context.RegisterSourceOutput(dbContextClasses, static (spc, dbContext) =>
public class DbContextInfo
{
/// <summary>
/// Gets or sets the simple class name of the DbContext.
/// </summary>
public string ClassName { get; set; } = "";
/// <summary>
/// Gets the fully qualified class name of the DbContext.
/// </summary>
public string FullClassName => string.IsNullOrEmpty(Namespace) ? ClassName : $"{Namespace}.{ClassName}";
/// <summary>
/// Gets or sets the namespace that contains the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the source file path where the DbContext was found.
/// </summary>
public string FilePath { get; set; } = "";
/// <summary>
/// Gets or sets a value indicating whether the DbContext is nested.
/// </summary>
public bool IsNested { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext is partial.
/// </summary>
public bool IsPartial { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext inherits from another DbContext.
/// </summary>
public bool
HasBaseDbContext { get; set; } // True if inherits from another DbContext (not DocumentDbContext directly)
/// <summary>
/// Gets or sets the entities discovered for this DbContext.
/// </summary>
public List<EntityInfo> Entities { get; set; } = new();
/// <summary>
/// Gets or sets the collected nested types keyed by full type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; set; } = new();
}
[Generator]
public class MapperGenerator : IIncrementalGenerator
{
/// <summary>
/// Initializes the mapper source generator pipeline.
/// </summary>
/// <param name="context">The incremental generator initialization context.</param>
public void Initialize(IncrementalGeneratorInitializationContext context)
{
// Find all classes that inherit from DocumentDbContext
var dbContextClasses = context.SyntaxProvider
.CreateSyntaxProvider(
static (node, _) => IsPotentialDbContext(node),
static (ctx, _) => GetDbContextInfo(ctx))
.Where(static context => context is not null)
.Collect()
.SelectMany(static (contexts, _) => contexts.GroupBy(c => c!.FullClassName).Select(g => g.First())!);
// Generate code for each DbContext
context.RegisterSourceOutput(dbContextClasses, static (spc, dbContext) =>
{
if (dbContext == null) return;
var sb = new StringBuilder();
sb.AppendLine($"// Found DbContext: {dbContext.ClassName}");
sb.AppendLine(
$"// BaseType: {(dbContext.HasBaseDbContext ? "inherits from another DbContext" : "inherits from DocumentDbContext directly")}");
foreach (var entity in dbContext.Entities)
// Aggregate nested types recursively
CollectNestedTypes(entity.NestedTypes, dbContext.GlobalNestedTypes);
// Collect namespaces
var namespaces = new HashSet<string>
{
if (dbContext == null) return;
"System",
"System.Collections.Generic",
"ZB.MOM.WW.CBDD.Bson",
"ZB.MOM.WW.CBDD.Core.Collections"
};
// Add Entity namespaces
foreach (var entity in dbContext.Entities)
if (!string.IsNullOrEmpty(entity.Namespace))
namespaces.Add(entity.Namespace);
foreach (var nested in dbContext.GlobalNestedTypes.Values)
if (!string.IsNullOrEmpty(nested.Namespace))
namespaces.Add(nested.Namespace);
// Sanitize file path for name uniqueness
string safeName = dbContext.ClassName;
if (!string.IsNullOrEmpty(dbContext.FilePath))
{
string fileName = Path.GetFileNameWithoutExtension(dbContext.FilePath);
safeName += $"_{fileName}";
}
sb.AppendLine("// <auto-generated/>");
sb.AppendLine("#nullable enable");
foreach (string ns in namespaces.OrderBy(n => n)) sb.AppendLine($"using {ns};");
sb.AppendLine();
// Use safeName (Context + Filename) to avoid collisions
var mapperNamespace = $"{dbContext.Namespace}.{safeName}_Mappers";
sb.AppendLine($"namespace {mapperNamespace}");
sb.AppendLine("{");
var generatedMappers = new HashSet<string>();
// Generate Entity Mappers
foreach (var entity in dbContext.Entities)
if (generatedMappers.Add(entity.FullTypeName))
sb.AppendLine(CodeGenerator.GenerateMapperClass(entity, mapperNamespace));
// Generate Nested Mappers
foreach (var nested in dbContext.GlobalNestedTypes.Values)
if (generatedMappers.Add(nested.FullTypeName))
{
var nestedEntity = new EntityInfo
{
Name = nested.Name,
Namespace = nested.Namespace,
FullTypeName = nested.FullTypeName // Ensure FullTypeName is copied
// Helper to copy properties
};
nestedEntity.Properties.AddRange(nested.Properties);
sb.AppendLine(CodeGenerator.GenerateMapperClass(nestedEntity, mapperNamespace));
}
sb.AppendLine("}");
sb.AppendLine();
// Partial DbContext for InitializeCollections (Only for top-level partial classes)
if (!dbContext.IsNested && dbContext.IsPartial)
{
sb.AppendLine($"namespace {dbContext.Namespace}");
sb.AppendLine("{");
sb.AppendLine($" public partial class {dbContext.ClassName}");
sb.AppendLine(" {");
sb.AppendLine(" protected override void InitializeCollections()");
sb.AppendLine(" {");
// Call base.InitializeCollections() if this context inherits from another DbContext
if (dbContext.HasBaseDbContext) sb.AppendLine(" base.InitializeCollections();");
var sb = new StringBuilder();
sb.AppendLine($"// Found DbContext: {dbContext.ClassName}");
sb.AppendLine($"// BaseType: {(dbContext.HasBaseDbContext ? "inherits from another DbContext" : "inherits from DocumentDbContext directly")}");
foreach (var entity in dbContext.Entities)
{
// Aggregate nested types recursively
CollectNestedTypes(entity.NestedTypes, dbContext.GlobalNestedTypes);
}
if (!string.IsNullOrEmpty(entity.CollectionPropertyName))
{
var mapperName =
$"global::{mapperNamespace}.{CodeGenerator.GetMapperName(entity.FullTypeName)}";
sb.AppendLine(
$" this.{entity.CollectionPropertyName} = CreateCollection(new {mapperName}());");
}
// Collect namespaces
var namespaces = new HashSet<string>
{
"System",
"System.Collections.Generic",
"ZB.MOM.WW.CBDD.Bson",
"ZB.MOM.WW.CBDD.Core.Collections"
};
// Add Entity namespaces
foreach (var entity in dbContext.Entities)
{
if (!string.IsNullOrEmpty(entity.Namespace))
namespaces.Add(entity.Namespace);
}
foreach (var nested in dbContext.GlobalNestedTypes.Values)
{
if (!string.IsNullOrEmpty(nested.Namespace))
namespaces.Add(nested.Namespace);
}
// Sanitize file path for name uniqueness
var safeName = dbContext.ClassName;
if (!string.IsNullOrEmpty(dbContext.FilePath))
{
var fileName = System.IO.Path.GetFileNameWithoutExtension(dbContext.FilePath);
safeName += $"_{fileName}";
}
sb.AppendLine("// <auto-generated/>");
sb.AppendLine("#nullable enable");
foreach (var ns in namespaces.OrderBy(n => n))
{
sb.AppendLine($"using {ns};");
}
sb.AppendLine(" }");
sb.AppendLine();
// Use safeName (Context + Filename) to avoid collisions
var mapperNamespace = $"{dbContext.Namespace}.{safeName}_Mappers";
sb.AppendLine($"namespace {mapperNamespace}");
sb.AppendLine($"{{");
// Generate Set<TId, T>() override
var collectionsWithProperties = dbContext.Entities
.Where(e => !string.IsNullOrEmpty(e.CollectionPropertyName) &&
!string.IsNullOrEmpty(e.CollectionIdTypeFullName))
.ToList();
var generatedMappers = new HashSet<string>();
// Generate Entity Mappers
foreach (var entity in dbContext.Entities)
if (collectionsWithProperties.Any())
{
if (generatedMappers.Add(entity.FullTypeName))
sb.AppendLine(
" public override global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T> Set<TId, T>()");
sb.AppendLine(" {");
foreach (var entity in collectionsWithProperties)
{
sb.AppendLine(CodeGenerator.GenerateMapperClass(entity, mapperNamespace));
var entityTypeStr = $"global::{entity.FullTypeName}";
string? idTypeStr = entity.CollectionIdTypeFullName;
sb.AppendLine(
$" if (typeof(TId) == typeof({idTypeStr}) && typeof(T) == typeof({entityTypeStr}))");
sb.AppendLine(
$" return (global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T>)(object)this.{entity.CollectionPropertyName};");
}
}
// Generate Nested Mappers
foreach (var nested in dbContext.GlobalNestedTypes.Values)
{
if (generatedMappers.Add(nested.FullTypeName))
{
var nestedEntity = new EntityInfo
{
Name = nested.Name,
Namespace = nested.Namespace,
FullTypeName = nested.FullTypeName, // Ensure FullTypeName is copied
// Helper to copy properties
};
nestedEntity.Properties.AddRange(nested.Properties);
sb.AppendLine(CodeGenerator.GenerateMapperClass(nestedEntity, mapperNamespace));
}
}
sb.AppendLine($"}}");
sb.AppendLine();
// Partial DbContext for InitializeCollections (Only for top-level partial classes)
if (!dbContext.IsNested && dbContext.IsPartial)
{
sb.AppendLine($"namespace {dbContext.Namespace}");
sb.AppendLine($"{{");
sb.AppendLine($" public partial class {dbContext.ClassName}");
sb.AppendLine($" {{");
sb.AppendLine($" protected override void InitializeCollections()");
sb.AppendLine($" {{");
// Call base.InitializeCollections() if this context inherits from another DbContext
if (dbContext.HasBaseDbContext)
{
sb.AppendLine($" base.InitializeCollections();");
}
foreach (var entity in dbContext.Entities)
{
if (!string.IsNullOrEmpty(entity.CollectionPropertyName))
{
var mapperName = $"global::{mapperNamespace}.{CodeGenerator.GetMapperName(entity.FullTypeName)}";
sb.AppendLine($" this.{entity.CollectionPropertyName} = CreateCollection(new {mapperName}());");
}
}
sb.AppendLine($" }}");
sb.AppendLine();
sb.AppendLine(" return base.Set<TId, T>();");
else
sb.AppendLine(
" throw new global::System.InvalidOperationException($\"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.\");");
// Generate Set<TId, T>() override
var collectionsWithProperties = dbContext.Entities
.Where(e => !string.IsNullOrEmpty(e.CollectionPropertyName) && !string.IsNullOrEmpty(e.CollectionIdTypeFullName))
.ToList();
sb.AppendLine(" }");
}
if (collectionsWithProperties.Any())
{
sb.AppendLine($" public override global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T> Set<TId, T>()");
sb.AppendLine($" {{");
sb.AppendLine(" }");
sb.AppendLine("}");
}
foreach (var entity in collectionsWithProperties)
{
var entityTypeStr = $"global::{entity.FullTypeName}";
var idTypeStr = entity.CollectionIdTypeFullName;
sb.AppendLine($" if (typeof(TId) == typeof({idTypeStr}) && typeof(T) == typeof({entityTypeStr}))");
sb.AppendLine($" return (global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T>)(object)this.{entity.CollectionPropertyName};");
}
spc.AddSource($"{dbContext.Namespace}.{safeName}.Mappers.g.cs", sb.ToString());
});
}
if (dbContext.HasBaseDbContext)
{
sb.AppendLine($" return base.Set<TId, T>();");
}
else
{
sb.AppendLine($" throw new global::System.InvalidOperationException($\"No collection registered for entity type '{{typeof(T).Name}}' with key type '{{typeof(TId).Name}}'.\");");
}
sb.AppendLine($" }}");
}
sb.AppendLine($" }}");
sb.AppendLine($"}}");
}
spc.AddSource($"{dbContext.Namespace}.{safeName}.Mappers.g.cs", sb.ToString());
});
}
private static void CollectNestedTypes(Dictionary<string, NestedTypeInfo> source, Dictionary<string, NestedTypeInfo> target)
{
foreach (var kvp in source)
private static void CollectNestedTypes(Dictionary<string, NestedTypeInfo> source,
Dictionary<string, NestedTypeInfo> target)
{
foreach (var kvp in source)
if (!target.ContainsKey(kvp.Value.FullTypeName))
{
if (!target.ContainsKey(kvp.Value.FullTypeName))
target[kvp.Value.FullTypeName] = kvp.Value;
CollectNestedTypes(kvp.Value.NestedTypes, target);
}
}
private static void PrintNestedTypes(StringBuilder sb, Dictionary<string, NestedTypeInfo> nestedTypes,
string indent)
{
foreach (var nt in nestedTypes.Values)
{
sb.AppendLine($"//{indent}- {nt.Name} (Depth: {nt.Depth})");
if (nt.Properties.Count > 0)
// Print properties for nested type to be sure
foreach (var p in nt.Properties)
{
target[kvp.Value.FullTypeName] = kvp.Value;
CollectNestedTypes(kvp.Value.NestedTypes, target);
var flags = new List<string>();
if (p.IsCollection) flags.Add($"Collection<{p.CollectionItemType}>");
if (p.IsNestedObject) flags.Add($"Nested<{p.NestedTypeName}>");
string flagStr = flags.Any() ? $" [{string.Join(", ", flags)}]" : "";
sb.AppendLine($"//{indent} - {p.Name}: {p.TypeName}{flagStr}");
}
if (nt.NestedTypes.Any()) PrintNestedTypes(sb, nt.NestedTypes, indent + " ");
}
}
private static bool IsPotentialDbContext(SyntaxNode node)
{
if (node.SyntaxTree.FilePath.EndsWith(".g.cs")) return false;
return node is ClassDeclarationSyntax classDecl &&
classDecl.BaseList != null &&
classDecl.Identifier.Text.EndsWith("Context");
}
private static DbContextInfo? GetDbContextInfo(GeneratorSyntaxContext context)
{
var classDecl = (ClassDeclarationSyntax)context.Node;
var semanticModel = context.SemanticModel;
var classSymbol = ModelExtensions.GetDeclaredSymbol(semanticModel, classDecl) as INamedTypeSymbol;
if (classSymbol == null) return null;
if (!SyntaxHelper.InheritsFrom(classSymbol, "DocumentDbContext"))
return null;
// Check if this context inherits from another DbContext (not DocumentDbContext directly)
var baseType = classSymbol.BaseType;
bool hasBaseDbContext = baseType != null &&
baseType.Name != "DocumentDbContext" &&
SyntaxHelper.InheritsFrom(baseType, "DocumentDbContext");
var info = new DbContextInfo
{
ClassName = classSymbol.Name,
Namespace = classSymbol.ContainingNamespace.ToDisplayString(),
FilePath = classDecl.SyntaxTree.FilePath,
IsNested = classSymbol.ContainingType != null,
IsPartial = classDecl.Modifiers.Any(m => m.IsKind(SyntaxKind.PartialKeyword)),
HasBaseDbContext = hasBaseDbContext
};
// Analyze OnModelCreating to find entities
var onModelCreating = classDecl.Members
.OfType<MethodDeclarationSyntax>()
.FirstOrDefault(m => m.Identifier.Text == "OnModelCreating");
if (onModelCreating != null)
{
var entityCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "Entity");
foreach (var call in entityCalls)
{
string? typeName = SyntaxHelper.GetGenericTypeArgument(call);
if (typeName != null)
{
// Try to find the symbol
INamedTypeSymbol? entityType = null;
// 1. Try by name in current compilation (simple name)
var symbols = semanticModel.Compilation.GetSymbolsWithName(typeName);
entityType = symbols.OfType<INamedTypeSymbol>().FirstOrDefault();
// 2. Try by metadata name (if fully qualified)
if (entityType == null) entityType = semanticModel.Compilation.GetTypeByMetadataName(typeName);
if (entityType != null)
{
// Check for duplicates
string fullTypeName = SyntaxHelper.GetFullName(entityType);
if (!info.Entities.Any(e => e.FullTypeName == fullTypeName))
{
var entityInfo = EntityAnalyzer.Analyze(entityType, semanticModel);
info.Entities.Add(entityInfo);
}
}
}
}
}
private static void PrintNestedTypes(StringBuilder sb, Dictionary<string, NestedTypeInfo> nestedTypes, string indent)
{
foreach (var nt in nestedTypes.Values)
{
sb.AppendLine($"//{indent}- {nt.Name} (Depth: {nt.Depth})");
if (nt.Properties.Count > 0)
{
// Print properties for nested type to be sure
foreach (var p in nt.Properties)
{
var flags = new List<string>();
if (p.IsCollection) flags.Add($"Collection<{p.CollectionItemType}>");
if (p.IsNestedObject) flags.Add($"Nested<{p.NestedTypeName}>");
var flagStr = flags.Any() ? $" [{string.Join(", ", flags)}]" : "";
sb.AppendLine($"//{indent} - {p.Name}: {p.TypeName}{flagStr}");
}
}
if (nt.NestedTypes.Any())
{
PrintNestedTypes(sb, nt.NestedTypes, indent + " ");
}
}
}
private static bool IsPotentialDbContext(SyntaxNode node)
{
if (node.SyntaxTree.FilePath.EndsWith(".g.cs")) return false;
}
return node is ClassDeclarationSyntax classDecl &&
classDecl.BaseList != null &&
classDecl.Identifier.Text.EndsWith("Context");
}
private static DbContextInfo? GetDbContextInfo(GeneratorSyntaxContext context)
// Analyze OnModelCreating for HasConversion
if (onModelCreating != null)
{
var classDecl = (ClassDeclarationSyntax)context.Node;
var semanticModel = context.SemanticModel;
var classSymbol = semanticModel.GetDeclaredSymbol(classDecl) as INamedTypeSymbol;
if (classSymbol == null) return null;
if (!SyntaxHelper.InheritsFrom(classSymbol, "DocumentDbContext"))
return null;
// Check if this context inherits from another DbContext (not DocumentDbContext directly)
var baseType = classSymbol.BaseType;
bool hasBaseDbContext = baseType != null &&
baseType.Name != "DocumentDbContext" &&
SyntaxHelper.InheritsFrom(baseType, "DocumentDbContext");
var info = new DbContextInfo
var conversionCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "HasConversion");
foreach (var call in conversionCalls)
{
ClassName = classSymbol.Name,
Namespace = classSymbol.ContainingNamespace.ToDisplayString(),
FilePath = classDecl.SyntaxTree.FilePath,
IsNested = classSymbol.ContainingType != null,
IsPartial = classDecl.Modifiers.Any(m => m.IsKind(Microsoft.CodeAnalysis.CSharp.SyntaxKind.PartialKeyword)),
HasBaseDbContext = hasBaseDbContext
};
// Analyze OnModelCreating to find entities
var onModelCreating = classDecl.Members
.OfType<MethodDeclarationSyntax>()
.FirstOrDefault(m => m.Identifier.Text == "OnModelCreating");
if (onModelCreating != null)
{
var entityCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "Entity");
foreach (var call in entityCalls)
{
var typeName = SyntaxHelper.GetGenericTypeArgument(call);
if (typeName != null)
{
// Try to find the symbol
INamedTypeSymbol? entityType = null;
// 1. Try by name in current compilation (simple name)
var symbols = semanticModel.Compilation.GetSymbolsWithName(typeName);
entityType = symbols.OfType<INamedTypeSymbol>().FirstOrDefault();
// 2. Try by metadata name (if fully qualified)
if (entityType == null)
{
entityType = semanticModel.Compilation.GetTypeByMetadataName(typeName);
}
string? converterName = SyntaxHelper.GetGenericTypeArgument(call);
if (converterName == null) continue;
if (entityType != null)
// Trace back: .Property(x => x.Id).HasConversion<T>() or .HasKey(x => x.Id).HasConversion<T>()
if (call.Expression is MemberAccessExpressionSyntax
{
Expression: InvocationExpressionSyntax propertyCall
} &&
propertyCall.Expression is MemberAccessExpressionSyntax
{
Name: IdentifierNameSyntax { Identifier: { Text: var propertyMethod } }
} &&
(propertyMethod == "Property" || propertyMethod == "HasKey"))
{
string? propertyName =
SyntaxHelper.GetPropertyName(propertyCall.ArgumentList.Arguments.FirstOrDefault()?.Expression);
if (propertyName == null) continue;
// Trace further back: Entity<T>().Property(...)
if (propertyCall.Expression is MemberAccessExpressionSyntax
{
// Check for duplicates
var fullTypeName = SyntaxHelper.GetFullName(entityType);
if (!info.Entities.Any(e => e.FullTypeName == fullTypeName))
Expression: InvocationExpressionSyntax entityCall
} &&
entityCall.Expression is MemberAccessExpressionSyntax
{
Name: GenericNameSyntax { Identifier: { Text: "Entity" } }
})
{
string? entityTypeName = SyntaxHelper.GetGenericTypeArgument(entityCall);
if (entityTypeName != null)
{
var entity = info.Entities.FirstOrDefault(e =>
e.Name == entityTypeName || e.FullTypeName.EndsWith("." + entityTypeName));
if (entity != null)
{
var entityInfo = EntityAnalyzer.Analyze(entityType, semanticModel);
info.Entities.Add(entityInfo);
}
}
}
}
}
// Analyze OnModelCreating for HasConversion
if (onModelCreating != null)
{
var conversionCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "HasConversion");
foreach (var call in conversionCalls)
{
var converterName = SyntaxHelper.GetGenericTypeArgument(call);
if (converterName == null) continue;
// Trace back: .Property(x => x.Id).HasConversion<T>() or .HasKey(x => x.Id).HasConversion<T>()
if (call.Expression is MemberAccessExpressionSyntax { Expression: InvocationExpressionSyntax propertyCall } &&
propertyCall.Expression is MemberAccessExpressionSyntax { Name: IdentifierNameSyntax { Identifier: { Text: var propertyMethod } } } &&
(propertyMethod == "Property" || propertyMethod == "HasKey"))
{
var propertyName = SyntaxHelper.GetPropertyName(propertyCall.ArgumentList.Arguments.FirstOrDefault()?.Expression);
if (propertyName == null) continue;
// Trace further back: Entity<T>().Property(...)
if (propertyCall.Expression is MemberAccessExpressionSyntax { Expression: InvocationExpressionSyntax entityCall } &&
entityCall.Expression is MemberAccessExpressionSyntax { Name: GenericNameSyntax { Identifier: { Text: "Entity" } } })
{
var entityTypeName = SyntaxHelper.GetGenericTypeArgument(entityCall);
if (entityTypeName != null)
{
var entity = info.Entities.FirstOrDefault(e => e.Name == entityTypeName || e.FullTypeName.EndsWith("." + entityTypeName));
if (entity != null)
var prop = entity.Properties.FirstOrDefault(p => p.Name == propertyName);
if (prop != null)
{
var prop = entity.Properties.FirstOrDefault(p => p.Name == propertyName);
if (prop != null)
// Resolve TProvider from ValueConverter<TModel, TProvider>
var converterType =
semanticModel.Compilation.GetTypeByMetadataName(converterName) ??
semanticModel.Compilation.GetSymbolsWithName(converterName)
.OfType<INamedTypeSymbol>().FirstOrDefault();
prop.ConverterTypeName = converterType != null
? SyntaxHelper.GetFullName(converterType)
: converterName;
if (converterType != null && converterType.BaseType != null &&
converterType.BaseType.Name == "ValueConverter" &&
converterType.BaseType.TypeArguments.Length == 2)
{
// Resolve TProvider from ValueConverter<TModel, TProvider>
var converterType = semanticModel.Compilation.GetTypeByMetadataName(converterName) ??
semanticModel.Compilation.GetSymbolsWithName(converterName).OfType<INamedTypeSymbol>().FirstOrDefault();
prop.ConverterTypeName = converterType != null ? SyntaxHelper.GetFullName(converterType) : converterName;
if (converterType != null && converterType.BaseType != null &&
converterType.BaseType.Name == "ValueConverter" &&
converterType.BaseType.TypeArguments.Length == 2)
prop.ProviderTypeName = converterType.BaseType.TypeArguments[1].Name;
}
else if (converterType != null)
{
// Fallback: search deeper in base types
var converterBaseType = converterType.BaseType;
while (converterBaseType != null)
{
prop.ProviderTypeName = converterType.BaseType.TypeArguments[1].Name;
}
else if (converterType != null)
{
// Fallback: search deeper in base types
var converterBaseType = converterType.BaseType;
while (converterBaseType != null)
if (converterBaseType.Name == "ValueConverter" &&
converterBaseType.TypeArguments.Length == 2)
{
if (converterBaseType.Name == "ValueConverter" && converterBaseType.TypeArguments.Length == 2)
{
prop.ProviderTypeName = converterBaseType.TypeArguments[1].Name;
break;
}
converterBaseType = converterBaseType.BaseType;
prop.ProviderTypeName = converterBaseType.TypeArguments[1].Name;
break;
}
converterBaseType = converterBaseType.BaseType;
}
}
}
@@ -406,31 +404,28 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
}
}
}
// Analyze properties to find DocumentCollection<TId, TEntity>
var properties = classSymbol.GetMembers().OfType<IPropertySymbol>();
foreach (var prop in properties)
{
if (prop.Type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.Name == "DocumentCollection")
// Analyze properties to find DocumentCollection<TId, TEntity>
var properties = classSymbol.GetMembers().OfType<IPropertySymbol>();
foreach (var prop in properties)
if (prop.Type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.Name == "DocumentCollection")
// Expecting 2 type arguments: TId, TEntity
if (namedType.TypeArguments.Length == 2)
{
// Expecting 2 type arguments: TId, TEntity
if (namedType.TypeArguments.Length == 2)
var entityType = namedType.TypeArguments[1];
var entityInfo = info.Entities.FirstOrDefault(e => e.FullTypeName == entityType.ToDisplayString());
// If found, update
if (entityInfo != null)
{
var entityType = namedType.TypeArguments[1];
var entityInfo = info.Entities.FirstOrDefault(e => e.FullTypeName == entityType.ToDisplayString());
// If found, update
if (entityInfo != null)
{
entityInfo.CollectionPropertyName = prop.Name;
entityInfo.CollectionIdTypeFullName = namedType.TypeArguments[0].ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
}
entityInfo.CollectionPropertyName = prop.Name;
entityInfo.CollectionIdTypeFullName = namedType.TypeArguments[0]
.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
}
}
}
return info;
}
return info;
}
}
}

232
src/CBDD.SourceGenerators/Helpers/AttributeHelper.cs
View File

@@ -1,121 +1,115 @@
using System;
using System.Linq;
using Microsoft.CodeAnalysis;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
{
public static class AttributeHelper
using System.Linq;
using Microsoft.CodeAnalysis;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
public static class AttributeHelper
{
/// <summary>
/// Determines whether a property should be ignored during mapping.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true" /> when the property has an ignore attribute; otherwise, <see langword="false" />.</returns>
public static bool ShouldIgnore(IPropertySymbol property)
{
/// <summary>
/// Determines whether a property should be ignored during mapping.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true"/> when the property has an ignore attribute; otherwise, <see langword="false"/>.</returns>
public static bool ShouldIgnore(IPropertySymbol property)
{
return HasAttribute(property, "BsonIgnore") ||
HasAttribute(property, "JsonIgnore") ||
HasAttribute(property, "NotMapped");
}
/// <summary>
/// Determines whether a property is marked as a key.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true"/> when the property has a key attribute; otherwise, <see langword="false"/>.</returns>
public static bool IsKey(IPropertySymbol property)
{
return HasAttribute(property, "Key") ||
HasAttribute(property, "BsonId");
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a string.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
public static string? GetAttributeStringValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
return attr.ConstructorArguments[0].Value?.ToString();
}
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as an integer.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
public static int? GetAttributeIntValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
if (attr.ConstructorArguments[0].Value is int val) return val;
}
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a double.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
public static double? GetAttributeDoubleValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
if (attr.ConstructorArguments[0].Value is double val) return val;
if (attr.ConstructorArguments[0].Value is float fval) return (double)fval;
if (attr.ConstructorArguments[0].Value is int ival) return (double)ival;
}
return null;
}
/// <summary>
/// Gets a named argument value from an attribute.
/// </summary>
/// <param name="attr">The attribute data.</param>
/// <param name="name">The named argument key.</param>
/// <returns>The named argument value if present; otherwise, <see langword="null"/>.</returns>
public static string? GetNamedArgumentValue(AttributeData attr, string name)
{
return attr.NamedArguments.FirstOrDefault(a => a.Key == name).Value.Value?.ToString();
}
/// <summary>
/// Gets the first attribute that matches the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The matching attribute data if found; otherwise, <see langword="null"/>.</returns>
public static AttributeData? GetAttribute(ISymbol symbol, string attributeName)
{
return symbol.GetAttributes().FirstOrDefault(a =>
a.AttributeClass != null &&
(a.AttributeClass.Name == attributeName ||
a.AttributeClass.Name == attributeName + "Attribute"));
}
/// <summary>
/// Determines whether a symbol has the specified attribute.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns><see langword="true"/> when a matching attribute exists; otherwise, <see langword="false"/>.</returns>
public static bool HasAttribute(ISymbol symbol, string attributeName)
{
return GetAttribute(symbol, attributeName) != null;
}
return HasAttribute(property, "BsonIgnore") ||
HasAttribute(property, "JsonIgnore") ||
HasAttribute(property, "NotMapped");
}
}
/// <summary>
/// Determines whether a property is marked as a key.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true" /> when the property has a key attribute; otherwise, <see langword="false" />.</returns>
public static bool IsKey(IPropertySymbol property)
{
return HasAttribute(property, "Key") ||
HasAttribute(property, "BsonId");
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a string.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static string? GetAttributeStringValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0) return attr.ConstructorArguments[0].Value?.ToString();
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as an integer.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static int? GetAttributeIntValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
if (attr.ConstructorArguments[0].Value is int val)
return val;
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a double.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static double? GetAttributeDoubleValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
if (attr.ConstructorArguments[0].Value is double val) return val;
if (attr.ConstructorArguments[0].Value is float fval) return (double)fval;
if (attr.ConstructorArguments[0].Value is int ival) return (double)ival;
}
return null;
}
/// <summary>
/// Gets a named argument value from an attribute.
/// </summary>
/// <param name="attr">The attribute data.</param>
/// <param name="name">The named argument key.</param>
/// <returns>The named argument value if present; otherwise, <see langword="null" />.</returns>
public static string? GetNamedArgumentValue(AttributeData attr, string name)
{
return attr.NamedArguments.FirstOrDefault(a => a.Key == name).Value.Value?.ToString();
}
/// <summary>
/// Gets the first attribute that matches the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The matching attribute data if found; otherwise, <see langword="null" />.</returns>
public static AttributeData? GetAttribute(ISymbol symbol, string attributeName)
{
return symbol.GetAttributes().FirstOrDefault(a =>
a.AttributeClass != null &&
(a.AttributeClass.Name == attributeName ||
a.AttributeClass.Name == attributeName + "Attribute"));
}
/// <summary>
/// Determines whether a symbol has the specified attribute.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns><see langword="true" /> when a matching attribute exists; otherwise, <see langword="false" />.</returns>
public static bool HasAttribute(ISymbol symbol, string attributeName)
{
return GetAttribute(symbol, attributeName) != null;
}
}

440
src/CBDD.SourceGenerators/Helpers/SyntaxHelper.cs
View File

@@ -1,253 +1,229 @@
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
{
public static class SyntaxHelper
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
public static class SyntaxHelper
{
/// <summary>
/// Determines whether a symbol inherits from a base type with the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="baseTypeName">The base type name to match.</param>
/// <returns><see langword="true" /> if the symbol inherits from the base type; otherwise, <see langword="false" />.</returns>
public static bool InheritsFrom(INamedTypeSymbol symbol, string baseTypeName)
{
/// <summary>
/// Determines whether a symbol inherits from a base type with the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="baseTypeName">The base type name to match.</param>
/// <returns><see langword="true"/> if the symbol inherits from the base type; otherwise, <see langword="false"/>.</returns>
public static bool InheritsFrom(INamedTypeSymbol symbol, string baseTypeName)
var current = symbol.BaseType;
while (current != null)
{
var current = symbol.BaseType;
while (current != null)
{
if (current.Name == baseTypeName)
return true;
current = current.BaseType;
}
return false;
if (current.Name == baseTypeName)
return true;
current = current.BaseType;
}
/// <summary>
/// Finds method invocations with a matching method name under the provided syntax node.
/// </summary>
/// <param name="node">The root syntax node to search.</param>
/// <param name="methodName">The method name to match.</param>
/// <returns>A list of matching invocation expressions.</returns>
public static List<InvocationExpressionSyntax> FindMethodInvocations(SyntaxNode node, string methodName)
{
return node.DescendantNodes()
.OfType<InvocationExpressionSyntax>()
.Where(invocation =>
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess)
{
return memberAccess.Name.Identifier.Text == methodName;
}
return false;
})
.ToList();
}
return false;
}
/// <summary>
/// Gets the first generic type argument from an invocation, if present.
/// </summary>
/// <param name="invocation">The invocation to inspect.</param>
/// <returns>The generic type argument text, or <see langword="null"/> when not available.</returns>
public static string? GetGenericTypeArgument(InvocationExpressionSyntax invocation)
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess &&
memberAccess.Name is GenericNameSyntax genericName &&
genericName.TypeArgumentList.Arguments.Count > 0)
{
return genericName.TypeArgumentList.Arguments[0].ToString();
}
return null;
}
/// <summary>
/// Extracts a property name from an expression.
/// </summary>
/// <param name="expression">The expression to analyze.</param>
/// <returns>The property name when resolved; otherwise, <see langword="null"/>.</returns>
public static string? GetPropertyName(ExpressionSyntax? expression)
{
if (expression == null) return null;
if (expression is LambdaExpressionSyntax lambda)
{
return GetPropertyName(lambda.Body as ExpressionSyntax);
}
if (expression is MemberAccessExpressionSyntax memberAccess)
{
return memberAccess.Name.Identifier.Text;
}
if (expression is PrefixUnaryExpressionSyntax prefixUnary && prefixUnary.Operand is MemberAccessExpressionSyntax prefixMember)
{
return prefixMember.Name.Identifier.Text;
}
if (expression is PostfixUnaryExpressionSyntax postfixUnary && postfixUnary.Operand is MemberAccessExpressionSyntax postfixMember)
{
return postfixMember.Name.Identifier.Text;
}
return null;
}
/// <summary>
/// Gets the fully-qualified type name without the global prefix.
/// </summary>
/// <param name="symbol">The symbol to format.</param>
/// <returns>The formatted full type name.</returns>
public static string GetFullName(INamedTypeSymbol symbol)
{
return symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat)
.Replace("global::", "");
}
/// <summary>
/// Gets a display name for a type symbol.
/// </summary>
/// <param name="type">The type symbol to format.</param>
/// <returns>The display name.</returns>
public static string GetTypeName(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
var underlyingType = namedType.TypeArguments[0];
return GetTypeName(underlyingType) + "?";
}
if (type is IArrayTypeSymbol arrayType)
{
return GetTypeName(arrayType.ElementType) + "[]";
}
if (type is INamedTypeSymbol nt && nt.IsTupleType)
{
return type.ToDisplayString();
}
return type.ToDisplayString();
}
/// <summary>
/// Determines whether a type is nullable.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is nullable; otherwise, <see langword="false"/>.</returns>
public static bool IsNullableType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
return true;
}
return type.NullableAnnotation == NullableAnnotation.Annotated;
}
/// <summary>
/// Determines whether a type is a collection and returns its item type when available.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <param name="itemType">When this method returns, contains the collection item type if the type is a collection.</param>
/// <returns><see langword="true"/> if the type is a collection; otherwise, <see langword="false"/>.</returns>
public static bool IsCollectionType(ITypeSymbol type, out ITypeSymbol? itemType)
{
itemType = null;
// Exclude string (it's IEnumerable<char> but not a collection for our purposes)
if (type.SpecialType == SpecialType.System_String)
/// <summary>
/// Finds method invocations with a matching method name under the provided syntax node.
/// </summary>
/// <param name="node">The root syntax node to search.</param>
/// <param name="methodName">The method name to match.</param>
/// <returns>A list of matching invocation expressions.</returns>
public static List<InvocationExpressionSyntax> FindMethodInvocations(SyntaxNode node, string methodName)
{
return node.DescendantNodes()
.OfType<InvocationExpressionSyntax>()
.Where(invocation =>
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess)
return memberAccess.Name.Identifier.Text == methodName;
return false;
})
.ToList();
}
// Handle arrays
if (type is IArrayTypeSymbol arrayType)
{
itemType = arrayType.ElementType;
return true;
}
/// <summary>
/// Gets the first generic type argument from an invocation, if present.
/// </summary>
/// <param name="invocation">The invocation to inspect.</param>
/// <returns>The generic type argument text, or <see langword="null" /> when not available.</returns>
public static string? GetGenericTypeArgument(InvocationExpressionSyntax invocation)
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess &&
memberAccess.Name is GenericNameSyntax genericName &&
genericName.TypeArgumentList.Arguments.Count > 0)
return genericName.TypeArgumentList.Arguments[0].ToString();
return null;
}
// Check if the type itself is IEnumerable<T>
if (type is INamedTypeSymbol namedType && namedType.IsGenericType)
{
var typeDefName = namedType.OriginalDefinition.ToDisplayString();
if (typeDefName == "System.Collections.Generic.IEnumerable<T>" && namedType.TypeArguments.Length == 1)
{
itemType = namedType.TypeArguments[0];
return true;
}
}
/// <summary>
/// Extracts a property name from an expression.
/// </summary>
/// <param name="expression">The expression to analyze.</param>
/// <returns>The property name when resolved; otherwise, <see langword="null" />.</returns>
public static string? GetPropertyName(ExpressionSyntax? expression)
{
if (expression == null) return null;
if (expression is LambdaExpressionSyntax lambda) return GetPropertyName(lambda.Body as ExpressionSyntax);
if (expression is MemberAccessExpressionSyntax memberAccess) return memberAccess.Name.Identifier.Text;
if (expression is PrefixUnaryExpressionSyntax prefixUnary &&
prefixUnary.Operand is MemberAccessExpressionSyntax prefixMember) return prefixMember.Name.Identifier.Text;
if (expression is PostfixUnaryExpressionSyntax postfixUnary &&
postfixUnary.Operand is MemberAccessExpressionSyntax postfixMember)
return postfixMember.Name.Identifier.Text;
return null;
}
// Check if the type implements IEnumerable<T> by walking all interfaces
var enumerableInterface = type.AllInterfaces
.FirstOrDefault(i => i.IsGenericType &&
i.OriginalDefinition.ToDisplayString() == "System.Collections.Generic.IEnumerable<T>");
/// <summary>
/// Gets the fully-qualified type name without the global prefix.
/// </summary>
/// <param name="symbol">The symbol to format.</param>
/// <returns>The formatted full type name.</returns>
public static string GetFullName(INamedTypeSymbol symbol)
{
return symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat)
.Replace("global::", "");
}
if (enumerableInterface != null && enumerableInterface.TypeArguments.Length == 1)
{
itemType = enumerableInterface.TypeArguments[0];
return true;
}
/// <summary>
/// Gets a display name for a type symbol.
/// </summary>
/// <param name="type">The type symbol to format.</param>
/// <returns>The display name.</returns>
public static string GetTypeName(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
var underlyingType = namedType.TypeArguments[0];
return GetTypeName(underlyingType) + "?";
}
if (type is IArrayTypeSymbol arrayType) return GetTypeName(arrayType.ElementType) + "[]";
if (type is INamedTypeSymbol nt && nt.IsTupleType) return type.ToDisplayString();
return type.ToDisplayString();
}
/// <summary>
/// Determines whether a type is nullable.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true" /> if the type is nullable; otherwise, <see langword="false" />.</returns>
public static bool IsNullableType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
return true;
return type.NullableAnnotation == NullableAnnotation.Annotated;
}
/// <summary>
/// Determines whether a type is a collection and returns its item type when available.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <param name="itemType">When this method returns, contains the collection item type if the type is a collection.</param>
/// <returns><see langword="true" /> if the type is a collection; otherwise, <see langword="false" />.</returns>
public static bool IsCollectionType(ITypeSymbol type, out ITypeSymbol? itemType)
{
itemType = null;
// Exclude string (it's IEnumerable<char> but not a collection for our purposes)
if (type.SpecialType == SpecialType.System_String)
return false;
// Handle arrays
if (type is IArrayTypeSymbol arrayType)
{
itemType = arrayType.ElementType;
return true;
}
/// <summary>
/// Determines whether a type should be treated as a primitive value.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is primitive-like; otherwise, <see langword="false"/>.</returns>
public static bool IsPrimitiveType(ITypeSymbol type)
// Check if the type itself is IEnumerable<T>
if (type is INamedTypeSymbol namedType && namedType.IsGenericType)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
type = namedType.TypeArguments[0];
string typeDefName = namedType.OriginalDefinition.ToDisplayString();
if (typeDefName == "System.Collections.Generic.IEnumerable<T>" && namedType.TypeArguments.Length == 1)
{
itemType = namedType.TypeArguments[0];
return true;
}
if (type.SpecialType != SpecialType.None && type.SpecialType != SpecialType.System_Object)
return true;
var typeName = type.Name;
if (typeName == "Guid" || typeName == "DateTime" || typeName == "DateTimeOffset" ||
typeName == "TimeSpan" || typeName == "DateOnly" || typeName == "TimeOnly" ||
typeName == "Decimal" || typeName == "ObjectId")
return true;
if (type.TypeKind == TypeKind.Enum)
return true;
if (type is INamedTypeSymbol nt && nt.IsTupleType)
return true;
return false;
}
/// <summary>
/// Determines whether a type should be treated as a nested object.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is a nested object; otherwise, <see langword="false"/>.</returns>
public static bool IsNestedObjectType(ITypeSymbol type)
{
if (IsPrimitiveType(type)) return false;
if (type.SpecialType == SpecialType.System_String) return false;
if (IsCollectionType(type, out _)) return false;
if (type.SpecialType == SpecialType.System_Object) return false;
if (type is INamedTypeSymbol nt && nt.IsTupleType) return false;
// Check if the type implements IEnumerable<T> by walking all interfaces
var enumerableInterface = type.AllInterfaces
.FirstOrDefault(i => i.IsGenericType &&
i.OriginalDefinition.ToDisplayString() == "System.Collections.Generic.IEnumerable<T>");
return type.TypeKind == TypeKind.Class || type.TypeKind == TypeKind.Struct;
if (enumerableInterface != null && enumerableInterface.TypeArguments.Length == 1)
{
itemType = enumerableInterface.TypeArguments[0];
return true;
}
/// <summary>
/// Determines whether a property has an associated backing field.
/// </summary>
/// <param name="property">The property to inspect.</param>
/// <returns><see langword="true"/> if a backing field is found; otherwise, <see langword="false"/>.</returns>
public static bool HasBackingField(IPropertySymbol property)
{
// Auto-properties have compiler-generated backing fields
// Check if there's a field with the pattern <PropertyName>k__BackingField
return property.ContainingType.GetMembers()
.OfType<IFieldSymbol>()
.Any(f => f.AssociatedSymbol?.Equals(property, SymbolEqualityComparer.Default) == true);
}
}
}
return false;
}
/// <summary>
/// Determines whether a type should be treated as a primitive value.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true" /> if the type is primitive-like; otherwise, <see langword="false" />.</returns>
public static bool IsPrimitiveType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
type = namedType.TypeArguments[0];
if (type.SpecialType != SpecialType.None && type.SpecialType != SpecialType.System_Object)
return true;
string typeName = type.Name;
if (typeName == "Guid" || typeName == "DateTime" || typeName == "DateTimeOffset" ||
typeName == "TimeSpan" || typeName == "DateOnly" || typeName == "TimeOnly" ||
typeName == "Decimal" || typeName == "ObjectId")
return true;
if (type.TypeKind == TypeKind.Enum)
return true;
if (type is INamedTypeSymbol nt && nt.IsTupleType)
return true;
return false;
}
/// <summary>
/// Determines whether a type should be treated as a nested object.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true" /> if the type is a nested object; otherwise, <see langword="false" />.</returns>
public static bool IsNestedObjectType(ITypeSymbol type)
{
if (IsPrimitiveType(type)) return false;
if (type.SpecialType == SpecialType.System_String) return false;
if (IsCollectionType(type, out _)) return false;
if (type.SpecialType == SpecialType.System_Object) return false;
if (type is INamedTypeSymbol nt && nt.IsTupleType) return false;
return type.TypeKind == TypeKind.Class || type.TypeKind == TypeKind.Struct;
}
/// <summary>
/// Determines whether a property has an associated backing field.
/// </summary>
/// <param name="property">The property to inspect.</param>
/// <returns><see langword="true" /> if a backing field is found; otherwise, <see langword="false" />.</returns>
public static bool HasBackingField(IPropertySymbol property)
{
// Auto-properties have compiler-generated backing fields
// Check if there's a field with the pattern <PropertyName>k__BackingField
return property.ContainingType.GetMembers()
.OfType<IFieldSymbol>()
.Any(f => f.AssociatedSymbol?.Equals(property, SymbolEqualityComparer.Default) == true);
}
}

55
src/CBDD.SourceGenerators/Models/DbContextInfo.cs
View File

@@ -1,32 +1,31 @@
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
{
public class DbContextInfo
{
/// <summary>
/// Gets or sets the DbContext class name.
/// </summary>
public string ClassName { get; set; } = "";
using System.Collections.Generic;
/// <summary>
/// Gets or sets the namespace containing the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models;
/// <summary>
/// Gets or sets the source file path for the DbContext.
/// </summary>
public string FilePath { get; set; } = "";
public class DbContextInfo
{
/// <summary>
/// Gets or sets the DbContext class name.
/// </summary>
public string ClassName { get; set; } = "";
/// <summary>
/// Gets the entity types discovered for the DbContext.
/// </summary>
public List<EntityInfo> Entities { get; } = new List<EntityInfo>();
/// <summary>
/// Gets or sets the namespace containing the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets global nested types keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
}
}
/// <summary>
/// Gets or sets the source file path for the DbContext.
/// </summary>
public string FilePath { get; set; } = "";
/// <summary>
/// Gets the entity types discovered for the DbContext.
/// </summary>
public List<EntityInfo> Entities { get; } = new();
/// <summary>
/// Gets global nested types keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; } = new();
}

438
src/CBDD.SourceGenerators/Models/EntityInfo.cs
View File

@@ -1,213 +1,247 @@
using System.Collections.Generic;
using System.Linq;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
using System.Collections.Generic;
using System.Linq;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models;
/// <summary>
/// Contains metadata describing an entity discovered by source generation.
/// </summary>
public class EntityInfo
{
/// <summary>
/// Contains metadata describing an entity discovered by source generation.
/// Gets or sets the entity name.
/// </summary>
public class EntityInfo
{
/// <summary>
/// Gets or sets the entity name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the entity namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified entity type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the collection name for the entity.
/// </summary>
public string CollectionName { get; set; } = "";
/// <summary>
/// Gets or sets the collection property name.
/// </summary>
public string? CollectionPropertyName { get; set; }
/// <summary>
/// Gets or sets the fully qualified collection identifier type name.
/// </summary>
public string? CollectionIdTypeFullName { get; set; }
/// <summary>
/// Gets the key property for the entity if one exists.
/// </summary>
public PropertyInfo? IdProperty => Properties.FirstOrDefault(p => p.IsKey);
/// <summary>
/// Gets or sets a value indicating whether IDs are automatically generated.
/// </summary>
public bool AutoId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity uses private setters.
/// </summary>
public bool HasPrivateSetters { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity has a private or missing constructor.
/// </summary>
public bool HasPrivateOrNoConstructor { get; set; }
/// <summary>
/// Gets the entity properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new List<PropertyInfo>();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
/// <summary>
/// Gets property names that should be ignored by mapping.
/// </summary>
public HashSet<string> IgnoredProperties { get; } = new HashSet<string>();
}
public string Name { get; set; } = "";
/// <summary>
/// Contains metadata describing a mapped property.
/// Gets or sets the entity namespace.
/// </summary>
public class PropertyInfo
{
/// <summary>
/// Gets or sets the property name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the property type name.
/// </summary>
public string TypeName { get; set; } = "";
/// <summary>
/// Gets or sets the BSON field name.
/// </summary>
public string BsonFieldName { get; set; } = "";
/// <summary>
/// Gets or sets the database column type name.
/// </summary>
public string? ColumnTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is nullable.
/// </summary>
public bool IsNullable { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has a public setter.
/// </summary>
public bool HasPublicSetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property uses an init-only setter.
/// </summary>
public bool HasInitOnlySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has any setter.
/// </summary>
public bool HasAnySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the getter is read-only.
/// </summary>
public bool IsReadOnlyGetter { get; set; }
/// <summary>
/// Gets or sets the backing field name if available.
/// </summary>
public string? BackingFieldName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is the key.
/// </summary>
public bool IsKey { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is required.
/// </summary>
public bool IsRequired { get; set; }
/// <summary>
/// Gets or sets the maximum allowed length.
/// </summary>
public int? MaxLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed length.
/// </summary>
public int? MinLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed range value.
/// </summary>
public double? RangeMin { get; set; }
/// <summary>
/// Gets or sets the maximum allowed range value.
/// </summary>
public double? RangeMax { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a collection.
/// </summary>
public bool IsCollection { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is an array.
/// </summary>
public bool IsArray { get; set; }
/// <summary>
/// Gets or sets the collection item type name.
/// </summary>
public string? CollectionItemType { get; set; }
/// <summary>
/// Gets or sets the concrete collection type name.
/// </summary>
public string? CollectionConcreteTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a nested object.
/// </summary>
public bool IsNestedObject { get; set; }
/// <summary>
/// Gets or sets a value indicating whether collection items are nested objects.
/// </summary>
public bool IsCollectionItemNested { get; set; }
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string? NestedTypeName { get; set; }
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string? NestedTypeFullName { get; set; }
/// <summary>
/// Gets or sets the converter type name.
/// </summary>
public string? ConverterTypeName { get; set; }
/// <summary>
/// Gets or sets the provider type name used by the converter.
/// </summary>
public string? ProviderTypeName { get; set; }
}
public string Namespace { get; set; } = "";
/// <summary>
/// Contains metadata describing a nested type.
/// Gets or sets the fully qualified entity type name.
/// </summary>
public class NestedTypeInfo
{
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the nested type namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the depth of the nested type.
/// </summary>
public int Depth { get; set; }
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets the nested type properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new List<PropertyInfo>();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
}
/// <summary>
/// Gets or sets the collection name for the entity.
/// </summary>
public string CollectionName { get; set; } = "";
/// <summary>
/// Gets or sets the collection property name.
/// </summary>
public string? CollectionPropertyName { get; set; }
/// <summary>
/// Gets or sets the fully qualified collection identifier type name.
/// </summary>
public string? CollectionIdTypeFullName { get; set; }
/// <summary>
/// Gets the key property for the entity if one exists.
/// </summary>
public PropertyInfo? IdProperty => Properties.FirstOrDefault(p => p.IsKey);
/// <summary>
/// Gets or sets a value indicating whether IDs are automatically generated.
/// </summary>
public bool AutoId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity uses private setters.
/// </summary>
public bool HasPrivateSetters { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity has a private or missing constructor.
/// </summary>
public bool HasPrivateOrNoConstructor { get; set; }
/// <summary>
/// Gets the entity properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new();
/// <summary>
/// Gets property names that should be ignored by mapping.
/// </summary>
public HashSet<string> IgnoredProperties { get; } = new();
}
/// <summary>
/// Contains metadata describing a mapped property.
/// </summary>
public class PropertyInfo
{
/// <summary>
/// Gets or sets the property name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the property type name.
/// </summary>
public string TypeName { get; set; } = "";
/// <summary>
/// Gets or sets the BSON field name.
/// </summary>
public string BsonFieldName { get; set; } = "";
/// <summary>
/// Gets or sets the database column type name.
/// </summary>
public string? ColumnTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is nullable.
/// </summary>
public bool IsNullable { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has a public setter.
/// </summary>
public bool HasPublicSetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property uses an init-only setter.
/// </summary>
public bool HasInitOnlySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has any setter.
/// </summary>
public bool HasAnySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the getter is read-only.
/// </summary>
public bool IsReadOnlyGetter { get; set; }
/// <summary>
/// Gets or sets the backing field name if available.
/// </summary>
public string? BackingFieldName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is the key.
/// </summary>
public bool IsKey { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is required.
/// </summary>
public bool IsRequired { get; set; }
/// <summary>
/// Gets or sets the maximum allowed length.
/// </summary>
public int? MaxLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed length.
/// </summary>
public int? MinLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed range value.
/// </summary>
public double? RangeMin { get; set; }
/// <summary>
/// Gets or sets the maximum allowed range value.
/// </summary>
public double? RangeMax { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a collection.
/// </summary>
public bool IsCollection { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is an array.
/// </summary>
public bool IsArray { get; set; }
/// <summary>
/// Gets or sets the collection item type name.
/// </summary>
public string? CollectionItemType { get; set; }
/// <summary>
/// Gets or sets the concrete collection type name.
/// </summary>
public string? CollectionConcreteTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a nested object.
/// </summary>
public bool IsNestedObject { get; set; }
/// <summary>
/// Gets or sets a value indicating whether collection items are nested objects.
/// </summary>
public bool IsCollectionItemNested { get; set; }
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string? NestedTypeName { get; set; }
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string? NestedTypeFullName { get; set; }
/// <summary>
/// Gets or sets the converter type name.
/// </summary>
public string? ConverterTypeName { get; set; }
/// <summary>
/// Gets or sets the provider type name used by the converter.
/// </summary>
public string? ProviderTypeName { get; set; }
}
/// <summary>
/// Contains metadata describing a nested type.
/// </summary>
public class NestedTypeInfo
{
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the nested type namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the depth of the nested type.
/// </summary>
public int Depth { get; set; }
/// <summary>
/// Gets the nested type properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new();
}

62
src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj
View File

@@ -1,38 +1,38 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.SourceGenerators</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.SourceGenerators</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
<IsRoslynComponent>true</IsRoslynComponent>
<PackageId>ZB.MOM.WW.CBDD.SourceGenerators</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>Source Generators for CBDD High-Performance BSON Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation;source-generator</PackageTags>
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
<IncludeBuildOutput>false</IncludeBuildOutput>
<DevelopmentDependency>true</DevelopmentDependency>
<NoPackageAnalysis>true</NoPackageAnalysis>
</PropertyGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.SourceGenerators</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.SourceGenerators</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
<IsRoslynComponent>true</IsRoslynComponent>
<PackageId>ZB.MOM.WW.CBDD.SourceGenerators</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>Source Generators for CBDD High-Performance BSON Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation;source-generator</PackageTags>
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
<IncludeBuildOutput>false</IncludeBuildOutput>
<DevelopmentDependency>true</DevelopmentDependency>
<NoPackageAnalysis>true</NoPackageAnalysis>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.8.0" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" PrivateAssets="all" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.8.0" PrivateAssets="all"/>
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" PrivateAssets="all"/>
</ItemGroup>
<ItemGroup>
<None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
</ItemGroup>
<ItemGroup>
<None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false"/>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

56
src/CBDD/ZB.MOM.WW.CBDD.csproj
View File

@@ -1,34 +1,34 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PackageId>ZB.MOM.WW.CBDD</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<ItemGroup>
<ProjectReference Include="..\CBDD.Core\ZB.MOM.WW.CBDD.Core.csproj" />
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj" />
<ProjectReference Include="..\CBDD.SourceGenerators\ZB.MOM.WW.CBDD.SourceGenerators.csproj" PrivateAssets="none" />
</ItemGroup>
<PackageId>ZB.MOM.WW.CBDD</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Core\ZB.MOM.WW.CBDD.Core.csproj"/>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj"/>
<ProjectReference Include="..\CBDD.SourceGenerators\ZB.MOM.WW.CBDD.SourceGenerators.csproj" PrivateAssets="none"/>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>

36
tests/CBDD.Tests.Benchmark/Compaction/CompactionBenchmarks.cs
View File

@@ -1,6 +1,6 @@
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
@@ -15,21 +15,22 @@ namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
[JsonExporterAttribute.Full]
public class CompactionBenchmarks
{
private readonly List<ObjectId> _insertedIds = [];
private DocumentCollection<Person> _collection = null!;
private string _dbPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private string _walPath = string.Empty;
/// <summary>
/// Gets or sets the number of documents used per benchmark iteration.
/// Gets or sets the number of documents used per benchmark iteration.
/// </summary>
[Params(2_000)]
public int DocumentCount { get; set; }
private string _dbPath = string.Empty;
private string _walPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private DocumentCollection<Person> _collection = null!;
private List<ObjectId> _insertedIds = [];
/// <summary>
/// Prepares benchmark state and seed data for each iteration.
/// Prepares benchmark state and seed data for each iteration.
/// </summary>
[IterationSetup]
public void Setup()
@@ -53,17 +54,14 @@ public class CompactionBenchmarks
_transactionHolder.CommitAndReset();
_storage.Checkpoint();
for (var i = _insertedIds.Count - 1; i >= _insertedIds.Count / 3; i--)
{
_collection.Delete(_insertedIds[i]);
}
for (int i = _insertedIds.Count - 1; i >= _insertedIds.Count / 3; i--) _collection.Delete(_insertedIds[i]);
_transactionHolder.CommitAndReset();
_storage.Checkpoint();
}
/// <summary>
/// Cleans up benchmark resources and temporary files after each iteration.
/// Cleans up benchmark resources and temporary files after each iteration.
/// </summary>
[IterationCleanup]
public void Cleanup()
@@ -76,7 +74,7 @@ public class CompactionBenchmarks
}
/// <summary>
/// Benchmarks reclaimed file bytes reported by offline compaction.
/// Benchmarks reclaimed file bytes reported by offline compaction.
/// </summary>
/// <returns>The reclaimed file byte count.</returns>
[Benchmark(Baseline = true)]
@@ -95,7 +93,7 @@ public class CompactionBenchmarks
}
/// <summary>
/// Benchmarks tail bytes truncated by offline compaction.
/// Benchmarks tail bytes truncated by offline compaction.
/// </summary>
/// <returns>The truncated tail byte count.</returns>
[Benchmark]
@@ -135,7 +133,7 @@ public class CompactionBenchmarks
private static string BuildPayload(int seed)
{
var builder = new System.Text.StringBuilder(2500);
var builder = new StringBuilder(2500);
for (var i = 0; i < 80; i++)
{
builder.Append("compact-");
@@ -147,4 +145,4 @@ public class CompactionBenchmarks
return builder.ToString();
}
}
}

53
tests/CBDD.Tests.Benchmark/Compression/CompressionBenchmarks.cs
View File

@@ -1,7 +1,7 @@
using System.IO.Compression;
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using System.IO.Compression;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Compression;
@@ -19,36 +19,36 @@ public class CompressionBenchmarks
{
private const int SeedCount = 300;
private const int WorkloadCount = 100;
private DocumentCollection<Person> _collection = null!;
private string _dbPath = string.Empty;
private Person[] _insertBatch = Array.Empty<Person>();
private ObjectId[] _seedIds = Array.Empty<ObjectId>();
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private string _walPath = string.Empty;
/// <summary>
/// Gets or sets whether compression is enabled for the benchmark run.
/// Gets or sets whether compression is enabled for the benchmark run.
/// </summary>
[Params(false, true)]
public bool EnableCompression { get; set; }
/// <summary>
/// Gets or sets the compression codec for the benchmark run.
/// Gets or sets the compression codec for the benchmark run.
/// </summary>
[Params(CompressionCodec.Brotli, CompressionCodec.Deflate)]
public CompressionCodec Codec { get; set; }
/// <summary>
/// Gets or sets the compression level for the benchmark run.
/// Gets or sets the compression level for the benchmark run.
/// </summary>
[Params(CompressionLevel.Fastest, CompressionLevel.Optimal)]
public CompressionLevel Level { get; set; }
private string _dbPath = string.Empty;
private string _walPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private DocumentCollection<Person> _collection = null!;
private Person[] _insertBatch = Array.Empty<Person>();
private ObjectId[] _seedIds = Array.Empty<ObjectId>();
/// <summary>
/// Prepares benchmark storage and seed data for each iteration.
/// Prepares benchmark storage and seed data for each iteration.
/// </summary>
[IterationSetup]
public void Setup()
@@ -73,19 +73,19 @@ public class CompressionBenchmarks
_seedIds = new ObjectId[SeedCount];
for (var i = 0; i < SeedCount; i++)
{
var doc = CreatePerson(i, includeLargeBio: true);
var doc = CreatePerson(i, true);
_seedIds[i] = _collection.Insert(doc);
}
_transactionHolder.CommitAndReset();
_insertBatch = Enumerable.Range(SeedCount, WorkloadCount)
.Select(i => CreatePerson(i, includeLargeBio: true))
.Select(i => CreatePerson(i, true))
.ToArray();
}
/// <summary>
/// Cleans up benchmark resources for each iteration.
/// Cleans up benchmark resources for each iteration.
/// </summary>
[IterationCleanup]
public void Cleanup()
@@ -98,7 +98,7 @@ public class CompressionBenchmarks
}
/// <summary>
/// Benchmarks insert workload performance.
/// Benchmarks insert workload performance.
/// </summary>
[Benchmark(Baseline = true)]
[BenchmarkCategory("Compression_InsertUpdateRead")]
@@ -109,7 +109,7 @@ public class CompressionBenchmarks
}
/// <summary>
/// Benchmarks update workload performance.
/// Benchmarks update workload performance.
/// </summary>
[Benchmark]
[BenchmarkCategory("Compression_InsertUpdateRead")]
@@ -131,7 +131,7 @@ public class CompressionBenchmarks
}
/// <summary>
/// Benchmarks read workload performance.
/// Benchmarks read workload performance.
/// </summary>
[Benchmark]
[BenchmarkCategory("Compression_InsertUpdateRead")]
@@ -141,10 +141,7 @@ public class CompressionBenchmarks
for (var i = 0; i < WorkloadCount; i++)
{
var person = _collection.FindById(_seedIds[i]);
if (person != null)
{
checksum += person.Age;
}
if (person != null) checksum += person.Age;
}
_transactionHolder.CommitAndReset();
@@ -158,7 +155,7 @@ public class CompressionBenchmarks
Id = ObjectId.NewObjectId(),
FirstName = $"First_{i}",
LastName = $"Last_{i}",
Age = 20 + (i % 50),
Age = 20 + i % 50,
Bio = includeLargeBio ? BuildBio(i) : $"bio-{i}",
CreatedAt = DateTime.UnixEpoch.AddMinutes(i),
Balance = 100 + i,
@@ -183,7 +180,7 @@ public class CompressionBenchmarks
private static string BuildBio(int seed)
{
var builder = new System.Text.StringBuilder(4500);
var builder = new StringBuilder(4500);
for (var i = 0; i < 150; i++)
{
builder.Append("bio-");
@@ -195,4 +192,4 @@ public class CompressionBenchmarks
return builder.ToString();
}
}
}

48
tests/CBDD.Tests.Benchmark/Fixtures/Person.cs
View File

@@ -1,21 +1,21 @@
using ZB.MOM.WW.CBDD.Bson;
using System;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
public class Address
{
/// <summary>
/// Gets or sets the Street.
/// Gets or sets the Street.
/// </summary>
public string Street { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the City.
/// Gets or sets the City.
/// </summary>
public string City { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the ZipCode.
/// Gets or sets the ZipCode.
/// </summary>
public string ZipCode { get; set; } = string.Empty;
}
@@ -23,19 +23,22 @@ public class Address
public class WorkHistory
{
/// <summary>
/// Gets or sets the CompanyName.
/// Gets or sets the CompanyName.
/// </summary>
public string CompanyName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the Title.
/// Gets or sets the Title.
/// </summary>
public string Title { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the DurationYears.
/// Gets or sets the DurationYears.
/// </summary>
public int DurationYears { get; set; }
/// <summary>
/// Gets or sets the Tags.
/// Gets or sets the Tags.
/// </summary>
public List<string> Tags { get; set; } = new();
}
@@ -43,41 +46,48 @@ public class WorkHistory
public class Person
{
/// <summary>
/// Gets or sets the Id.
/// Gets or sets the Id.
/// </summary>
public ObjectId Id { get; set; }
/// <summary>
/// Gets or sets the FirstName.
/// Gets or sets the FirstName.
/// </summary>
public string FirstName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the LastName.
/// Gets or sets the LastName.
/// </summary>
public string LastName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the Age.
/// Gets or sets the Age.
/// </summary>
public int Age { get; set; }
/// <summary>
/// Gets or sets the Bio.
/// Gets or sets the Bio.
/// </summary>
public string? Bio { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the CreatedAt.
/// Gets or sets the CreatedAt.
/// </summary>
public DateTime CreatedAt { get; set; }
// Complex fields
/// <summary>
/// Gets or sets the Balance.
/// Gets or sets the Balance.
/// </summary>
public decimal Balance { get; set; }
/// <summary>
/// Gets or sets the HomeAddress.
/// Gets or sets the HomeAddress.
/// </summary>
public Address HomeAddress { get; set; } = new();
/// <summary>
/// Gets or sets the EmploymentHistory.
/// Gets or sets the EmploymentHistory.
/// </summary>
public List<WorkHistory> EmploymentHistory { get; set; } = new();
}
}

177
tests/CBDD.Tests.Benchmark/Fixtures/PersonMapper.cs
View File

@@ -1,26 +1,30 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using System.Buffers;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
public class PersonMapper : ObjectIdMapperBase<Person>
{
/// <inheritdoc />
public override string CollectionName => "people";
/// <inheritdoc />
public override ObjectId GetId(Person entity) => entity.Id;
/// <inheritdoc />
public override void SetId(Person entity, ObjectId id) => entity.Id = id;
/// <inheritdoc />
public override int Serialize(Person entity, BsonSpanWriter writer)
{
var sizePos = writer.BeginDocument();
public class PersonMapper : ObjectIdMapperBase<Person>
{
/// <inheritdoc />
public override string CollectionName => "people";
/// <inheritdoc />
public override ObjectId GetId(Person entity)
{
return entity.Id;
}
/// <inheritdoc />
public override void SetId(Person entity, ObjectId id)
{
entity.Id = id;
}
/// <inheritdoc />
public override int Serialize(Person entity, BsonSpanWriter writer)
{
int sizePos = writer.BeginDocument();
writer.WriteObjectId("_id", entity.Id);
writer.WriteString("firstname", entity.FirstName);
writer.WriteString("lastname", entity.LastName);
@@ -30,111 +34,119 @@ public class PersonMapper : ObjectIdMapperBase<Person>
else
writer.WriteNull("bio");
writer.WriteInt64("createdat", entity.CreatedAt.Ticks);
// Complex fields
writer.WriteDouble("balance", (double)entity.Balance);
// Nested Object: Address
var addrPos = writer.BeginDocument("homeaddress");
writer.WriteInt64("createdat", entity.CreatedAt.Ticks);
// Complex fields
writer.WriteDouble("balance", (double)entity.Balance);
// Nested Object: Address
int addrPos = writer.BeginDocument("homeaddress");
writer.WriteString("street", entity.HomeAddress.Street);
writer.WriteString("city", entity.HomeAddress.City);
writer.WriteString("zipcode", entity.HomeAddress.ZipCode);
writer.EndDocument(addrPos);
// Collection: EmploymentHistory
var histPos = writer.BeginArray("employmenthistory");
for (int i = 0; i < entity.EmploymentHistory.Count; i++)
writer.EndDocument(addrPos);
// Collection: EmploymentHistory
int histPos = writer.BeginArray("employmenthistory");
for (var i = 0; i < entity.EmploymentHistory.Count; i++)
{
var item = entity.EmploymentHistory[i];
// Array elements are keys "0", "1", "2"...
var itemPos = writer.BeginDocument(i.ToString());
int itemPos = writer.BeginDocument(i.ToString());
writer.WriteString("companyname", item.CompanyName);
writer.WriteString("title", item.Title);
writer.WriteInt32("durationyears", item.DurationYears);
// Nested Collection: Tags
var tagsPos = writer.BeginArray("tags");
for (int j = 0; j < item.Tags.Count; j++)
{
writer.WriteString(j.ToString(), item.Tags[j]);
}
writer.EndArray(tagsPos);
writer.WriteInt32("durationyears", item.DurationYears);
// Nested Collection: Tags
int tagsPos = writer.BeginArray("tags");
for (var j = 0; j < item.Tags.Count; j++) writer.WriteString(j.ToString(), item.Tags[j]);
writer.EndArray(tagsPos);
writer.EndDocument(itemPos);
}
writer.EndArray(histPos);
writer.EndDocument(sizePos);
writer.EndArray(histPos);
writer.EndDocument(sizePos);
return writer.Position;
}
/// <inheritdoc />
public override Person Deserialize(BsonSpanReader reader)
{
var person = new Person();
reader.ReadDocumentSize();
/// <inheritdoc />
public override Person Deserialize(BsonSpanReader reader)
{
var person = new Person();
reader.ReadDocumentSize();
while (reader.Remaining > 0)
{
var type = reader.ReadBsonType();
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
break;
string name = reader.ReadElementHeader();
switch (name)
{
case "_id": person.Id = reader.ReadObjectId(); break;
case "firstname": person.FirstName = reader.ReadString(); break;
case "lastname": person.LastName = reader.ReadString(); break;
case "age": person.Age = reader.ReadInt32(); break;
case "bio":
case "bio":
if (type == BsonType.Null) person.Bio = null;
else person.Bio = reader.ReadString();
else person.Bio = reader.ReadString();
break;
case "createdat": person.CreatedAt = new DateTime(reader.ReadInt64()); break;
case "balance": person.Balance = (decimal)reader.ReadDouble(); break;
case "balance": person.Balance = (decimal)reader.ReadDouble(); break;
case "homeaddress":
reader.ReadDocumentSize(); // Enter document
while (reader.Remaining > 0)
{
var addrType = reader.ReadBsonType();
if (addrType == BsonType.EndOfDocument) break;
var addrName = reader.ReadElementHeader();
// We assume strict schema for benchmark speed, but should handle skipping
string addrName = reader.ReadElementHeader();
// We assume strict schema for benchmark speed, but should handle skipping
if (addrName == "street") person.HomeAddress.Street = reader.ReadString();
else if (addrName == "city") person.HomeAddress.City = reader.ReadString();
else if (addrName == "zipcode") person.HomeAddress.ZipCode = reader.ReadString();
else reader.SkipValue(addrType);
}
break;
break;
case "employmenthistory":
reader.ReadDocumentSize(); // Enter Array
while (reader.Remaining > 0)
{
var arrType = reader.ReadBsonType();
if (arrType == BsonType.EndOfDocument) break;
reader.ReadElementHeader(); // Array index "0", "1"... ignore
// Read WorkHistory item
reader.ReadElementHeader(); // Array index "0", "1"... ignore
// Read WorkHistory item
var workItem = new WorkHistory();
reader.ReadDocumentSize(); // Enter Item Document
while (reader.Remaining > 0)
{
var itemType = reader.ReadBsonType();
if (itemType == BsonType.EndOfDocument) break;
var itemName = reader.ReadElementHeader();
if (itemName == "companyname") workItem.CompanyName = reader.ReadString();
else if (itemName == "title") workItem.Title = reader.ReadString();
else if (itemName == "durationyears") workItem.DurationYears = reader.ReadInt32();
string itemName = reader.ReadElementHeader();
if (itemName == "companyname")
{
workItem.CompanyName = reader.ReadString();
}
else if (itemName == "title")
{
workItem.Title = reader.ReadString();
}
else if (itemName == "durationyears")
{
workItem.DurationYears = reader.ReadInt32();
}
else if (itemName == "tags")
{
reader.ReadDocumentSize(); // Enter Tags Array
@@ -149,18 +161,23 @@ public class PersonMapper : ObjectIdMapperBase<Person>
reader.SkipValue(tagType);
}
}
else reader.SkipValue(itemType);
else
{
reader.SkipValue(itemType);
}
}
person.EmploymentHistory.Add(workItem);
}
break;
break;
default:
reader.SkipValue(type);
break;
}
}
}
return person;
}
}
}

Some files were not shown because too many files have changed in this diff Show More