dohertj2/CBDD
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reformat / cleanup
NuGet Publish / build-and-pack (push) Successful in 46s
Details
NuGet Publish / publish-to-gitea (push) Successful in 56s
Details

Browse Source
This commit is contained in:
Joseph Doherty
2026-02-21 08:10:36 -05:00
parent 4c6aaa5a3f
commit a70d8befae
176 changed files with 50555 additions and 49587 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CBDD.slnx
+10 -10
View File
@@ -1,12 +1,12 @@
<Solution>
<Folder Name="/src/">
<Project Path="src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj" />
<Project Path="src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj" />
<Project Path="src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj" />
<Project Path="src/CBDD/ZB.MOM.WW.CBDD.csproj" />
</Folder>
<Folder Name="/tests/">
<Project Path="tests/CBDD.Tests.Benchmark/ZB.MOM.WW.CBDD.Tests.Benchmark.csproj" />
<Project Path="tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj" />
</Folder>
<Folder Name="/src/">
<Project Path="src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj"/>
<Project Path="src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj"/>
<Project Path="src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj"/>
<Project Path="src/CBDD/ZB.MOM.WW.CBDD.csproj"/>
</Folder>
<Folder Name="/tests/">
<Project Path="tests/CBDD.Tests.Benchmark/ZB.MOM.WW.CBDD.Tests.Benchmark.csproj"/>
<Project Path="tests/CBDD.Tests/ZB.MOM.WW.CBDD.Tests.csproj"/>
</Folder>
</Solution>
README.md
+27 -7
View File
@@ -1,16 +1,21 @@
# CBDD
CBDD is an embedded, document-oriented database engine for .NET 10. It targets internal platform teams that need predictable ACID behavior, low-latency local persistence, and typed access patterns without running an external database server.
CBDD is an embedded, document-oriented database engine for .NET 10. It targets internal platform teams that need
predictable ACID behavior, low-latency local persistence, and typed access patterns without running an external database
server.
## Purpose And Business Context
CBDD provides a local data layer for services and tools that need transactional durability, deterministic startup, and high-throughput reads/writes. The primary business outcome is reducing operational overhead for workloads that do not require a networked database cluster.
CBDD provides a local data layer for services and tools that need transactional durability, deterministic startup, and
high-throughput reads/writes. The primary business outcome is reducing operational overhead for workloads that do not
require a networked database cluster.
## Ownership And Support
- Owning team: CBDD maintainers (repository owner: `@dohertj2`)
- Primary support path: open a Gitea issue in this repository with labels `incident` or `bug`
- Escalation path: follow [`docs/runbook.md`](docs/runbook.md) and page the release maintainer listed in the active release PR
- Escalation path: follow [`docs/runbook.md`](docs/runbook.md) and page the release maintainer listed in the active
release PR
## Architecture Overview
@@ -22,6 +27,7 @@ CBDD has four primary layers:
4. Source-generated mapping (`src/CBDD.SourceGenerators`)
Detailed architecture material:
- [`docs/architecture.md`](docs/architecture.md)
- [`RFC.md`](RFC.md)
- [`C-BSON.md`](C-BSON.md)
@@ -36,34 +42,44 @@ Detailed architecture material:
## Setup And Local Run
1. Clone the repository.
```bash
git clone https://gitea.dohertylan.com/dohertj2/CBDD.git
cd CBDD
```
Expected outcome: local repository checkout with `CBDD.slnx` present.
2. Restore dependencies.
```bash
dotnet restore
```
Expected outcome: restore completes without package errors.
3. Build the solution.
```bash
dotnet build CBDD.slnx -c Release
```
Expected outcome: solution builds without compiler errors.
4. Run tests.
```bash
dotnet test CBDD.slnx -c Release
```
Expected outcome: all tests pass.
5. Run the full repository fitness check.
```bash
bash scripts/fitness-check.sh
```
Expected outcome: format, build, tests, coverage threshold, and package checks complete.
## Configuration And Secrets
@@ -135,9 +151,12 @@ if (!result.Executed)
Common issues and remediation:
- Build/test environment failures: [`docs/troubleshooting.md#build-and-test-failures`](docs/troubleshooting.md#build-and-test-failures)
- Data-file recovery procedures: [`docs/troubleshooting.md#data-file-and-recovery-issues`](docs/troubleshooting.md#data-file-and-recovery-issues)
- Query/index behavior verification: [`docs/troubleshooting.md#query-and-index-issues`](docs/troubleshooting.md#query-and-index-issues)
- Build/test environment failures: [
`docs/troubleshooting.md#build-and-test-failures`](docs/troubleshooting.md#build-and-test-failures)
- Data-file recovery procedures: [
`docs/troubleshooting.md#data-file-and-recovery-issues`](docs/troubleshooting.md#data-file-and-recovery-issues)
- Query/index behavior verification: [
`docs/troubleshooting.md#query-and-index-issues`](docs/troubleshooting.md#query-and-index-issues)
## Change Governance
@@ -150,4 +169,5 @@ Common issues and remediation:
- Documentation home: [`docs/README.md`](docs/README.md)
- Major feature inventory: [`docs/features/README.md`](docs/features/README.md)
- Architecture decisions: [`docs/adr/0001-storage-engine-and-source-generation.md`](docs/adr/0001-storage-engine-and-source-generation.md)
- Architecture decisions: [
`docs/adr/0001-storage-engine-and-source-generation.md`](docs/adr/0001-storage-engine-and-source-generation.md)
src/CBDD.Bson/Document/BsonDocument.cs
+41 -36
View File
@@ -1,60 +1,64 @@
using System;
using System.Collections.Concurrent;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// Represents an in-memory BSON document with lazy parsing.
/// Uses Memory&lt;byte&gt; to store raw BSON data for zero-copy operations.
/// Represents an in-memory BSON document with lazy parsing.
/// Uses Memory&lt;byte&gt; to store raw BSON data for zero-copy operations.
/// </summary>
public sealed class BsonDocument
{
private readonly ConcurrentDictionary<ushort, string>? _keys;
private readonly Memory<byte> _rawData;
private readonly System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? _keys;
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocument"/> class from raw BSON memory.
/// Initializes a new instance of the <see cref="BsonDocument" /> class from raw BSON memory.
/// </summary>
/// <param name="rawBsonData">The raw BSON data.</param>
/// <param name="keys">The optional key dictionary.</param>
public BsonDocument(Memory<byte> rawBsonData, System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? keys = null)
public BsonDocument(Memory<byte> rawBsonData, ConcurrentDictionary<ushort, string>? keys = null)
{
_rawData = rawBsonData;
_keys = keys;
}
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocument"/> class from raw BSON bytes.
/// Initializes a new instance of the <see cref="BsonDocument" /> class from raw BSON bytes.
/// </summary>
/// <param name="rawBsonData">The raw BSON data.</param>
/// <param name="keys">The optional key dictionary.</param>
public BsonDocument(byte[] rawBsonData, System.Collections.Concurrent.ConcurrentDictionary<ushort, string>? keys = null)
public BsonDocument(byte[] rawBsonData, ConcurrentDictionary<ushort, string>? keys = null)
{
_rawData = rawBsonData;
_keys = keys;
}
/// <summary>
/// Gets the raw BSON bytes
/// Gets the raw BSON bytes
/// </summary>
public ReadOnlySpan<byte> RawData => _rawData.Span;
/// <summary>
/// Gets the document size in bytes
/// Gets the document size in bytes
/// </summary>
public int Size => BitConverter.ToInt32(_rawData.Span[..4]);
/// <summary>
/// Creates a reader for this document
/// Creates a reader for this document
/// </summary>
public BsonSpanReader GetReader() => new BsonSpanReader(_rawData.Span, _keys ?? new System.Collections.Concurrent.ConcurrentDictionary<ushort, string>());
public BsonSpanReader GetReader()
{
return new BsonSpanReader(_rawData.Span,
_keys ?? new ConcurrentDictionary<ushort, string>());
}
/// <summary>
/// Tries to get a field value by name.
/// Returns false if field not found.
/// Tries to get a field value by name.
/// Returns false if field not found.
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise <see langword="null"/>.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <param name="value">When this method returns, contains the field value if found; otherwise <see langword="null" />.</param>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetString(string fieldName, out string? value)
{
value = null;
@@ -70,7 +74,7 @@ public sealed class BsonDocument
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.String)
{
@@ -85,11 +89,11 @@ public sealed class BsonDocument
}
/// <summary>
/// Tries to get an Int32 field value by name.
/// Tries to get an Int32 field value by name.
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise zero.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetInt32(string fieldName, out int value)
{
value = 0;
@@ -105,7 +109,7 @@ public sealed class BsonDocument
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.Int32)
{
@@ -120,11 +124,11 @@ public sealed class BsonDocument
}
/// <summary>
/// Tries to get an ObjectId field value by name.
/// Tries to get an ObjectId field value by name.
/// </summary>
/// <param name="fieldName">The field name.</param>
/// <param name="value">When this method returns, contains the field value if found; otherwise default.</param>
/// <returns><see langword="true"/> if the field is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the field is found; otherwise, <see langword="false" />.</returns>
public bool TryGetObjectId(string fieldName, out ObjectId value)
{
value = default;
@@ -140,7 +144,7 @@ public sealed class BsonDocument
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == fieldName && type == BsonType.ObjectId)
{
@@ -155,12 +159,13 @@ public sealed class BsonDocument
}
/// <summary>
/// Creates a new BsonDocument from field values using a builder pattern
/// Creates a new BsonDocument from field values using a builder pattern
/// </summary>
/// <param name="keyMap">The key map used for field name encoding.</param>
/// <param name="buildAction">The action that populates the builder.</param>
/// <returns>The created BSON document.</returns>
public static BsonDocument Create(System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap, Action<BsonDocumentBuilder> buildAction)
public static BsonDocument Create(ConcurrentDictionary<string, ushort> keyMap,
Action<BsonDocumentBuilder> buildAction)
{
var builder = new BsonDocumentBuilder(keyMap);
buildAction(builder);
@@ -169,19 +174,19 @@ public sealed class BsonDocument
}
/// <summary>
/// Builder for creating BSON documents
/// Builder for creating BSON documents
/// </summary>
public sealed class BsonDocumentBuilder
{
private readonly ConcurrentDictionary<string, ushort> _keyMap;
private byte[] _buffer = new byte[1024]; // Start with 1KB
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, ushort> _keyMap;
/// <summary>
/// Initializes a new instance of the <see cref="BsonDocumentBuilder"/> class.
/// Initializes a new instance of the <see cref="BsonDocumentBuilder" /> class.
/// </summary>
/// <param name="keyMap">The key map used for field name encoding.</param>
public BsonDocumentBuilder(System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap)
public BsonDocumentBuilder(ConcurrentDictionary<string, ushort> keyMap)
{
_keyMap = keyMap;
var writer = new BsonSpanWriter(_buffer, _keyMap);
@@ -189,7 +194,7 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Adds a string field to the document.
/// Adds a string field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -204,7 +209,7 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Adds an Int32 field to the document.
/// Adds an Int32 field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -219,7 +224,7 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Adds an Int64 field to the document.
/// Adds an Int64 field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -234,7 +239,7 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Adds a Boolean field to the document.
/// Adds a Boolean field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -249,7 +254,7 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Adds an ObjectId field to the document.
/// Adds an ObjectId field to the document.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The field value.</param>
@@ -264,13 +269,13 @@ public sealed class BsonDocumentBuilder
}
/// <summary>
/// Builds a BSON document from the accumulated fields.
/// Builds a BSON document from the accumulated fields.
/// </summary>
/// <returns>The constructed BSON document.</returns>
public BsonDocument Build()
{
// Layout: [int32 size][field bytes...][0x00 terminator]
var totalSize = _position + 5;
int totalSize = _position + 5;
var finalBuffer = new byte[totalSize];
BitConverter.TryWriteBytes(finalBuffer.AsSpan(0, 4), totalSize);
src/CBDD.Bson/Document/BsonType.cs
+1 -1
View File
@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// BSON type codes as defined in BSON spec
/// BSON type codes as defined in BSON spec
/// </summary>
public enum BsonType : byte
{
src/CBDD.Bson/IO/BsonBufferWriter.cs
+34 -33
View File
@@ -1,4 +1,3 @@
using System;
using System.Buffers;
using System.Buffers.Binary;
using System.Text;
@@ -6,35 +5,34 @@ using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// BSON writer that serializes to an IBufferWriter, enabling streaming serialization
/// without fixed buffer size limits.
/// BSON writer that serializes to an IBufferWriter, enabling streaming serialization
/// without fixed buffer size limits.
/// </summary>
public ref struct BsonBufferWriter
{
private IBufferWriter<byte> _writer;
private int _totalBytesWritten;
private readonly IBufferWriter<byte> _writer;
/// <summary>
/// Initializes a new instance of the <see cref="BsonBufferWriter"/> struct.
/// Initializes a new instance of the <see cref="BsonBufferWriter" /> struct.
/// </summary>
/// <param name="writer">The buffer writer to write BSON bytes to.</param>
public BsonBufferWriter(IBufferWriter<byte> writer)
{
_writer = writer;
_totalBytesWritten = 0;
Position = 0;
}
/// <summary>
/// Gets the current write position in bytes.
/// Gets the current write position in bytes.
/// </summary>
public int Position => _totalBytesWritten;
public int Position { get; private set; }
private void WriteBytes(ReadOnlySpan<byte> data)
{
var destination = _writer.GetSpan(data.Length);
data.CopyTo(destination);
_writer.Advance(data.Length);
_totalBytesWritten += data.Length;
Position += data.Length;
}
private void WriteByte(byte value)
@@ -42,11 +40,11 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(1);
span[0] = value;
_writer.Advance(1);
_totalBytesWritten++;
Position++;
}
/// <summary>
/// Writes a BSON date-time field.
/// Writes a BSON date-time field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date-time value.</param>
@@ -61,23 +59,26 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Begins writing a BSON document.
/// Begins writing a BSON document.
/// </summary>
/// <returns>The position where the document size placeholder was written.</returns>
public int BeginDocument()
{
// Write placeholder for size (4 bytes)
var sizePosition = _totalBytesWritten;
int sizePosition = Position;
var span = _writer.GetSpan(4);
// Initialize with default value (will be patched later)
span[0] = 0; span[1] = 0; span[2] = 0; span[3] = 0;
span[0] = 0;
span[1] = 0;
span[2] = 0;
span[3] = 0;
_writer.Advance(4);
_totalBytesWritten += 4;
Position += 4;
return sizePosition;
}
/// <summary>
/// Ends the current BSON document by writing the document terminator.
/// Ends the current BSON document by writing the document terminator.
/// </summary>
/// <param name="sizePosition">The position of the size placeholder for this document.</param>
public void EndDocument(int sizePosition)
@@ -90,7 +91,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Begins writing a nested BSON document field.
/// Begins writing a nested BSON document field.
/// </summary>
/// <param name="name">The field name.</param>
/// <returns>The position where the nested document size placeholder was written.</returns>
@@ -102,7 +103,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Begins writing a BSON array field.
/// Begins writing a BSON array field.
/// </summary>
/// <param name="name">The field name.</param>
/// <returns>The position where the array document size placeholder was written.</returns>
@@ -114,7 +115,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Ends the current BSON array.
/// Ends the current BSON array.
/// </summary>
/// <param name="sizePosition">The position of the size placeholder for this array.</param>
public void EndArray(int sizePosition)
@@ -129,7 +130,7 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(4);
BinaryPrimitives.WriteInt32LittleEndian(span, value);
_writer.Advance(4);
_totalBytesWritten += 4;
Position += 4;
}
private void WriteInt64Internal(long value)
@@ -137,11 +138,11 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteInt64LittleEndian(span, value);
_writer.Advance(8);
_totalBytesWritten += 8;
Position += 8;
}
/// <summary>
/// Writes a BSON ObjectId field.
/// Writes a BSON ObjectId field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The ObjectId value.</param>
@@ -153,7 +154,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Writes a BSON string field.
/// Writes a BSON string field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The string value.</param>
@@ -165,7 +166,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Writes a BSON boolean field.
/// Writes a BSON boolean field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The boolean value.</param>
@@ -177,7 +178,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Writes a BSON null field.
/// Writes a BSON null field.
/// </summary>
/// <param name="name">The field name.</param>
public void WriteNull(string name)
@@ -189,7 +190,7 @@ public ref struct BsonBufferWriter
private void WriteStringValue(string value)
{
// String: length (int32) + UTF8 bytes + null terminator
var bytes = Encoding.UTF8.GetBytes(value);
byte[] bytes = Encoding.UTF8.GetBytes(value);
WriteInt32Internal(bytes.Length + 1); // +1 for null terminator
WriteBytes(bytes);
WriteByte(0);
@@ -200,11 +201,11 @@ public ref struct BsonBufferWriter
var span = _writer.GetSpan(8);
BinaryPrimitives.WriteDoubleLittleEndian(span, value);
_writer.Advance(8);
_totalBytesWritten += 8;
Position += 8;
}
/// <summary>
/// Writes a BSON binary field.
/// Writes a BSON binary field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="data">The binary data.</param>
@@ -218,7 +219,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Writes a BSON 64-bit integer field.
/// Writes a BSON 64-bit integer field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 64-bit integer value.</param>
@@ -230,7 +231,7 @@ public ref struct BsonBufferWriter
}
/// <summary>
/// Writes a BSON double field.
/// Writes a BSON double field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The double value.</param>
@@ -243,13 +244,13 @@ public ref struct BsonBufferWriter
private void WriteCString(string value)
{
var bytes = Encoding.UTF8.GetBytes(value);
byte[] bytes = Encoding.UTF8.GetBytes(value);
WriteBytes(bytes);
WriteByte(0); // Null terminator
}
/// <summary>
/// Writes a BSON 32-bit integer field.
/// Writes a BSON 32-bit integer field.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 32-bit integer value.</param>
src/CBDD.Bson/IO/BsonSpanReader.cs
+111 -111
View File
@@ -1,187 +1,186 @@
using System;
using System.Buffers.Binary;
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// Zero-allocation BSON reader using ReadOnlySpan&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// Zero-allocation BSON reader using ReadOnlySpan&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// </summary>
public ref struct BsonSpanReader
{
private ReadOnlySpan<byte> _buffer;
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<ushort, string> _keys;
private readonly ConcurrentDictionary<ushort, string> _keys;
/// <summary>
/// Initializes a new instance of the <see cref="BsonSpanReader"/> struct.
/// Initializes a new instance of the <see cref="BsonSpanReader" /> struct.
/// </summary>
/// <param name="buffer">The BSON buffer to read.</param>
/// <param name="keys">The reverse key dictionary used for compressed element headers.</param>
public BsonSpanReader(ReadOnlySpan<byte> buffer, System.Collections.Concurrent.ConcurrentDictionary<ushort, string> keys)
public BsonSpanReader(ReadOnlySpan<byte> buffer, ConcurrentDictionary<ushort, string> keys)
{
_buffer = buffer;
_position = 0;
Position = 0;
_keys = keys;
}
/// <summary>
/// Gets the current read position in the buffer.
/// Gets the current read position in the buffer.
/// </summary>
public int Position => _position;
public int Position { get; private set; }
/// <summary>
/// Gets the number of unread bytes remaining in the buffer.
/// Gets the number of unread bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - _position;
public int Remaining => _buffer.Length - Position;
/// <summary>
/// Reads the document size (first 4 bytes of a BSON document)
/// Reads the document size (first 4 bytes of a BSON document)
/// </summary>
public int ReadDocumentSize()
{
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to read document size");
var size = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int size = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
return size;
}
/// <summary>
/// Reads a BSON element type
/// Reads a BSON element type
/// </summary>
public BsonType ReadBsonType()
{
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read BSON type");
var type = (BsonType)_buffer[_position];
_position++;
var type = (BsonType)_buffer[Position];
Position++;
return type;
}
/// <summary>
/// Reads a C-style null-terminated string (e-name in BSON spec)
/// Reads a C-style null-terminated string (e-name in BSON spec)
/// </summary>
public string ReadCString()
{
var start = _position;
while (_position < _buffer.Length && _buffer[_position] != 0)
_position++;
int start = Position;
while (Position < _buffer.Length && _buffer[Position] != 0)
Position++;
if (_position >= _buffer.Length)
if (Position >= _buffer.Length)
throw new InvalidOperationException("Unterminated C-string");
var nameBytes = _buffer.Slice(start, _position - start);
_position++; // Skip null terminator
var nameBytes = _buffer.Slice(start, Position - start);
Position++; // Skip null terminator
return Encoding.UTF8.GetString(nameBytes);
}
/// <summary>
/// Reads a C-string into a destination span. Returns the number of bytes written.
/// Reads a C-string into a destination span. Returns the number of bytes written.
/// </summary>
/// <param name="destination">The destination character span.</param>
public int ReadCString(Span<char> destination)
{
var start = _position;
while (_position < _buffer.Length && _buffer[_position] != 0)
_position++;
int start = Position;
while (Position < _buffer.Length && _buffer[Position] != 0)
Position++;
if (_position >= _buffer.Length)
if (Position >= _buffer.Length)
throw new InvalidOperationException("Unterminated C-string");
var nameBytes = _buffer.Slice(start, _position - start);
_position++; // Skip null terminator
var nameBytes = _buffer.Slice(start, Position - start);
Position++; // Skip null terminator
return Encoding.UTF8.GetChars(nameBytes, destination);
}
/// <summary>
/// Reads a BSON string (4-byte length + UTF-8 bytes + null terminator)
/// Reads a BSON string (4-byte length + UTF-8 bytes + null terminator)
/// </summary>
public string ReadString()
{
var length = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int length = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
if (length < 1)
throw new InvalidOperationException("Invalid string length");
var stringBytes = _buffer.Slice(_position, length - 1); // Exclude null terminator
_position += length;
var stringBytes = _buffer.Slice(Position, length - 1); // Exclude null terminator
Position += length;
return Encoding.UTF8.GetString(stringBytes);
}
/// <summary>
/// Reads a 32-bit integer.
/// Reads a 32-bit integer.
/// </summary>
public int ReadInt32()
{
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to read Int32");
var value = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += 4;
int value = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += 4;
return value;
}
/// <summary>
/// Reads a 64-bit integer.
/// Reads a 64-bit integer.
/// </summary>
public long ReadInt64()
{
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Int64");
var value = BinaryPrimitives.ReadInt64LittleEndian(_buffer.Slice(_position, 8));
_position += 8;
long value = BinaryPrimitives.ReadInt64LittleEndian(_buffer.Slice(Position, 8));
Position += 8;
return value;
}
/// <summary>
/// Reads a double-precision floating point value.
/// Reads a double-precision floating point value.
/// </summary>
public double ReadDouble()
{
if (Remaining < 8)
throw new InvalidOperationException("Not enough bytes to read Double");
var value = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
double value = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
return value;
}
/// <summary>
/// Reads spatial coordinates from a BSON array [X, Y].
/// Returns a (double, double) tuple.
/// Reads spatial coordinates from a BSON array [X, Y].
/// Returns a (double, double) tuple.
/// </summary>
public (double, double) ReadCoordinates()
{
// Skip array size (4 bytes)
_position += 4;
Position += 4;
// Skip element 0 header: Type(1) + Name("0\0") (3 bytes)
_position += 3;
var x = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
Position += 3;
double x = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
// Skip element 1 header: Type(1) + Name("1\0") (3 bytes)
_position += 3;
var y = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(_position, 8));
_position += 8;
Position += 3;
double y = BinaryPrimitives.ReadDoubleLittleEndian(_buffer.Slice(Position, 8));
Position += 8;
// Skip end of array marker (1 byte)
_position++;
Position++;
return (x, y);
}
/// <summary>
/// Reads a Decimal128 value.
/// Reads a Decimal128 value.
/// </summary>
public decimal ReadDecimal128()
{
@@ -189,75 +188,75 @@ public ref struct BsonSpanReader
throw new InvalidOperationException("Not enough bytes to read Decimal128");
var bits = new int[4];
bits[0] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
bits[1] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 4, 4));
bits[2] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 8, 4));
bits[3] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position + 12, 4));
_position += 16;
bits[0] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
bits[1] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 4, 4));
bits[2] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 8, 4));
bits[3] = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position + 12, 4));
Position += 16;
return new decimal(bits);
}
/// <summary>
/// Reads a boolean value.
/// Reads a boolean value.
/// </summary>
public bool ReadBoolean()
{
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read Boolean");
var value = _buffer[_position] != 0;
_position++;
bool value = _buffer[Position] != 0;
Position++;
return value;
}
/// <summary>
/// Reads a BSON DateTime (UTC milliseconds since Unix epoch)
/// Reads a BSON DateTime (UTC milliseconds since Unix epoch)
/// </summary>
public DateTime ReadDateTime()
{
var milliseconds = ReadInt64();
long milliseconds = ReadInt64();
return DateTimeOffset.FromUnixTimeMilliseconds(milliseconds).UtcDateTime;
}
/// <summary>
/// Reads a BSON DateTime as DateTimeOffset (UTC milliseconds since Unix epoch)
/// Reads a BSON DateTime as DateTimeOffset (UTC milliseconds since Unix epoch)
/// </summary>
public DateTimeOffset ReadDateTimeOffset()
{
var milliseconds = ReadInt64();
long milliseconds = ReadInt64();
return DateTimeOffset.FromUnixTimeMilliseconds(milliseconds);
}
/// <summary>
/// Reads a TimeSpan from BSON Int64 (ticks)
/// Reads a TimeSpan from BSON Int64 (ticks)
/// </summary>
public TimeSpan ReadTimeSpan()
{
var ticks = ReadInt64();
long ticks = ReadInt64();
return TimeSpan.FromTicks(ticks);
}
/// <summary>
/// Reads a DateOnly from BSON Int32 (day number)
/// Reads a DateOnly from BSON Int32 (day number)
/// </summary>
public DateOnly ReadDateOnly()
{
var dayNumber = ReadInt32();
int dayNumber = ReadInt32();
return DateOnly.FromDayNumber(dayNumber);
}
/// <summary>
/// Reads a TimeOnly from BSON Int64 (ticks)
/// Reads a TimeOnly from BSON Int64 (ticks)
/// </summary>
public TimeOnly ReadTimeOnly()
{
var ticks = ReadInt64();
long ticks = ReadInt64();
return new TimeOnly(ticks);
}
/// <summary>
/// Reads a GUID value.
/// Reads a GUID value.
/// </summary>
public Guid ReadGuid()
{
@@ -265,42 +264,42 @@ public ref struct BsonSpanReader
}
/// <summary>
/// Reads a BSON ObjectId (12 bytes)
/// Reads a BSON ObjectId (12 bytes)
/// </summary>
public ObjectId ReadObjectId()
{
if (Remaining < 12)
throw new InvalidOperationException("Not enough bytes to read ObjectId");
var oidBytes = _buffer.Slice(_position, 12);
_position += 12;
var oidBytes = _buffer.Slice(Position, 12);
Position += 12;
return new ObjectId(oidBytes);
}
/// <summary>
/// Reads binary data (subtype + length + bytes)
/// Reads binary data (subtype + length + bytes)
/// </summary>
/// <param name="subtype">When this method returns, contains the BSON binary subtype.</param>
public ReadOnlySpan<byte> ReadBinary(out byte subtype)
{
var length = ReadInt32();
int length = ReadInt32();
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read binary subtype");
subtype = _buffer[_position];
_position++;
subtype = _buffer[Position];
Position++;
if (Remaining < length)
throw new InvalidOperationException("Not enough bytes to read binary data");
var data = _buffer.Slice(_position, length);
_position += length;
var data = _buffer.Slice(Position, length);
Position += length;
return data;
}
/// <summary>
/// Skips the current value based on type
/// Skips the current value based on type
/// </summary>
/// <param name="type">The BSON type of the value to skip.</param>
public void SkipValue(BsonType type)
@@ -308,37 +307,37 @@ public ref struct BsonSpanReader
switch (type)
{
case BsonType.Double:
_position += 8;
Position += 8;
break;
case BsonType.String:
var stringLength = ReadInt32();
_position += stringLength;
int stringLength = ReadInt32();
Position += stringLength;
break;
case BsonType.Document:
case BsonType.Array:
var docLength = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
_position += docLength;
int docLength = BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
Position += docLength;
break;
case BsonType.Binary:
var binaryLength = ReadInt32();
_position += 1 + binaryLength; // subtype + data
int binaryLength = ReadInt32();
Position += 1 + binaryLength; // subtype + data
break;
case BsonType.ObjectId:
_position += 12;
Position += 12;
break;
case BsonType.Boolean:
_position += 1;
Position += 1;
break;
case BsonType.DateTime:
case BsonType.Int64:
case BsonType.Timestamp:
_position += 8;
Position += 8;
break;
case BsonType.Decimal128:
_position += 16;
Position += 16;
break;
case BsonType.Int32:
_position += 4;
Position += 4;
break;
case BsonType.Null:
// No data
@@ -349,48 +348,49 @@ public ref struct BsonSpanReader
}
/// <summary>
/// Reads a single byte.
/// Reads a single byte.
/// </summary>
public byte ReadByte()
{
if (Remaining < 1)
throw new InvalidOperationException("Not enough bytes to read byte");
var value = _buffer[_position];
_position++;
byte value = _buffer[Position];
Position++;
return value;
}
/// <summary>
/// Peeks a 32-bit integer at the current position without advancing.
/// Peeks a 32-bit integer at the current position without advancing.
/// </summary>
public int PeekInt32()
{
if (Remaining < 4)
throw new InvalidOperationException("Not enough bytes to peek Int32");
return BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(_position, 4));
return BinaryPrimitives.ReadInt32LittleEndian(_buffer.Slice(Position, 4));
}
/// <summary>
/// Reads an element header key identifier and resolves it to a key name.
/// Reads an element header key identifier and resolves it to a key name.
/// </summary>
public string ReadElementHeader()
{
if (Remaining < 2)
throw new InvalidOperationException("Not enough bytes to read BSON element key ID");
var id = BinaryPrimitives.ReadUInt16LittleEndian(_buffer.Slice(_position, 2));
_position += 2;
ushort id = BinaryPrimitives.ReadUInt16LittleEndian(_buffer.Slice(Position, 2));
Position += 2;
if (!_keys.TryGetValue(id, out var key))
{
if (!_keys.TryGetValue(id, out string? key))
throw new InvalidOperationException($"BSON Key ID {id} not found in reverse key dictionary.");
}
return key;
}
/// <summary>
/// Returns a span containing all unread bytes.
/// Returns a span containing all unread bytes.
/// </summary>
public ReadOnlySpan<byte> RemainingBytes() => _buffer[_position..];
public ReadOnlySpan<byte> RemainingBytes()
{
return _buffer[Position..];
}
}
src/CBDD.Bson/IO/BsonSpanWriter.cs
+113 -115
View File
@@ -1,103 +1,101 @@
using System;
using System.Buffers.Binary;
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// Zero-allocation BSON writer using Span&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// Zero-allocation BSON writer using Span&lt;byte&gt;.
/// Implemented as ref struct to ensure stack-only allocation.
/// </summary>
public ref struct BsonSpanWriter
{
private Span<byte> _buffer;
private int _position;
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, ushort> _keyMap;
private readonly ConcurrentDictionary<string, ushort> _keyMap;
/// <summary>
/// Initializes a new instance of the <see cref="BsonSpanWriter"/> struct.
/// Initializes a new instance of the <see cref="BsonSpanWriter" /> struct.
/// </summary>
/// <param name="buffer">The destination buffer to write BSON bytes into.</param>
/// <param name="keyMap">The cached key-name to key-id mapping.</param>
public BsonSpanWriter(Span<byte> buffer, System.Collections.Concurrent.ConcurrentDictionary<string, ushort> keyMap)
public BsonSpanWriter(Span<byte> buffer, ConcurrentDictionary<string, ushort> keyMap)
{
_buffer = buffer;
_keyMap = keyMap;
_position = 0;
Position = 0;
}
/// <summary>
/// Gets the current write position in the buffer.
/// Gets the current write position in the buffer.
/// </summary>
public int Position => _position;
public int Position { get; private set; }
/// <summary>
/// Gets the number of bytes remaining in the buffer.
/// Gets the number of bytes remaining in the buffer.
/// </summary>
public int Remaining => _buffer.Length - _position;
public int Remaining => _buffer.Length - Position;
/// <summary>
/// Writes document size placeholder and returns the position to patch later
/// Writes document size placeholder and returns the position to patch later
/// </summary>
public int WriteDocumentSizePlaceholder()
{
var sizePosition = _position;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), 0);
_position += 4;
int sizePosition = Position;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), 0);
Position += 4;
return sizePosition;
}
/// <summary>
/// Patches the document size at the given position
/// Patches the document size at the given position
/// </summary>
/// <param name="sizePosition">The position where the size placeholder was written.</param>
public void PatchDocumentSize(int sizePosition)
{
var size = _position - sizePosition;
int size = Position - sizePosition;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(sizePosition, 4), size);
}
/// <summary>
/// Writes a BSON element header (type + name)
/// Writes a BSON element header (type + name)
/// </summary>
/// <param name="type">The BSON element type.</param>
/// <param name="name">The field name.</param>
public void WriteElementHeader(BsonType type, string name)
{
_buffer[_position] = (byte)type;
_position++;
_buffer[Position] = (byte)type;
Position++;
if (!_keyMap.TryGetValue(name, out var id))
{
throw new InvalidOperationException($"BSON Key '{name}' not found in dictionary cache. Ensure all keys are registered before serialization.");
}
if (!_keyMap.TryGetValue(name, out ushort id))
throw new InvalidOperationException(
$"BSON Key '{name}' not found in dictionary cache. Ensure all keys are registered before serialization.");
BinaryPrimitives.WriteUInt16LittleEndian(_buffer.Slice(_position, 2), id);
_position += 2;
BinaryPrimitives.WriteUInt16LittleEndian(_buffer.Slice(Position, 2), id);
Position += 2;
}
/// <summary>
/// Writes a C-style null-terminated string
/// Writes a C-style null-terminated string
/// </summary>
private void WriteCString(string value)
{
var bytesWritten = Encoding.UTF8.GetBytes(value, _buffer[_position..]);
_position += bytesWritten;
_buffer[_position] = 0; // Null terminator
_position++;
int bytesWritten = Encoding.UTF8.GetBytes(value, _buffer[Position..]);
Position += bytesWritten;
_buffer[Position] = 0; // Null terminator
Position++;
}
/// <summary>
/// Writes end-of-document marker
/// Writes end-of-document marker
/// </summary>
public void WriteEndOfDocument()
{
_buffer[_position] = 0;
_position++;
_buffer[Position] = 0;
Position++;
}
/// <summary>
/// Writes a BSON string element
/// Writes a BSON string element
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The string value.</param>
@@ -105,58 +103,58 @@ public ref struct BsonSpanWriter
{
WriteElementHeader(BsonType.String, name);
var valueBytes = Encoding.UTF8.GetByteCount(value);
var stringLength = valueBytes + 1; // Include null terminator
int valueBytes = Encoding.UTF8.GetByteCount(value);
int stringLength = valueBytes + 1; // Include null terminator
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), stringLength);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), stringLength);
Position += 4;
Encoding.UTF8.GetBytes(value, _buffer[_position..]);
_position += valueBytes;
Encoding.UTF8.GetBytes(value, _buffer[Position..]);
Position += valueBytes;
_buffer[_position] = 0; // Null terminator
_position++;
_buffer[Position] = 0; // Null terminator
Position++;
}
/// <summary>
/// Writes a BSON int32 element.
/// Writes a BSON int32 element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 32-bit integer value.</param>
public void WriteInt32(string name, int value)
{
WriteElementHeader(BsonType.Int32, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), value);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), value);
Position += 4;
}
/// <summary>
/// Writes a BSON int64 element.
/// Writes a BSON int64 element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The 64-bit integer value.</param>
public void WriteInt64(string name, long value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value);
Position += 8;
}
/// <summary>
/// Writes a BSON double element.
/// Writes a BSON double element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The double-precision value.</param>
public void WriteDouble(string name, double value)
{
WriteElementHeader(BsonType.Double, name);
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), value);
_position += 8;
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), value);
Position += 8;
}
/// <summary>
/// Writes spatial coordinates as a BSON array [X, Y].
/// Optimized for (double, double) tuples.
/// Writes spatial coordinates as a BSON array [X, Y].
/// Optimized for (double, double) tuples.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="coordinates">The coordinate tuple as (X, Y).</param>
@@ -164,32 +162,32 @@ public ref struct BsonSpanWriter
{
WriteElementHeader(BsonType.Array, name);
var startPos = _position;
_position += 4; // Placeholder for array size
int startPos = Position;
Position += 4; // Placeholder for array size
// Element 0: X
_buffer[_position++] = (byte)BsonType.Double;
_buffer[_position++] = 0x30; // '0'
_buffer[_position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), coordinates.Item1);
_position += 8;
_buffer[Position++] = (byte)BsonType.Double;
_buffer[Position++] = 0x30; // '0'
_buffer[Position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), coordinates.Item1);
Position += 8;
// Element 1: Y
_buffer[_position++] = (byte)BsonType.Double;
_buffer[_position++] = 0x31; // '1'
_buffer[_position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(_position, 8), coordinates.Item2);
_position += 8;
_buffer[Position++] = (byte)BsonType.Double;
_buffer[Position++] = 0x31; // '1'
_buffer[Position++] = 0x00; // Null
BinaryPrimitives.WriteDoubleLittleEndian(_buffer.Slice(Position, 8), coordinates.Item2);
Position += 8;
_buffer[_position++] = 0x00; // End of array marker
_buffer[Position++] = 0x00; // End of array marker
// Patch array size
var size = _position - startPos;
int size = Position - startPos;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(startPos, 4), size);
}
/// <summary>
/// Writes a BSON Decimal128 element from a <see cref="decimal"/> value.
/// Writes a BSON Decimal128 element from a <see cref="decimal" /> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The decimal value.</param>
@@ -198,90 +196,90 @@ public ref struct BsonSpanWriter
WriteElementHeader(BsonType.Decimal128, name);
// Note: usage of C# decimal bits for round-trip fidelity within ZB.MOM.WW.CBDD.
// This makes it compatible with CBDD Reader but strictly speaking not standard IEEE 754-2008 Decimal128.
var bits = decimal.GetBits(value);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), bits[0]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 4, 4), bits[1]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 8, 4), bits[2]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position + 12, 4), bits[3]);
_position += 16;
int[] bits = decimal.GetBits(value);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), bits[0]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 4, 4), bits[1]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 8, 4), bits[2]);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position + 12, 4), bits[3]);
Position += 16;
}
/// <summary>
/// Writes a BSON boolean element.
/// Writes a BSON boolean element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The boolean value.</param>
public void WriteBoolean(string name, bool value)
{
WriteElementHeader(BsonType.Boolean, name);
_buffer[_position] = (byte)(value ? 1 : 0);
_position++;
_buffer[Position] = (byte)(value ? 1 : 0);
Position++;
}
/// <summary>
/// Writes a BSON UTC datetime element.
/// Writes a BSON UTC datetime element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date and time value.</param>
public void WriteDateTime(string name, DateTime value)
{
WriteElementHeader(BsonType.DateTime, name);
var milliseconds = new DateTimeOffset(value.ToUniversalTime()).ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), milliseconds);
_position += 8;
long milliseconds = new DateTimeOffset(value.ToUniversalTime()).ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), milliseconds);
Position += 8;
}
/// <summary>
/// Writes a BSON UTC datetime element from a <see cref="DateTimeOffset"/> value.
/// Writes a BSON UTC datetime element from a <see cref="DateTimeOffset" /> value.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date and time offset value.</param>
public void WriteDateTimeOffset(string name, DateTimeOffset value)
{
WriteElementHeader(BsonType.DateTime, name);
var milliseconds = value.ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), milliseconds);
_position += 8;
long milliseconds = value.ToUnixTimeMilliseconds();
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), milliseconds);
Position += 8;
}
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeSpan"/>.
/// Writes a BSON int64 element containing ticks from a <see cref="TimeSpan" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time span value.</param>
public void WriteTimeSpan(string name, TimeSpan value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value.Ticks);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value.Ticks);
Position += 8;
}
/// <summary>
/// Writes a BSON int32 element containing the <see cref="DateOnly.DayNumber"/>.
/// Writes a BSON int32 element containing the <see cref="DateOnly.DayNumber" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The date-only value.</param>
public void WriteDateOnly(string name, DateOnly value)
{
WriteElementHeader(BsonType.Int32, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), value.DayNumber);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), value.DayNumber);
Position += 4;
}
/// <summary>
/// Writes a BSON int64 element containing ticks from a <see cref="TimeOnly"/>.
/// Writes a BSON int64 element containing ticks from a <see cref="TimeOnly" />.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The time-only value.</param>
public void WriteTimeOnly(string name, TimeOnly value)
{
WriteElementHeader(BsonType.Int64, name);
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(_position, 8), value.Ticks);
_position += 8;
BinaryPrimitives.WriteInt64LittleEndian(_buffer.Slice(Position, 8), value.Ticks);
Position += 8;
}
/// <summary>
/// Writes a GUID as a BSON string element.
/// Writes a GUID as a BSON string element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The GUID value.</param>
@@ -291,19 +289,19 @@ public ref struct BsonSpanWriter
}
/// <summary>
/// Writes a BSON ObjectId element.
/// Writes a BSON ObjectId element.
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="value">The ObjectId value.</param>
public void WriteObjectId(string name, ObjectId value)
{
WriteElementHeader(BsonType.ObjectId, name);
value.WriteTo(_buffer.Slice(_position, 12));
_position += 12;
value.WriteTo(_buffer.Slice(Position, 12));
Position += 12;
}
/// <summary>
/// Writes a BSON null element.
/// Writes a BSON null element.
/// </summary>
/// <param name="name">The field name.</param>
public void WriteNull(string name)
@@ -313,7 +311,7 @@ public ref struct BsonSpanWriter
}
/// <summary>
/// Writes binary data
/// Writes binary data
/// </summary>
/// <param name="name">The field name.</param>
/// <param name="data">The binary payload.</param>
@@ -322,18 +320,18 @@ public ref struct BsonSpanWriter
{
WriteElementHeader(BsonType.Binary, name);
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(_position, 4), data.Length);
_position += 4;
BinaryPrimitives.WriteInt32LittleEndian(_buffer.Slice(Position, 4), data.Length);
Position += 4;
_buffer[_position] = subtype;
_position++;
_buffer[Position] = subtype;
Position++;
data.CopyTo(_buffer[_position..]);
_position += data.Length;
data.CopyTo(_buffer[Position..]);
Position += data.Length;
}
/// <summary>
/// Begins writing a subdocument and returns the size position to patch later
/// Begins writing a subdocument and returns the size position to patch later
/// </summary>
/// <param name="name">The field name for the subdocument.</param>
public int BeginDocument(string name)
@@ -343,7 +341,7 @@ public ref struct BsonSpanWriter
}
/// <summary>
/// Begins writing the root document and returns the size position to patch later
/// Begins writing the root document and returns the size position to patch later
/// </summary>
public int BeginDocument()
{
@@ -351,9 +349,9 @@ public ref struct BsonSpanWriter
}
/// <summary>
/// Ends the current document
/// Ends the current document
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder"/>.</param>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder" />.</param>
public void EndDocument(int sizePosition)
{
WriteEndOfDocument();
@@ -361,7 +359,7 @@ public ref struct BsonSpanWriter
}
/// <summary>
/// Begins writing a BSON array and returns the size position to patch later
/// Begins writing a BSON array and returns the size position to patch later
/// </summary>
/// <param name="name">The field name for the array.</param>
public int BeginArray(string name)
@@ -371,9 +369,9 @@ public ref struct BsonSpanWriter
}
/// <summary>
/// Ends the current BSON array
/// Ends the current BSON array
/// </summary>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder"/>.</param>
/// <param name="sizePosition">The position returned by <see cref="WriteDocumentSizePlaceholder" />.</param>
public void EndArray(int sizePosition)
{
WriteEndOfDocument();
src/CBDD.Bson/Metadata/Attributes.cs
+8 -11
View File
@@ -1,14 +1,11 @@
using System;
namespace ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Bson
[AttributeUsage(AttributeTargets.Property)]
public class BsonIdAttribute : Attribute
{
}
[AttributeUsage(AttributeTargets.Property)]
public class BsonIgnoreAttribute : Attribute
{
[AttributeUsage(AttributeTargets.Property)]
public class BsonIdAttribute : Attribute
{
}
[AttributeUsage(AttributeTargets.Property)]
public class BsonIgnoreAttribute : Attribute
{
}
}
src/CBDD.Bson/Schema/BsonField.cs
+25 -22
View File
@@ -1,39 +1,39 @@
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public partial class BsonField
public class BsonField
{
/// <summary>
/// Gets the field name.
/// Gets the field name.
/// </summary>
public required string Name { get; init; }
/// <summary>
/// Gets the field BSON type.
/// Gets the field BSON type.
/// </summary>
public BsonType Type { get; init; }
/// <summary>
/// Gets a value indicating whether the field is nullable.
/// Gets a value indicating whether the field is nullable.
/// </summary>
public bool IsNullable { get; init; }
/// <summary>
/// Gets the nested schema when this field is a document.
/// Gets the nested schema when this field is a document.
/// </summary>
public BsonSchema? NestedSchema { get; init; }
/// <summary>
/// Gets the array item type when this field is an array.
/// Gets the array item type when this field is an array.
/// </summary>
public BsonType? ArrayItemType { get; init; }
/// <summary>
/// Writes this field definition to BSON.
/// Writes this field definition to BSON.
/// </summary>
/// <param name="writer">The BSON writer.</param>
public void ToBson(ref BsonSpanWriter writer)
{
var size = writer.BeginDocument();
int size = writer.BeginDocument();
writer.WriteString("n", Name);
writer.WriteInt32("t", (int)Type);
writer.WriteBoolean("b", IsNullable);
@@ -44,16 +44,13 @@ public partial class BsonField
NestedSchema.ToBson(ref writer);
}
if (ArrayItemType != null)
{
writer.WriteInt32("a", (int)ArrayItemType.Value);
}
if (ArrayItemType != null) writer.WriteInt32("a", (int)ArrayItemType.Value);
writer.EndDocument(size);
}
/// <summary>
/// Reads a field definition from BSON.
/// Reads a field definition from BSON.
/// </summary>
/// <param name="reader">The BSON reader.</param>
/// <returns>The deserialized field.</returns>
@@ -61,9 +58,9 @@ public partial class BsonField
{
reader.ReadInt32(); // Read doc size
string name = "";
BsonType type = BsonType.Null;
bool isNullable = false;
var name = "";
var type = BsonType.Null;
var isNullable = false;
BsonSchema? nestedSchema = null;
BsonType? arrayItemType = null;
@@ -72,7 +69,7 @@ public partial class BsonField
var btype = reader.ReadBsonType();
if (btype == BsonType.EndOfDocument) break;
var key = reader.ReadElementHeader();
string key = reader.ReadElementHeader();
switch (key)
{
case "n": name = reader.ReadString(); break;
@@ -95,7 +92,7 @@ public partial class BsonField
}
/// <summary>
/// Computes a hash representing the field definition.
/// Computes a hash representing the field definition.
/// </summary>
/// <returns>The computed hash value.</returns>
public long GetHash()
@@ -110,10 +107,10 @@ public partial class BsonField
}
/// <summary>
/// Determines whether this field is equal to another field.
/// Determines whether this field is equal to another field.
/// </summary>
/// <param name="other">The other field.</param>
/// <returns><see langword="true"/> if the fields are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the fields are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(BsonField? other)
{
if (other == null) return false;
@@ -121,8 +118,14 @@ public partial class BsonField
}
/// <inheritdoc />
public override bool Equals(object? obj) => Equals(obj as BsonField);
public override bool Equals(object? obj)
{
return Equals(obj as BsonField);
}
/// <inheritdoc />
public override int GetHashCode() => (int)GetHash();
public override int GetHashCode()
{
return (int)GetHash();
}
}
src/CBDD.Bson/Schema/BsonSchema.cs
+26 -25
View File
@@ -1,45 +1,46 @@
namespace ZB.MOM.WW.CBDD.Bson.Schema;
public partial class BsonSchema
public class BsonSchema
{
/// <summary>
/// Gets or sets the schema title.
/// Gets or sets the schema title.
/// </summary>
public string? Title { get; set; }
/// <summary>
/// Gets or sets the schema version.
/// Gets or sets the schema version.
/// </summary>
public int? Version { get; set; }
/// <summary>
/// Gets the schema fields.
/// Gets the schema fields.
/// </summary>
public List<BsonField> Fields { get; } = new();
/// <summary>
/// Serializes this schema instance to BSON.
/// Serializes this schema instance to BSON.
/// </summary>
/// <param name="writer">The BSON writer to write into.</param>
public void ToBson(ref BsonSpanWriter writer)
{
var size = writer.BeginDocument();
int size = writer.BeginDocument();
if (Title != null) writer.WriteString("t", Title);
if (Version != null) writer.WriteInt32("_v", Version.Value);
var fieldsSize = writer.BeginArray("f");
for (int i = 0; i < Fields.Count; i++)
int fieldsSize = writer.BeginArray("f");
for (var i = 0; i < Fields.Count; i++)
{
writer.WriteElementHeader(BsonType.Document, i.ToString());
Fields[i].ToBson(ref writer);
}
writer.EndArray(fieldsSize);
writer.EndDocument(size);
}
/// <summary>
/// Deserializes a schema instance from BSON.
/// Deserializes a schema instance from BSON.
/// </summary>
/// <param name="reader">The BSON reader to read from.</param>
/// <returns>The deserialized schema.</returns>
@@ -54,7 +55,7 @@ public partial class BsonSchema
var btype = reader.ReadBsonType();
if (btype == BsonType.EndOfDocument) break;
var key = reader.ReadElementHeader();
string key = reader.ReadElementHeader();
switch (key)
{
case "t": schema.Title = reader.ReadString(); break;
@@ -68,6 +69,7 @@ public partial class BsonSchema
reader.ReadElementHeader(); // index
schema.Fields.Add(BsonField.FromBson(ref reader));
}
break;
default: reader.SkipValue(btype); break;
}
@@ -77,25 +79,22 @@ public partial class BsonSchema
}
/// <summary>
/// Computes a hash value for this schema based on its contents.
/// Computes a hash value for this schema based on its contents.
/// </summary>
/// <returns>The computed hash value.</returns>
public long GetHash()
{
var hash = new HashCode();
hash.Add(Title);
foreach (var field in Fields)
{
hash.Add(field.GetHash());
}
foreach (var field in Fields) hash.Add(field.GetHash());
return hash.ToHashCode();
}
/// <summary>
/// Determines whether this schema is equal to another schema.
/// Determines whether this schema is equal to another schema.
/// </summary>
/// <param name="other">The schema to compare with.</param>
/// <returns><see langword="true"/> when schemas are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when schemas are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(BsonSchema? other)
{
if (other == null) return false;
@@ -103,13 +102,19 @@ public partial class BsonSchema
}
/// <inheritdoc />
public override bool Equals(object? obj) => Equals(obj as BsonSchema);
public override bool Equals(object? obj)
{
return Equals(obj as BsonSchema);
}
/// <inheritdoc />
public override int GetHashCode() => (int)GetHash();
public override int GetHashCode()
{
return (int)GetHash();
}
/// <summary>
/// Enumerates all field keys in this schema, including nested schema keys.
/// Enumerates all field keys in this schema, including nested schema keys.
/// </summary>
/// <returns>An enumerable of field keys.</returns>
public IEnumerable<string> GetAllKeys()
@@ -118,12 +123,8 @@ public partial class BsonSchema
{
yield return field.Name;
if (field.NestedSchema != null)
{
foreach (var nestedKey in field.NestedSchema.GetAllKeys())
{
foreach (string nestedKey in field.NestedSchema.GetAllKeys())
yield return nestedKey;
}
}
}
}
}
src/CBDD.Bson/Types/ObjectId.cs
+36 -22
View File
@@ -1,11 +1,10 @@
using System;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Bson;
/// <summary>
/// 12-byte ObjectId compatible with MongoDB ObjectId.
/// Implemented as readonly struct for zero allocation.
/// 12-byte ObjectId compatible with MongoDB ObjectId.
/// Implemented as readonly struct for zero allocation.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 12)]
public readonly struct ObjectId : IEquatable<ObjectId>
@@ -14,17 +13,17 @@ public readonly struct ObjectId : IEquatable<ObjectId>
[FieldOffset(4)] private readonly long _randomAndCounter;
/// <summary>
/// Empty ObjectId (all zeros)
/// Empty ObjectId (all zeros)
/// </summary>
public static readonly ObjectId Empty = new ObjectId(0, 0);
public static readonly ObjectId Empty = new(0, 0);
/// <summary>
/// Maximum ObjectId (all 0xFF bytes) - useful for range queries
/// Maximum ObjectId (all 0xFF bytes) - useful for range queries
/// </summary>
public static readonly ObjectId MaxValue = new ObjectId(int.MaxValue, long.MaxValue);
public static readonly ObjectId MaxValue = new(int.MaxValue, long.MaxValue);
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId"/> struct from raw bytes.
/// Initializes a new instance of the <see cref="ObjectId" /> struct from raw bytes.
/// </summary>
/// <param name="bytes">The 12-byte ObjectId value.</param>
public ObjectId(ReadOnlySpan<byte> bytes)
@@ -37,7 +36,7 @@ public readonly struct ObjectId : IEquatable<ObjectId>
}
/// <summary>
/// Initializes a new instance of the <see cref="ObjectId"/> struct from its components.
/// Initializes a new instance of the <see cref="ObjectId" /> struct from its components.
/// </summary>
/// <param name="timestamp">The Unix timestamp portion.</param>
/// <param name="randomAndCounter">The random and counter portion.</param>
@@ -48,17 +47,17 @@ public readonly struct ObjectId : IEquatable<ObjectId>
}
/// <summary>
/// Creates a new ObjectId with current timestamp
/// Creates a new ObjectId with current timestamp
/// </summary>
public static ObjectId NewObjectId()
{
var timestamp = (int)DateTimeOffset.UtcNow.ToUnixTimeSeconds();
var random = Random.Shared.NextInt64();
long random = Random.Shared.NextInt64();
return new ObjectId(timestamp, random);
}
/// <summary>
/// Writes the ObjectId to the destination span (must be 12 bytes)
/// Writes the ObjectId to the destination span (must be 12 bytes)
/// </summary>
/// <param name="destination">The destination span to write into.</param>
public void WriteTo(Span<byte> destination)
@@ -71,7 +70,7 @@ public readonly struct ObjectId : IEquatable<ObjectId>
}
/// <summary>
/// Converts ObjectId to byte array
/// Converts ObjectId to byte array
/// </summary>
public byte[] ToByteArray()
{
@@ -81,26 +80,41 @@ public readonly struct ObjectId : IEquatable<ObjectId>
}
/// <summary>
/// Gets timestamp portion as UTC DateTime
/// Gets timestamp portion as UTC DateTime
/// </summary>
public DateTime Timestamp => DateTimeOffset.FromUnixTimeSeconds(_timestamp).UtcDateTime;
/// <summary>
/// Determines whether this instance and another <see cref="ObjectId"/> have the same value.
/// Determines whether this instance and another <see cref="ObjectId" /> have the same value.
/// </summary>
/// <param name="other">The object to compare with this instance.</param>
/// <returns><see langword="true"/> if the values are equal; otherwise, <see langword="false"/>.</returns>
public bool Equals(ObjectId other) =>
_timestamp == other._timestamp && _randomAndCounter == other._randomAndCounter;
/// <returns><see langword="true" /> if the values are equal; otherwise, <see langword="false" />.</returns>
public bool Equals(ObjectId other)
{
return _timestamp == other._timestamp && _randomAndCounter == other._randomAndCounter;
}
/// <inheritdoc />
public override bool Equals(object? obj) => obj is ObjectId other && Equals(other);
public override bool Equals(object? obj)
{
return obj is ObjectId other && Equals(other);
}
/// <inheritdoc />
public override int GetHashCode() => HashCode.Combine(_timestamp, _randomAndCounter);
public override int GetHashCode()
{
return HashCode.Combine(_timestamp, _randomAndCounter);
}
public static bool operator ==(ObjectId left, ObjectId right) => left.Equals(right);
public static bool operator !=(ObjectId left, ObjectId right) => !left.Equals(right);
public static bool operator ==(ObjectId left, ObjectId right)
{
return left.Equals(right);
}
public static bool operator !=(ObjectId left, ObjectId right)
{
return !left.Equals(right);
}
/// <inheritdoc />
public override string ToString()
src/CBDD.Bson/ZB.MOM.WW.CBDD.Bson.csproj
+22 -22
View File
@@ -1,28 +1,28 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Bson</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Bson</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Bson</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Bson</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PackageId>ZB.MOM.WW.CBDD.Bson</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>BSON Serialization Library for High-Performance Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<PackageId>ZB.MOM.WW.CBDD.Bson</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>BSON Serialization Library for High-Performance Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>
src/CBDD.Core/CDC/ChangeStreamDispatcher.cs
+39 -55
View File
@@ -1,21 +1,19 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Channels;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.CDC;
internal sealed class ChangeStreamDispatcher : IDisposable
{
private readonly Channel<InternalChangeEvent> _channel;
private readonly ConcurrentDictionary<string, ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>> _subscriptions = new();
private readonly ConcurrentDictionary<string, int> _payloadWatcherCounts = new();
private readonly CancellationTokenSource _cts = new();
private readonly ConcurrentDictionary<string, int> _payloadWatcherCounts = new();
private readonly ConcurrentDictionary<string, ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>>
_subscriptions = new();
/// <summary>
/// Initializes a new change stream dispatcher.
/// Initializes a new change stream dispatcher.
/// </summary>
public ChangeStreamDispatcher()
{
@@ -29,7 +27,16 @@ internal sealed class ChangeStreamDispatcher : IDisposable
}
/// <summary>
/// Publishes a change event to subscribers.
/// Releases dispatcher resources.
/// </summary>
public void Dispose()
{
_cts.Cancel();
_cts.Dispose();
}
/// <summary>
/// Publishes a change event to subscribers.
/// </summary>
/// <param name="change">The change event to publish.</param>
public void Publish(InternalChangeEvent change)
@@ -38,40 +45,38 @@ internal sealed class ChangeStreamDispatcher : IDisposable
}
/// <summary>
/// Determines whether a collection has subscribers that require payloads.
/// Determines whether a collection has subscribers that require payloads.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true"/> if payload watchers exist; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if payload watchers exist; otherwise, <see langword="false" />.</returns>
public bool HasPayloadWatchers(string collectionName)
{
return _payloadWatcherCounts.TryGetValue(collectionName, out var count) && count > 0;
return _payloadWatcherCounts.TryGetValue(collectionName, out int count) && count > 0;
}
/// <summary>
/// Determines whether a collection has any subscribers.
/// Determines whether a collection has any subscribers.
/// </summary>
/// <param name="collectionName">The collection name.</param>
/// <returns><see langword="true"/> if subscribers exist; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if subscribers exist; otherwise, <see langword="false" />.</returns>
public bool HasAnyWatchers(string collectionName)
{
return _subscriptions.TryGetValue(collectionName, out var subs) && !subs.IsEmpty;
}
/// <summary>
/// Subscribes a channel writer to collection change events.
/// Subscribes a channel writer to collection change events.
/// </summary>
/// <param name="collectionName">The collection name to subscribe to.</param>
/// <param name="capturePayload">Whether this subscriber requires event payloads.</param>
/// <param name="writer">The destination channel writer.</param>
/// <returns>An <see cref="IDisposable"/> that removes the subscription when disposed.</returns>
/// <returns>An <see cref="IDisposable" /> that removes the subscription when disposed.</returns>
public IDisposable Subscribe(string collectionName, bool capturePayload, ChannelWriter<InternalChangeEvent> writer)
{
if (capturePayload)
{
_payloadWatcherCounts.AddOrUpdate(collectionName, 1, (_, count) => count + 1);
}
if (capturePayload) _payloadWatcherCounts.AddOrUpdate(collectionName, 1, (_, count) => count + 1);
var collectionSubs = _subscriptions.GetOrAdd(collectionName, _ => new ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>());
var collectionSubs = _subscriptions.GetOrAdd(collectionName,
_ => new ConcurrentDictionary<ChannelWriter<InternalChangeEvent>, byte>());
collectionSubs.TryAdd(writer, 0);
return new Subscription(() => Unsubscribe(collectionName, capturePayload, writer));
@@ -79,15 +84,9 @@ internal sealed class ChangeStreamDispatcher : IDisposable
private void Unsubscribe(string collectionName, bool capturePayload, ChannelWriter<InternalChangeEvent> writer)
{
if (_subscriptions.TryGetValue(collectionName, out var collectionSubs))
{
collectionSubs.TryRemove(writer, out _);
}
if (_subscriptions.TryGetValue(collectionName, out var collectionSubs)) collectionSubs.TryRemove(writer, out _);
if (capturePayload)
{
_payloadWatcherCounts.AddOrUpdate(collectionName, 0, (_, count) => Math.Max(0, count - 1));
}
if (capturePayload) _payloadWatcherCounts.AddOrUpdate(collectionName, 0, (_, count) => Math.Max(0, count - 1));
}
private async Task ProcessEventsAsync()
@@ -96,45 +95,30 @@ internal sealed class ChangeStreamDispatcher : IDisposable
{
var reader = _channel.Reader;
while (await reader.WaitToReadAsync(_cts.Token))
{
while (reader.TryRead(out var @event))
{
if (_subscriptions.TryGetValue(@event.CollectionName, out var collectionSubs))
{
foreach (var writer in collectionSubs.Keys)
{
// Optimized fan-out: non-blocking TryWrite.
// If a subscriber channel is full (unlikely with Unbounded),
// we skip or drop. Usually, subscribers will also use Unbounded.
writer.TryWrite(@event);
}
}
}
}
while (reader.TryRead(out var @event))
if (_subscriptions.TryGetValue(@event.CollectionName, out var collectionSubs))
foreach (var writer in collectionSubs.Keys)
// Optimized fan-out: non-blocking TryWrite.
// If a subscriber channel is full (unlikely with Unbounded),
// we skip or drop. Usually, subscribers will also use Unbounded.
writer.TryWrite(@event);
}
catch (OperationCanceledException)
{
}
catch (OperationCanceledException) { }
catch (Exception)
{
// Internal error logging could go here
}
}
/// <summary>
/// Releases dispatcher resources.
/// </summary>
public void Dispose()
{
_cts.Cancel();
_cts.Dispose();
}
private sealed class Subscription : IDisposable
{
private readonly Action _onDispose;
private bool _disposed;
/// <summary>
/// Initializes a new subscription token.
/// Initializes a new subscription token.
/// </summary>
/// <param name="onDispose">Callback executed when the subscription is disposed.</param>
public Subscription(Action onDispose)
@@ -143,7 +127,7 @@ internal sealed class ChangeStreamDispatcher : IDisposable
}
/// <summary>
/// Disposes the subscription and unregisters the subscriber.
/// Disposes the subscription and unregisters the subscriber.
/// </summary>
public void Dispose()
{
src/CBDD.Core/CDC/ChangeStreamEvent.cs
+14 -15
View File
@@ -1,76 +1,75 @@
using System;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.CDC;
/// <summary>
/// A generic, immutable struct representing a data change in a collection.
/// A generic, immutable struct representing a data change in a collection.
/// </summary>
public readonly struct ChangeStreamEvent<TId, T> where T : class
{
/// <summary>
/// Gets the UTC timestamp when the change was recorded.
/// Gets the UTC timestamp when the change was recorded.
/// </summary>
public long Timestamp { get; init; }
/// <summary>
/// Gets the transaction identifier that produced the change.
/// Gets the transaction identifier that produced the change.
/// </summary>
public ulong TransactionId { get; init; }
/// <summary>
/// Gets the collection name where the change occurred.
/// Gets the collection name where the change occurred.
/// </summary>
public string CollectionName { get; init; }
/// <summary>
/// Gets the operation type associated with the change.
/// Gets the operation type associated with the change.
/// </summary>
public OperationType Type { get; init; }
/// <summary>
/// Gets the changed document identifier.
/// Gets the changed document identifier.
/// </summary>
public TId DocumentId { get; init; }
/// <summary>
/// The deserialized entity. Null if capturePayload was false during Watch().
/// The deserialized entity. Null if capturePayload was false during Watch().
/// </summary>
public T? Entity { get; init; }
}
/// <summary>
/// Low-level event structure used internally to transport changes before deserialization.
/// Low-level event structure used internally to transport changes before deserialization.
/// </summary>
internal readonly struct InternalChangeEvent
{
/// <summary>
/// Gets the UTC timestamp when the change was recorded.
/// Gets the UTC timestamp when the change was recorded.
/// </summary>
public long Timestamp { get; init; }
/// <summary>
/// Gets the transaction identifier that produced the change.
/// Gets the transaction identifier that produced the change.
/// </summary>
public ulong TransactionId { get; init; }
/// <summary>
/// Gets the collection name where the change occurred.
/// Gets the collection name where the change occurred.
/// </summary>
public string CollectionName { get; init; }
/// <summary>
/// Gets the operation type associated with the change.
/// Gets the operation type associated with the change.
/// </summary>
public OperationType Type { get; init; }
/// <summary>
/// Raw BSON of the Document ID.
/// Raw BSON of the Document ID.
/// </summary>
public ReadOnlyMemory<byte> IdBytes { get; init; }
/// <summary>
/// Raw BSON of the Entity. Null if payload not captured.
/// Raw BSON of the Entity. Null if payload not captured.
/// </summary>
public ReadOnlyMemory<byte>? PayloadBytes { get; init; }
}
src/CBDD.Core/CDC/ChangeStreamObservable.cs
+39 -45
View File
@@ -1,9 +1,5 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading.Channels;
using System.Threading.Tasks;
using System.Threading;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
@@ -12,14 +8,14 @@ namespace ZB.MOM.WW.CBDD.Core.CDC;
internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamEvent<TId, T>> where T : class
{
private readonly ChangeStreamDispatcher _dispatcher;
private readonly string _collectionName;
private readonly bool _capturePayload;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly string _collectionName;
private readonly ChangeStreamDispatcher _dispatcher;
private readonly ConcurrentDictionary<ushort, string> _keyReverseMap;
private readonly IDocumentMapper<TId, T> _mapper;
/// <summary>
/// Initializes a new observable wrapper for collection change events.
/// Initializes a new observable wrapper for collection change events.
/// </summary>
/// <param name="dispatcher">The dispatcher producing internal change events.</param>
/// <param name="collectionName">The collection to subscribe to.</param>
@@ -60,46 +56,43 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
return new CompositeDisposable(dispatcherSubscription, cts, channel.Writer, bridgeTask);
}
private async Task BridgeChannelToObserverAsync(ChannelReader<InternalChangeEvent> reader, IObserver<ChangeStreamEvent<TId, T>> observer, CancellationToken ct)
private async Task BridgeChannelToObserverAsync(ChannelReader<InternalChangeEvent> reader,
IObserver<ChangeStreamEvent<TId, T>> observer, CancellationToken ct)
{
try
{
while (await reader.WaitToReadAsync(ct))
{
while (reader.TryRead(out var internalEvent))
while (reader.TryRead(out var internalEvent))
try
{
try
// Deserializza ID
var eventId = _mapper.FromIndexKey(new IndexKey(internalEvent.IdBytes.ToArray()));
// Deserializza Payload (se presente)
T? entity = default;
if (internalEvent.PayloadBytes.HasValue)
entity = _mapper.Deserialize(new BsonSpanReader(internalEvent.PayloadBytes.Value.Span,
_keyReverseMap));
var externalEvent = new ChangeStreamEvent<TId, T>
{
// Deserializza ID
var eventId = _mapper.FromIndexKey(new IndexKey(internalEvent.IdBytes.ToArray()));
Timestamp = internalEvent.Timestamp,
TransactionId = internalEvent.TransactionId,
CollectionName = internalEvent.CollectionName,
Type = internalEvent.Type,
DocumentId = eventId,
Entity = entity
};
// Deserializza Payload (se presente)
T? entity = default;
if (internalEvent.PayloadBytes.HasValue)
{
entity = _mapper.Deserialize(new BsonSpanReader(internalEvent.PayloadBytes.Value.Span, _keyReverseMap));
}
var externalEvent = new ChangeStreamEvent<TId, T>
{
Timestamp = internalEvent.Timestamp,
TransactionId = internalEvent.TransactionId,
CollectionName = internalEvent.CollectionName,
Type = internalEvent.Type,
DocumentId = eventId,
Entity = entity
};
observer.OnNext(externalEvent);
}
catch (Exception ex)
{
// In case of deserialization error, we notify and continue if possible
// Or we can stop the observer.
observer.OnError(ex);
}
observer.OnNext(externalEvent);
}
}
catch (Exception ex)
{
// In case of deserialization error, we notify and continue if possible
// Or we can stop the observer.
observer.OnError(ex);
}
observer.OnCompleted();
}
catch (OperationCanceledException)
@@ -114,20 +107,21 @@ internal sealed class ChangeStreamObservable<TId, T> : IObservable<ChangeStreamE
private sealed class CompositeDisposable : IDisposable
{
private readonly IDisposable _dispatcherSubscription;
private readonly CancellationTokenSource _cts;
private readonly ChannelWriter<InternalChangeEvent> _writer;
private readonly Task _bridgeTask;
private readonly CancellationTokenSource _cts;
private readonly IDisposable _dispatcherSubscription;
private readonly ChannelWriter<InternalChangeEvent> _writer;
private bool _disposed;
/// <summary>
/// Initializes a new disposable wrapper for change stream resources.
/// Initializes a new disposable wrapper for change stream resources.
/// </summary>
/// <param name="dispatcherSubscription">The dispatcher subscription handle.</param>
/// <param name="cts">The cancellation source controlling the bridge task.</param>
/// <param name="writer">The channel writer for internal change events.</param>
/// <param name="bridgeTask">The running bridge task.</param>
public CompositeDisposable(IDisposable dispatcherSubscription, CancellationTokenSource cts, ChannelWriter<InternalChangeEvent> writer, Task bridgeTask)
public CompositeDisposable(IDisposable dispatcherSubscription, CancellationTokenSource cts,
ChannelWriter<InternalChangeEvent> writer, Task bridgeTask)
{
_dispatcherSubscription = dispatcherSubscription;
_cts = cts;
src/CBDD.Core/CDC/CollectionCdcPublisher.cs
+9 -15
View File
@@ -1,26 +1,25 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.CDC;
/// <summary>
/// Handles CDC watch/notify behavior for a single collection.
/// Extracted from DocumentCollection to keep storage/query concerns separated from event plumbing.
/// Handles CDC watch/notify behavior for a single collection.
/// Extracted from DocumentCollection to keep storage/query concerns separated from event plumbing.
/// </summary>
/// <typeparam name="TId">Document identifier type.</typeparam>
/// <typeparam name="T">Document type.</typeparam>
internal sealed class CollectionCdcPublisher<TId, T> where T : class
{
private readonly ITransactionHolder _transactionHolder;
private readonly string _collectionName;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly ChangeStreamDispatcher? _dispatcher;
private readonly ConcurrentDictionary<ushort, string> _keyReverseMap;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly ITransactionHolder _transactionHolder;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionCdcPublisher"/> class.
/// Initializes a new instance of the <see cref="CollectionCdcPublisher" /> class.
/// </summary>
/// <param name="transactionHolder">The transaction holder.</param>
/// <param name="collectionName">The collection name.</param>
@@ -42,7 +41,7 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
}
/// <summary>
/// Executes Watch.
/// Executes Watch.
/// </summary>
/// <param name="capturePayload">Whether to include payload data.</param>
public IObservable<ChangeStreamEvent<TId, T>> Watch(bool capturePayload = false)
@@ -59,7 +58,7 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
}
/// <summary>
/// Executes Notify.
/// Executes Notify.
/// </summary>
/// <param name="type">The operation type.</param>
/// <param name="id">The document identifier.</param>
@@ -74,15 +73,11 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
return;
ReadOnlyMemory<byte>? payload = null;
if (!docData.IsEmpty && _dispatcher.HasPayloadWatchers(_collectionName))
{
payload = docData.ToArray();
}
if (!docData.IsEmpty && _dispatcher.HasPayloadWatchers(_collectionName)) payload = docData.ToArray();
var idBytes = _mapper.ToIndexKey(id).Data.ToArray();
byte[] idBytes = _mapper.ToIndexKey(id).Data.ToArray();
if (transaction is Transaction t)
{
t.AddChange(new InternalChangeEvent
{
Timestamp = DateTime.UtcNow.Ticks,
@@ -92,6 +87,5 @@ internal sealed class CollectionCdcPublisher<TId, T> where T : class
IdBytes = idBytes,
PayloadBytes = payload
});
}
}
}
src/CBDD.Core/Collections/BaseMappers.cs
+59 -30
View File
@@ -1,25 +1,21 @@
using System;
using System.Buffers;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using System.Linq;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson.Schema;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Collections;
/// <summary>
/// Base class for custom mappers that provides bidirectional IndexKey mapping for standard types.
/// Base class for custom mappers that provides bidirectional IndexKey mapping for standard types.
/// </summary>
public abstract class DocumentMapperBase<TId, T> : IDocumentMapper<TId, T> where T : class
{
/// <summary>
/// Gets the target collection name for the mapped entity type.
/// Gets the target collection name for the mapped entity type.
/// </summary>
public abstract string CollectionName { get; }
/// <summary>
/// Serializes an entity instance into BSON.
/// Serializes an entity instance into BSON.
/// </summary>
/// <param name="entity">The entity to serialize.</param>
/// <param name="writer">The BSON writer to write into.</param>
@@ -27,96 +23,129 @@ public abstract class DocumentMapperBase<TId, T> : IDocumentMapper<TId, T> where
public abstract int Serialize(T entity, BsonSpanWriter writer);
/// <summary>
/// Deserializes an entity instance from BSON.
/// Deserializes an entity instance from BSON.
/// </summary>
/// <param name="reader">The BSON reader to read from.</param>
/// <returns>The deserialized entity.</returns>
public abstract T Deserialize(BsonSpanReader reader);
/// <summary>
/// Gets the identifier value from an entity.
/// Gets the identifier value from an entity.
/// </summary>
/// <param name="entity">The entity to read the identifier from.</param>
/// <returns>The identifier value.</returns>
public abstract TId GetId(T entity);
/// <summary>
/// Sets the identifier value on an entity.
/// Sets the identifier value on an entity.
/// </summary>
/// <param name="entity">The entity to update.</param>
/// <param name="id">The identifier value to assign.</param>
public abstract void SetId(T entity, TId id);
/// <summary>
/// Converts a typed identifier value into an index key.
/// Converts a typed identifier value into an index key.
/// </summary>
/// <param name="id">The identifier value.</param>
/// <returns>The index key representation of the identifier.</returns>
public virtual IndexKey ToIndexKey(TId id) => IndexKey.Create(id);
public virtual IndexKey ToIndexKey(TId id)
{
return IndexKey.Create(id);
}
/// <summary>
/// Converts an index key back into a typed identifier value.
/// Converts an index key back into a typed identifier value.
/// </summary>
/// <param name="key">The index key to convert.</param>
/// <returns>The typed identifier value.</returns>
public virtual TId FromIndexKey(IndexKey key) => key.As<TId>();
public virtual TId FromIndexKey(IndexKey key)
{
return key.As<TId>();
}
/// <summary>
/// Gets all mapped field keys used by this mapper.
/// Gets all mapped field keys used by this mapper.
/// </summary>
public virtual IEnumerable<string> UsedKeys => GetSchema().GetAllKeys();
/// <summary>
/// Builds the BSON schema for the mapped entity type.
/// Builds the BSON schema for the mapped entity type.
/// </summary>
/// <returns>The generated BSON schema.</returns>
public virtual BsonSchema GetSchema() => BsonSchemaGenerator.FromType<T>();
public virtual BsonSchema GetSchema()
{
return BsonSchemaGenerator.FromType<T>();
}
}
/// <summary>
/// Base class for mappers using ObjectId as primary key.
/// Base class for mappers using ObjectId as primary key.
/// </summary>
public abstract class ObjectIdMapperBase<T> : DocumentMapperBase<ObjectId, T>, IDocumentMapper<T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(ObjectId id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(ObjectId id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override ObjectId FromIndexKey(IndexKey key) => key.As<ObjectId>();
public override ObjectId FromIndexKey(IndexKey key)
{
return key.As<ObjectId>();
}
}
/// <summary>
/// Base class for mappers using Int32 as primary key.
/// Base class for mappers using Int32 as primary key.
/// </summary>
public abstract class Int32MapperBase<T> : DocumentMapperBase<int, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(int id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(int id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override int FromIndexKey(IndexKey key) => key.As<int>();
public override int FromIndexKey(IndexKey key)
{
return key.As<int>();
}
}
/// <summary>
/// Base class for mappers using String as primary key.
/// Base class for mappers using String as primary key.
/// </summary>
public abstract class StringMapperBase<T> : DocumentMapperBase<string, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(string id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(string id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override string FromIndexKey(IndexKey key) => key.As<string>();
public override string FromIndexKey(IndexKey key)
{
return key.As<string>();
}
}
/// <summary>
/// Base class for mappers using Guid as primary key.
/// Base class for mappers using Guid as primary key.
/// </summary>
public abstract class GuidMapperBase<T> : DocumentMapperBase<Guid, T> where T : class
{
/// <inheritdoc />
public override IndexKey ToIndexKey(Guid id) => IndexKey.Create(id);
public override IndexKey ToIndexKey(Guid id)
{
return IndexKey.Create(id);
}
/// <inheritdoc />
public override Guid FromIndexKey(IndexKey key) => key.As<Guid>();
public override Guid FromIndexKey(IndexKey key)
{
return key.As<Guid>();
}
}
src/CBDD.Core/Collections/BsonSchemaGenerator.cs
+7 -20
View File
@@ -1,18 +1,17 @@
using System.Reflection;
using System.Linq;
using System.Collections;
using System.Collections.Generic;
using System.Collections.Concurrent;
using System.Reflection;
using ZB.MOM.WW.CBDD.Bson;
using System;
using ZB.MOM.WW.CBDD.Bson.Schema;
namespace ZB.MOM.WW.CBDD.Core.Collections;
public static class BsonSchemaGenerator
{
private static readonly ConcurrentDictionary<Type, BsonSchema> _cache = new();
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// Generates a BSON schema for the specified CLR type.
/// </summary>
/// <typeparam name="T">The CLR type to inspect.</typeparam>
/// <returns>The generated BSON schema.</returns>
@@ -21,10 +20,8 @@ public static class BsonSchemaGenerator
return FromType(typeof(T));
}
private static readonly ConcurrentDictionary<Type, BsonSchema> _cache = new();
/// <summary>
/// Generates a BSON schema for the specified CLR type.
/// Generates a BSON schema for the specified CLR type.
/// </summary>
/// <param name="type">The CLR type to inspect.</param>
/// <returns>The generated BSON schema.</returns>
@@ -47,10 +44,7 @@ public static class BsonSchemaGenerator
AddField(schema, prop.Name, prop.PropertyType);
}
foreach (var field in fields)
{
AddField(schema, field.Name, field.FieldType);
}
foreach (var field in fields) AddField(schema, field.Name, field.FieldType);
return schema;
}
@@ -60,10 +54,7 @@ public static class BsonSchemaGenerator
name = name.ToLowerInvariant();
// Convention: id -> _id for root document
if (name.Equals("id", StringComparison.OrdinalIgnoreCase))
{
name = "_id";
}
if (name.Equals("id", StringComparison.OrdinalIgnoreCase)) name = "_id";
var (bsonType, nestedSchema, itemType) = GetBsonType(type);
@@ -106,11 +97,9 @@ public static class BsonSchemaGenerator
// Nested Objects / Structs
// If it's not a string, not a primitive, and not an array/list, treat as Document
if (type != typeof(string) && !type.IsPrimitive && !type.IsEnum)
{
// Avoid infinite recursion?
// Simple approach: generating nested schema
return (BsonType.Document, FromType(type), null);
}
return (BsonType.Undefined, null, null);
}
@@ -126,9 +115,7 @@ public static class BsonSchemaGenerator
// If type itself is IEnumerable<T>
if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof(IEnumerable<>))
{
return type.GetGenericArguments()[0];
}
var enumerableType = type.GetInterfaces()
.FirstOrDefault(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IEnumerable<>));
src/CBDD.Core/Collections/DocumentCollection.Scan.cs
+11 -15
View File
@@ -8,8 +8,8 @@ namespace ZB.MOM.WW.CBDD.Core.Collections;
public partial class DocumentCollection<TId, T> where T : class
{
/// <summary>
/// Scans the entire collection using a raw BSON predicate.
/// This avoids deserializing documents that don't match the criteria.
/// Scans the entire collection using a raw BSON predicate.
/// This avoids deserializing documents that don't match the criteria.
/// </summary>
/// <param name="predicate">Function to evaluate raw BSON data</param>
/// <returns>Matching documents</returns>
@@ -18,8 +18,8 @@ public partial class DocumentCollection<TId, T> where T : class
if (predicate == null) throw new ArgumentNullException(nameof(predicate));
var transaction = _transactionHolder.GetCurrentTransactionOrStart();
var txnId = transaction.TransactionId;
var pageCount = _storage.PageCount;
ulong txnId = transaction.TransactionId;
uint pageCount = _storage.PageCount;
var buffer = new byte[_storage.PageSize];
var pageResults = new List<T>();
@@ -28,16 +28,13 @@ public partial class DocumentCollection<TId, T> where T : class
pageResults.Clear();
ScanPage(pageId, txnId, buffer, predicate, pageResults);
foreach (var doc in pageResults)
{
yield return doc;
}
foreach (var doc in pageResults) yield return doc;
}
}
/// <summary>
/// Scans the collection in parallel using multiple threads.
/// Useful for large collections on multi-core machines.
/// Scans the collection in parallel using multiple threads.
/// Useful for large collections on multi-core machines.
/// </summary>
/// <param name="predicate">Function to evaluate raw BSON data</param>
/// <param name="degreeOfParallelism">Number of threads to use (default: -1 = ProcessorCount)</param>
@@ -46,7 +43,7 @@ public partial class DocumentCollection<TId, T> where T : class
if (predicate == null) throw new ArgumentNullException(nameof(predicate));
var transaction = _transactionHolder.GetCurrentTransactionOrStart();
var txnId = transaction.TransactionId;
ulong txnId = transaction.TransactionId;
var pageCount = (int)_storage.PageCount;
if (degreeOfParallelism <= 0)
@@ -61,15 +58,14 @@ public partial class DocumentCollection<TId, T> where T : class
var localResults = new List<T>();
for (int i = range.Item1; i < range.Item2; i++)
{
ScanPage((uint)i, txnId, localBuffer, predicate, localResults);
}
return localResults;
});
}
private void ScanPage(uint pageId, ulong txnId, byte[] buffer, Func<BsonSpanReader, bool> predicate, List<T> results)
private void ScanPage(uint pageId, ulong txnId, byte[] buffer, Func<BsonSpanReader, bool> predicate,
List<T> results)
{
_storage.ReadPage(pageId, txnId, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
@@ -80,7 +76,7 @@ public partial class DocumentCollection<TId, T> where T : class
var slots = MemoryMarshal.Cast<byte, SlotEntry>(
buffer.AsSpan(SlottedPageHeader.Size, header.SlotCount * SlotEntry.Size));
for (int i = 0; i < header.SlotCount; i++)
for (var i = 0; i < header.SlotCount; i++)
{
var slot = slots[i];
src/CBDD.Core/Collections/DocumentCollection.cs
+667 -691
View File
File diff suppressed because it is too large Load Diff
src/CBDD.Core/Collections/IDocumentMapper.cs
+14 -17
View File
@@ -1,42 +1,39 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using System;
using System.Buffers;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson.Schema;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Collections;
/// <summary>
/// Non-generic interface for common mapper operations.
/// Non-generic interface for common mapper operations.
/// </summary>
public interface IDocumentMapper
{
/// <summary>
/// Gets the collection name handled by this mapper.
/// Gets the collection name handled by this mapper.
/// </summary>
string CollectionName { get; }
/// <summary>
/// Gets the set of document keys used during mapping.
/// Gets the set of document keys used during mapping.
/// </summary>
IEnumerable<string> UsedKeys { get; }
/// <summary>
/// Gets the BSON schema for the mapped document.
/// Gets the BSON schema for the mapped document.
/// </summary>
/// <returns>The BSON schema.</returns>
BsonSchema GetSchema();
}
/// <summary>
/// Interface for mapping between entities and BSON using zero-allocation serialization.
/// Handles bidirectional mapping between TId and IndexKey.
/// Interface for mapping between entities and BSON using zero-allocation serialization.
/// Handles bidirectional mapping between TId and IndexKey.
/// </summary>
public interface IDocumentMapper<TId, T> : IDocumentMapper where T : class
{
/// <summary>
/// Serializes an entity to BSON.
/// Serializes an entity to BSON.
/// </summary>
/// <param name="entity">The entity to serialize.</param>
/// <param name="writer">The BSON writer.</param>
@@ -44,35 +41,35 @@ public interface IDocumentMapper<TId, T> : IDocumentMapper where T : class
int Serialize(T entity, BsonSpanWriter writer);
/// <summary>
/// Deserializes an entity from BSON.
/// Deserializes an entity from BSON.
/// </summary>
/// <param name="reader">The BSON reader.</param>
/// <returns>The deserialized entity.</returns>
T Deserialize(BsonSpanReader reader);
/// <summary>
/// Gets the identifier value from an entity.
/// Gets the identifier value from an entity.
/// </summary>
/// <param name="entity">The entity.</param>
/// <returns>The identifier value.</returns>
TId GetId(T entity);
/// <summary>
/// Sets the identifier value on an entity.
/// Sets the identifier value on an entity.
/// </summary>
/// <param name="entity">The entity.</param>
/// <param name="id">The identifier value.</param>
void SetId(T entity, TId id);
/// <summary>
/// Converts an identifier to an index key.
/// Converts an identifier to an index key.
/// </summary>
/// <param name="id">The identifier value.</param>
/// <returns>The index key representation.</returns>
IndexKey ToIndexKey(TId id);
/// <summary>
/// Converts an index key back to an identifier.
/// Converts an index key back to an identifier.
/// </summary>
/// <param name="key">The index key.</param>
/// <returns>The identifier value.</returns>
@@ -80,7 +77,7 @@ public interface IDocumentMapper<TId, T> : IDocumentMapper where T : class
}
/// <summary>
/// Legacy interface for compatibility with existing ObjectId-based collections.
/// Legacy interface for compatibility with existing ObjectId-based collections.
/// </summary>
public interface IDocumentMapper<T> : IDocumentMapper<ObjectId, T> where T : class
{
src/CBDD.Core/Collections/SchemaVersion.cs
+7 -6
View File
@@ -1,21 +1,19 @@
using System;
namespace ZB.MOM.WW.CBDD.Core.Collections;
public readonly struct SchemaVersion
{
/// <summary>
/// Gets the schema version number.
/// Gets the schema version number.
/// </summary>
public int Version { get; }
/// <summary>
/// Gets the schema hash.
/// Gets the schema hash.
/// </summary>
public long Hash { get; }
/// <summary>
/// Initializes a new instance of the <see cref="SchemaVersion"/> struct.
/// Initializes a new instance of the <see cref="SchemaVersion" /> struct.
/// </summary>
/// <param name="version">The schema version number.</param>
/// <param name="hash">The schema hash.</param>
@@ -26,5 +24,8 @@ public readonly struct SchemaVersion
}
/// <inheritdoc />
public override string ToString() => $"v{Version} (0x{Hash:X16})";
public override string ToString()
{
return $"v{Version} (0x{Hash:X16})";
}
}
src/CBDD.Core/Compression/CompressedPayloadHeader.cs
+26 -25
View File
@@ -3,34 +3,34 @@ using System.Buffers.Binary;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Fixed header prefix for compressed payload blobs.
/// Fixed header prefix for compressed payload blobs.
/// </summary>
public readonly struct CompressedPayloadHeader
{
public const int Size = 16;
/// <summary>
/// Compression codec used for payload bytes.
/// Compression codec used for payload bytes.
/// </summary>
public CompressionCodec Codec { get; }
/// <summary>
/// Original uncompressed payload length.
/// Original uncompressed payload length.
/// </summary>
public int OriginalLength { get; }
/// <summary>
/// Compressed payload length.
/// Compressed payload length.
/// </summary>
public int CompressedLength { get; }
/// <summary>
/// CRC32 checksum of compressed payload bytes.
/// CRC32 checksum of compressed payload bytes.
/// </summary>
public uint Checksum { get; }
/// <summary>
/// Initializes a new instance of the <see cref="CompressedPayloadHeader"/> class.
/// Initializes a new instance of the <see cref="CompressedPayloadHeader" /> class.
/// </summary>
/// <param name="codec">Compression codec used for payload bytes.</param>
/// <param name="originalLength">Original uncompressed payload length.</param>
@@ -50,19 +50,20 @@ public readonly struct CompressedPayloadHeader
}
/// <summary>
/// Create.
/// Create.
/// </summary>
/// <param name="codec">Compression codec used for payload bytes.</param>
/// <param name="originalLength">Original uncompressed payload length.</param>
/// <param name="compressedPayload">Compressed payload bytes.</param>
public static CompressedPayloadHeader Create(CompressionCodec codec, int originalLength, ReadOnlySpan<byte> compressedPayload)
public static CompressedPayloadHeader Create(CompressionCodec codec, int originalLength,
ReadOnlySpan<byte> compressedPayload)
{
var checksum = ComputeChecksum(compressedPayload);
uint checksum = ComputeChecksum(compressedPayload);
return new CompressedPayloadHeader(codec, originalLength, compressedPayload.Length, checksum);
}
/// <summary>
/// Write To.
/// Write To.
/// </summary>
/// <param name="destination">Destination span that receives the serialized header.</param>
public void WriteTo(Span<byte> destination)
@@ -80,7 +81,7 @@ public readonly struct CompressedPayloadHeader
}
/// <summary>
/// Read From.
/// Read From.
/// </summary>
/// <param name="source">Source span containing a serialized header.</param>
public static CompressedPayloadHeader ReadFrom(ReadOnlySpan<byte> source)
@@ -89,14 +90,14 @@ public readonly struct CompressedPayloadHeader
throw new ArgumentException($"Source must be at least {Size} bytes.", nameof(source));
var codec = (CompressionCodec)source[0];
var originalLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(4, 4));
var compressedLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(8, 4));
var checksum = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(12, 4));
int originalLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(4, 4));
int compressedLength = BinaryPrimitives.ReadInt32LittleEndian(source.Slice(8, 4));
uint checksum = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(12, 4));
return new CompressedPayloadHeader(codec, originalLength, compressedLength, checksum);
}
/// <summary>
/// Validate Checksum.
/// Validate Checksum.
/// </summary>
/// <param name="compressedPayload">Compressed payload bytes to validate.</param>
public bool ValidateChecksum(ReadOnlySpan<byte> compressedPayload)
@@ -105,10 +106,13 @@ public readonly struct CompressedPayloadHeader
}
/// <summary>
/// Compute Checksum.
/// Compute Checksum.
/// </summary>
/// <param name="payload">Payload bytes.</param>
public static uint ComputeChecksum(ReadOnlySpan<byte> payload) => Crc32Calculator.Compute(payload);
public static uint ComputeChecksum(ReadOnlySpan<byte> payload)
{
return Crc32Calculator.Compute(payload);
}
private static class Crc32Calculator
{
@@ -116,15 +120,15 @@ public readonly struct CompressedPayloadHeader
private static readonly uint[] Table = CreateTable();
/// <summary>
/// Compute.
/// Compute.
/// </summary>
/// <param name="payload">Payload bytes.</param>
public static uint Compute(ReadOnlySpan<byte> payload)
{
uint crc = 0xFFFFFFFFu;
for (int i = 0; i < payload.Length; i++)
var crc = 0xFFFFFFFFu;
for (var i = 0; i < payload.Length; i++)
{
var index = (crc ^ payload[i]) & 0xFF;
uint index = (crc ^ payload[i]) & 0xFF;
crc = (crc >> 8) ^ Table[index];
}
@@ -137,10 +141,7 @@ public readonly struct CompressedPayloadHeader
for (uint i = 0; i < table.Length; i++)
{
uint value = i;
for (int bit = 0; bit < 8; bit++)
{
value = (value & 1) != 0 ? (value >> 1) ^ Polynomial : value >> 1;
}
for (var bit = 0; bit < 8; bit++) value = (value & 1) != 0 ? (value >> 1) ^ Polynomial : value >> 1;
table[i] = value;
}
src/CBDD.Core/Compression/CompressionCodec.cs
+1 -1
View File
@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Supported payload compression codecs.
/// Supported payload compression codecs.
/// </summary>
public enum CompressionCodec : byte
{
src/CBDD.Core/Compression/CompressionOptions.cs
+16 -13
View File
@@ -3,52 +3,52 @@ using System.IO.Compression;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Compression configuration for document payload processing.
/// Compression configuration for document payload processing.
/// </summary>
public sealed class CompressionOptions
{
/// <summary>
/// Default compression options (compression disabled).
/// Default compression options (compression disabled).
/// </summary>
public static CompressionOptions Default { get; } = new();
/// <summary>
/// Enables payload compression for new writes.
/// Enables payload compression for new writes.
/// </summary>
public bool EnableCompression { get; init; } = false;
/// <summary>
/// Minimum payload size (bytes) required before compression is attempted.
/// Minimum payload size (bytes) required before compression is attempted.
/// </summary>
public int MinSizeBytes { get; init; } = 1024;
/// <summary>
/// Minimum percentage of size reduction required to keep compressed output.
/// Minimum percentage of size reduction required to keep compressed output.
/// </summary>
public int MinSavingsPercent { get; init; } = 10;
/// <summary>
/// Preferred default codec for new writes.
/// Preferred default codec for new writes.
/// </summary>
public CompressionCodec Codec { get; init; } = CompressionCodec.Brotli;
/// <summary>
/// Compression level passed to codec implementations.
/// Compression level passed to codec implementations.
/// </summary>
public CompressionLevel Level { get; init; } = CompressionLevel.Fastest;
/// <summary>
/// Maximum allowed decompressed payload size.
/// Maximum allowed decompressed payload size.
/// </summary>
public int MaxDecompressedSizeBytes { get; init; } = 16 * 1024 * 1024;
/// <summary>
/// Optional maximum input size allowed for compression attempts.
/// Optional maximum input size allowed for compression attempts.
/// </summary>
public int? MaxCompressionInputBytes { get; init; }
/// <summary>
/// Normalizes and validates compression options.
/// Normalizes and validates compression options.
/// </summary>
/// <param name="options">Optional user-provided options.</param>
internal static CompressionOptions Normalize(CompressionOptions? options)
@@ -59,16 +59,19 @@ public sealed class CompressionOptions
throw new ArgumentOutOfRangeException(nameof(MinSizeBytes), "MinSizeBytes must be non-negative.");
if (candidate.MinSavingsPercent is < 0 or > 100)
throw new ArgumentOutOfRangeException(nameof(MinSavingsPercent), "MinSavingsPercent must be between 0 and 100.");
throw new ArgumentOutOfRangeException(nameof(MinSavingsPercent),
"MinSavingsPercent must be between 0 and 100.");
if (!Enum.IsDefined(candidate.Codec))
throw new ArgumentOutOfRangeException(nameof(Codec), $"Unsupported codec: {candidate.Codec}.");
if (candidate.MaxDecompressedSizeBytes <= 0)
throw new ArgumentOutOfRangeException(nameof(MaxDecompressedSizeBytes), "MaxDecompressedSizeBytes must be greater than 0.");
throw new ArgumentOutOfRangeException(nameof(MaxDecompressedSizeBytes),
"MaxDecompressedSizeBytes must be greater than 0.");
if (candidate.MaxCompressionInputBytes is <= 0)
throw new ArgumentOutOfRangeException(nameof(MaxCompressionInputBytes), "MaxCompressionInputBytes must be greater than 0 when provided.");
throw new ArgumentOutOfRangeException(nameof(MaxCompressionInputBytes),
"MaxCompressionInputBytes must be greater than 0 when provided.");
return candidate;
}
src/CBDD.Core/Compression/CompressionService.cs
+143 -121
View File
@@ -5,14 +5,14 @@ using System.IO.Compression;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Compression codec registry and utility service.
/// Compression codec registry and utility service.
/// </summary>
public sealed class CompressionService
{
private readonly ConcurrentDictionary<CompressionCodec, ICompressionCodec> _codecs = new();
/// <summary>
/// Initializes a new instance of the <see cref="CompressionService"/> class.
/// Initializes a new instance of the <see cref="CompressionService" /> class.
/// </summary>
/// <param name="additionalCodecs">Optional additional codecs to register.</param>
public CompressionService(IEnumerable<ICompressionCodec>? additionalCodecs = null)
@@ -24,14 +24,11 @@ public sealed class CompressionService
if (additionalCodecs == null)
return;
foreach (var codec in additionalCodecs)
{
RegisterCodec(codec);
}
foreach (var codec in additionalCodecs) RegisterCodec(codec);
}
/// <summary>
/// Registers or replaces a compression codec implementation.
/// Registers or replaces a compression codec implementation.
/// </summary>
/// <param name="codec">The codec implementation to register.</param>
public void RegisterCodec(ICompressionCodec codec)
@@ -41,18 +38,21 @@ public sealed class CompressionService
}
/// <summary>
/// Attempts to resolve a registered codec implementation.
/// Attempts to resolve a registered codec implementation.
/// </summary>
/// <param name="codec">The codec identifier to resolve.</param>
/// <param name="compressionCodec">When this method returns, contains the resolved codec when found.</param>
/// <returns><see langword="true"/> when a codec is registered for <paramref name="codec"/>; otherwise, <see langword="false"/>.</returns>
/// <returns>
/// <see langword="true" /> when a codec is registered for <paramref name="codec" />; otherwise,
/// <see langword="false" />.
/// </returns>
public bool TryGetCodec(CompressionCodec codec, out ICompressionCodec compressionCodec)
{
return _codecs.TryGetValue(codec, out compressionCodec!);
}
/// <summary>
/// Gets a registered codec implementation.
/// Gets a registered codec implementation.
/// </summary>
/// <param name="codec">The codec identifier to resolve.</param>
/// <returns>The registered codec implementation.</returns>
@@ -65,7 +65,7 @@ public sealed class CompressionService
}
/// <summary>
/// Compresses payload bytes using the selected codec and level.
/// Compresses payload bytes using the selected codec and level.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="codec">The codec to use.</param>
@@ -77,131 +77,40 @@ public sealed class CompressionService
}
/// <summary>
/// Decompresses payload bytes using the selected codec.
/// Decompresses payload bytes using the selected codec.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="codec">The codec to use.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, CompressionCodec codec, int expectedLength, int maxDecompressedSizeBytes)
public byte[] Decompress(ReadOnlySpan<byte> input, CompressionCodec codec, int expectedLength,
int maxDecompressedSizeBytes)
{
return GetCodec(codec).Decompress(input, expectedLength, maxDecompressedSizeBytes);
}
/// <summary>
/// Compresses and then decompresses payload bytes using the selected codec.
/// Compresses and then decompresses payload bytes using the selected codec.
/// </summary>
/// <param name="input">The payload bytes to roundtrip.</param>
/// <param name="codec">The codec to use.</param>
/// <param name="level">The compression level.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes after roundtrip.</returns>
public byte[] Roundtrip(ReadOnlySpan<byte> input, CompressionCodec codec, CompressionLevel level, int maxDecompressedSizeBytes)
public byte[] Roundtrip(ReadOnlySpan<byte> input, CompressionCodec codec, CompressionLevel level,
int maxDecompressedSizeBytes)
{
var compressed = Compress(input, codec, level);
byte[] compressed = Compress(input, codec, level);
return Decompress(compressed, codec, input.Length, maxDecompressedSizeBytes);
}
private sealed class NoneCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.None;
/// <summary>
/// Returns a copy of the input payload without compression.
/// </summary>
/// <param name="input">The payload bytes to copy.</param>
/// <param name="level">The requested compression level.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level) => input.ToArray();
/// <summary>
/// Validates and returns an uncompressed payload copy.
/// </summary>
/// <param name="input">The payload bytes to validate and copy.</param>
/// <param name="expectedLength">The expected payload length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed payload size in bytes.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
if (input.Length > maxDecompressedSizeBytes)
throw new InvalidDataException($"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
if (expectedLength >= 0 && expectedLength != input.Length)
throw new InvalidDataException($"Expected decompressed length {expectedLength}, actual {input.Length}.");
return input.ToArray();
}
}
private sealed class BrotliCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Brotli;
/// <summary>
/// Compresses payload bytes using Brotli.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new BrotliStream(stream, level, leaveOpen: true));
}
/// <summary>
/// Decompresses Brotli-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input, stream => new BrotliStream(stream, CompressionMode.Decompress, leaveOpen: true), expectedLength, maxDecompressedSizeBytes);
}
}
private sealed class DeflateCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Deflate;
/// <summary>
/// Compresses payload bytes using Deflate.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new DeflateStream(stream, level, leaveOpen: true));
}
/// <summary>
/// Decompresses Deflate-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">The expected decompressed byte length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input, stream => new DeflateStream(stream, CompressionMode.Decompress, leaveOpen: true), expectedLength, maxDecompressedSizeBytes);
}
}
private static byte[] CompressWithCodecStream(ReadOnlySpan<byte> input, Func<Stream, Stream> streamFactory)
{
using var output = new MemoryStream(capacity: input.Length);
using var output = new MemoryStream(input.Length);
using (var codecStream = streamFactory(output))
{
codecStream.Write(input);
@@ -220,31 +129,33 @@ public sealed class CompressionService
if (maxDecompressedSizeBytes <= 0)
throw new ArgumentOutOfRangeException(nameof(maxDecompressedSizeBytes));
using var compressed = new MemoryStream(input.ToArray(), writable: false);
using var compressed = new MemoryStream(input.ToArray(), false);
using var codecStream = streamFactory(compressed);
using var output = expectedLength > 0
? new MemoryStream(capacity: expectedLength)
? new MemoryStream(expectedLength)
: new MemoryStream();
var buffer = ArrayPool<byte>.Shared.Rent(8192);
byte[] buffer = ArrayPool<byte>.Shared.Rent(8192);
try
{
int totalWritten = 0;
var totalWritten = 0;
while (true)
{
var bytesRead = codecStream.Read(buffer, 0, buffer.Length);
int bytesRead = codecStream.Read(buffer, 0, buffer.Length);
if (bytesRead <= 0)
break;
totalWritten += bytesRead;
if (totalWritten > maxDecompressedSizeBytes)
throw new InvalidDataException($"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
throw new InvalidDataException(
$"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
output.Write(buffer, 0, bytesRead);
}
if (expectedLength >= 0 && totalWritten != expectedLength)
throw new InvalidDataException($"Expected decompressed length {expectedLength}, actual {totalWritten}.");
throw new InvalidDataException(
$"Expected decompressed length {expectedLength}, actual {totalWritten}.");
return output.ToArray();
}
@@ -253,4 +164,115 @@ public sealed class CompressionService
ArrayPool<byte>.Shared.Return(buffer);
}
}
private sealed class NoneCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.None;
/// <summary>
/// Returns a copy of the input payload without compression.
/// </summary>
/// <param name="input">The payload bytes to copy.</param>
/// <param name="level">The requested compression level.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return input.ToArray();
}
/// <summary>
/// Validates and returns an uncompressed payload copy.
/// </summary>
/// <param name="input">The payload bytes to validate and copy.</param>
/// <param name="expectedLength">The expected payload length, or a negative value to skip exact-length validation.</param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed payload size in bytes.</param>
/// <returns>The copied payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
if (input.Length > maxDecompressedSizeBytes)
throw new InvalidDataException(
$"Decompressed payload exceeds max allowed size ({maxDecompressedSizeBytes} bytes).");
if (expectedLength >= 0 && expectedLength != input.Length)
throw new InvalidDataException(
$"Expected decompressed length {expectedLength}, actual {input.Length}.");
return input.ToArray();
}
}
private sealed class BrotliCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Brotli;
/// <summary>
/// Compresses payload bytes using Brotli.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new BrotliStream(stream, level, true));
}
/// <summary>
/// Decompresses Brotli-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input,
stream => new BrotliStream(stream, CompressionMode.Decompress, true), expectedLength,
maxDecompressedSizeBytes);
}
}
private sealed class DeflateCompressionCodec : ICompressionCodec
{
/// <summary>
/// Gets the codec identifier.
/// </summary>
public CompressionCodec Codec => CompressionCodec.Deflate;
/// <summary>
/// Compresses payload bytes using Deflate.
/// </summary>
/// <param name="input">The payload bytes to compress.</param>
/// <param name="level">The compression level.</param>
/// <returns>The compressed payload bytes.</returns>
public byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level)
{
return CompressWithCodecStream(input, stream => new DeflateStream(stream, level, true));
}
/// <summary>
/// Decompresses Deflate-compressed payload bytes.
/// </summary>
/// <param name="input">The compressed payload bytes.</param>
/// <param name="expectedLength">
/// The expected decompressed byte length, or a negative value to skip exact-length
/// validation.
/// </param>
/// <param name="maxDecompressedSizeBytes">The maximum allowed decompressed byte length.</param>
/// <returns>The decompressed payload bytes.</returns>
public byte[] Decompress(ReadOnlySpan<byte> input, int expectedLength, int maxDecompressedSizeBytes)
{
return DecompressWithCodecStream(input,
stream => new DeflateStream(stream, CompressionMode.Decompress, true), expectedLength,
maxDecompressedSizeBytes);
}
}
}
src/CBDD.Core/Compression/CompressionStats.cs
+16 -9
View File
@@ -1,40 +1,47 @@
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Snapshot of aggregated compression and decompression telemetry.
/// Snapshot of aggregated compression and decompression telemetry.
/// </summary>
public readonly struct CompressionStats
{
/// <summary>
/// Gets or sets the CompressedDocumentCount.
/// Gets or sets the CompressedDocumentCount.
/// </summary>
public long CompressedDocumentCount { get; init; }
/// <summary>
/// Gets or sets the BytesBeforeCompression.
/// Gets or sets the BytesBeforeCompression.
/// </summary>
public long BytesBeforeCompression { get; init; }
/// <summary>
/// Gets or sets the BytesAfterCompression.
/// Gets or sets the BytesAfterCompression.
/// </summary>
public long BytesAfterCompression { get; init; }
/// <summary>
/// Gets or sets the CompressionCpuTicks.
/// Gets or sets the CompressionCpuTicks.
/// </summary>
public long CompressionCpuTicks { get; init; }
/// <summary>
/// Gets or sets the DecompressionCpuTicks.
/// Gets or sets the DecompressionCpuTicks.
/// </summary>
public long DecompressionCpuTicks { get; init; }
/// <summary>
/// Gets or sets the CompressionFailureCount.
/// Gets or sets the CompressionFailureCount.
/// </summary>
public long CompressionFailureCount { get; init; }
/// <summary>
/// Gets or sets the ChecksumFailureCount.
/// Gets or sets the ChecksumFailureCount.
/// </summary>
public long ChecksumFailureCount { get; init; }
/// <summary>
/// Gets or sets the SafetyLimitRejectionCount.
/// Gets or sets the SafetyLimitRejectionCount.
/// </summary>
public long SafetyLimitRejectionCount { get; init; }
}
src/CBDD.Core/Compression/CompressionTelemetry.cs
+76 -51
View File
@@ -1,111 +1,109 @@
using System.Threading;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Thread-safe counters for compression/decompression lifecycle events.
/// Thread-safe counters for compression/decompression lifecycle events.
/// </summary>
public sealed class CompressionTelemetry
{
private long _checksumFailureCount;
private long _compressedDocumentCount;
private long _compressionAttempts;
private long _compressionSuccesses;
private long _compressionCpuTicks;
private long _compressionFailures;
private long _compressionSkippedTooSmall;
private long _compressionSkippedInsufficientSavings;
private long _decompressionAttempts;
private long _decompressionSuccesses;
private long _decompressionFailures;
private long _compressionInputBytes;
private long _compressionOutputBytes;
private long _decompressionOutputBytes;
private long _compressedDocumentCount;
private long _compressionCpuTicks;
private long _compressionSkippedInsufficientSavings;
private long _compressionSkippedTooSmall;
private long _compressionSuccesses;
private long _decompressionAttempts;
private long _decompressionCpuTicks;
private long _checksumFailureCount;
private long _decompressionFailures;
private long _decompressionOutputBytes;
private long _decompressionSuccesses;
private long _safetyLimitRejectionCount;
/// <summary>
/// Gets the number of attempted compression operations.
/// Gets the number of attempted compression operations.
/// </summary>
public long CompressionAttempts => Interlocked.Read(ref _compressionAttempts);
/// <summary>
/// Gets the number of successful compression operations.
/// Gets the number of successful compression operations.
/// </summary>
public long CompressionSuccesses => Interlocked.Read(ref _compressionSuccesses);
/// <summary>
/// Gets the number of failed compression operations.
/// Gets the number of failed compression operations.
/// </summary>
public long CompressionFailures => Interlocked.Read(ref _compressionFailures);
/// <summary>
/// Gets the number of compression attempts skipped because payloads were too small.
/// Gets the number of compression attempts skipped because payloads were too small.
/// </summary>
public long CompressionSkippedTooSmall => Interlocked.Read(ref _compressionSkippedTooSmall);
/// <summary>
/// Gets the number of compression attempts skipped due to insufficient savings.
/// Gets the number of compression attempts skipped due to insufficient savings.
/// </summary>
public long CompressionSkippedInsufficientSavings => Interlocked.Read(ref _compressionSkippedInsufficientSavings);
/// <summary>
/// Gets the number of attempted decompression operations.
/// Gets the number of attempted decompression operations.
/// </summary>
public long DecompressionAttempts => Interlocked.Read(ref _decompressionAttempts);
/// <summary>
/// Gets the number of successful decompression operations.
/// Gets the number of successful decompression operations.
/// </summary>
public long DecompressionSuccesses => Interlocked.Read(ref _decompressionSuccesses);
/// <summary>
/// Gets the number of failed decompression operations.
/// Gets the number of failed decompression operations.
/// </summary>
public long DecompressionFailures => Interlocked.Read(ref _decompressionFailures);
/// <summary>
/// Gets the total input bytes observed by compression attempts.
/// Gets the total input bytes observed by compression attempts.
/// </summary>
public long CompressionInputBytes => Interlocked.Read(ref _compressionInputBytes);
/// <summary>
/// Gets the total output bytes produced by successful compression attempts.
/// Gets the total output bytes produced by successful compression attempts.
/// </summary>
public long CompressionOutputBytes => Interlocked.Read(ref _compressionOutputBytes);
/// <summary>
/// Gets the total output bytes produced by successful decompression attempts.
/// Gets the total output bytes produced by successful decompression attempts.
/// </summary>
public long DecompressionOutputBytes => Interlocked.Read(ref _decompressionOutputBytes);
/// <summary>
/// Gets the number of documents stored in compressed form.
/// Gets the number of documents stored in compressed form.
/// </summary>
public long CompressedDocumentCount => Interlocked.Read(ref _compressedDocumentCount);
/// <summary>
/// Gets the total CPU ticks spent on compression.
/// Gets the total CPU ticks spent on compression.
/// </summary>
public long CompressionCpuTicks => Interlocked.Read(ref _compressionCpuTicks);
/// <summary>
/// Gets the total CPU ticks spent on decompression.
/// Gets the total CPU ticks spent on decompression.
/// </summary>
public long DecompressionCpuTicks => Interlocked.Read(ref _decompressionCpuTicks);
/// <summary>
/// Gets the number of checksum validation failures.
/// Gets the number of checksum validation failures.
/// </summary>
public long ChecksumFailureCount => Interlocked.Read(ref _checksumFailureCount);
/// <summary>
/// Gets the number of decompression safety-limit rejections.
/// Gets the number of decompression safety-limit rejections.
/// </summary>
public long SafetyLimitRejectionCount => Interlocked.Read(ref _safetyLimitRejectionCount);
/// <summary>
/// Records a compression attempt and its input byte size.
/// Records a compression attempt and its input byte size.
/// </summary>
/// <param name="inputBytes">The number of input bytes provided to compression.</param>
public void RecordCompressionAttempt(int inputBytes)
@@ -115,7 +113,7 @@ public sealed class CompressionTelemetry
}
/// <summary>
/// Records a successful compression operation.
/// Records a successful compression operation.
/// </summary>
/// <param name="outputBytes">The number of compressed bytes produced.</param>
public void RecordCompressionSuccess(int outputBytes)
@@ -126,49 +124,73 @@ public sealed class CompressionTelemetry
}
/// <summary>
/// Records a failed compression operation.
/// Records a failed compression operation.
/// </summary>
public void RecordCompressionFailure() => Interlocked.Increment(ref _compressionFailures);
public void RecordCompressionFailure()
{
Interlocked.Increment(ref _compressionFailures);
}
/// <summary>
/// Records that compression was skipped because the payload was too small.
/// Records that compression was skipped because the payload was too small.
/// </summary>
public void RecordCompressionSkippedTooSmall() => Interlocked.Increment(ref _compressionSkippedTooSmall);
public void RecordCompressionSkippedTooSmall()
{
Interlocked.Increment(ref _compressionSkippedTooSmall);
}
/// <summary>
/// Records that compression was skipped due to insufficient expected savings.
/// Records that compression was skipped due to insufficient expected savings.
/// </summary>
public void RecordCompressionSkippedInsufficientSavings() => Interlocked.Increment(ref _compressionSkippedInsufficientSavings);
public void RecordCompressionSkippedInsufficientSavings()
{
Interlocked.Increment(ref _compressionSkippedInsufficientSavings);
}
/// <summary>
/// Records a decompression attempt.
/// Records a decompression attempt.
/// </summary>
public void RecordDecompressionAttempt() => Interlocked.Increment(ref _decompressionAttempts);
public void RecordDecompressionAttempt()
{
Interlocked.Increment(ref _decompressionAttempts);
}
/// <summary>
/// Adds CPU ticks spent performing compression.
/// Adds CPU ticks spent performing compression.
/// </summary>
/// <param name="ticks">The CPU ticks to add.</param>
public void RecordCompressionCpuTicks(long ticks) => Interlocked.Add(ref _compressionCpuTicks, ticks);
public void RecordCompressionCpuTicks(long ticks)
{
Interlocked.Add(ref _compressionCpuTicks, ticks);
}
/// <summary>
/// Adds CPU ticks spent performing decompression.
/// Adds CPU ticks spent performing decompression.
/// </summary>
/// <param name="ticks">The CPU ticks to add.</param>
public void RecordDecompressionCpuTicks(long ticks) => Interlocked.Add(ref _decompressionCpuTicks, ticks);
public void RecordDecompressionCpuTicks(long ticks)
{
Interlocked.Add(ref _decompressionCpuTicks, ticks);
}
/// <summary>
/// Records a checksum validation failure.
/// Records a checksum validation failure.
/// </summary>
public void RecordChecksumFailure() => Interlocked.Increment(ref _checksumFailureCount);
public void RecordChecksumFailure()
{
Interlocked.Increment(ref _checksumFailureCount);
}
/// <summary>
/// Records a decompression rejection due to safety limits.
/// Records a decompression rejection due to safety limits.
/// </summary>
public void RecordSafetyLimitRejection() => Interlocked.Increment(ref _safetyLimitRejectionCount);
public void RecordSafetyLimitRejection()
{
Interlocked.Increment(ref _safetyLimitRejectionCount);
}
/// <summary>
/// Records a successful decompression operation.
/// Records a successful decompression operation.
/// </summary>
/// <param name="outputBytes">The number of decompressed bytes produced.</param>
public void RecordDecompressionSuccess(int outputBytes)
@@ -178,12 +200,15 @@ public sealed class CompressionTelemetry
}
/// <summary>
/// Records a failed decompression operation.
/// Records a failed decompression operation.
/// </summary>
public void RecordDecompressionFailure() => Interlocked.Increment(ref _decompressionFailures);
public void RecordDecompressionFailure()
{
Interlocked.Increment(ref _decompressionFailures);
}
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry.
/// Returns a point-in-time snapshot of compression telemetry.
/// </summary>
/// <returns>The aggregated compression statistics.</returns>
public CompressionStats GetSnapshot()
src/CBDD.Core/Compression/ICompressionCodec.cs
+4 -4
View File
@@ -3,24 +3,24 @@ using System.IO.Compression;
namespace ZB.MOM.WW.CBDD.Core.Compression;
/// <summary>
/// Codec abstraction for payload compression and decompression.
/// Codec abstraction for payload compression and decompression.
/// </summary>
public interface ICompressionCodec
{
/// <summary>
/// Codec identifier.
/// Codec identifier.
/// </summary>
CompressionCodec Codec { get; }
/// <summary>
/// Compresses input bytes.
/// Compresses input bytes.
/// </summary>
/// <param name="input">Input payload bytes to compress.</param>
/// <param name="level">Compression level to apply.</param>
byte[] Compress(ReadOnlySpan<byte> input, CompressionLevel level);
/// <summary>
/// Decompresses payload bytes with output bounds validation.
/// Decompresses payload bytes with output bounds validation.
/// </summary>
/// <param name="input">Input payload bytes to decompress.</param>
/// <param name="expectedLength">Expected decompressed length.</param>
src/CBDD.Core/Context/DocumentDbContext.cs
+153 -161
View File
@@ -1,52 +1,39 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Compression;
using ZB.MOM.WW.CBDD.Core.Metadata;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Metadata;
using ZB.MOM.WW.CBDD.Core.Compression;
using System.Threading;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Core;
internal interface ICompactionAwareCollection
{
/// <summary>
/// Refreshes index bindings after compaction.
/// Refreshes index bindings after compaction.
/// </summary>
void RefreshIndexBindingsAfterCompaction();
}
/// <summary>
/// Base class for database contexts.
/// Inherit and add DocumentCollection{T} properties for your entities.
/// Use partial class for Source Generator integration.
/// Base class for database contexts.
/// Inherit and add DocumentCollection{T} properties for your entities.
/// Use partial class for Source Generator integration.
/// </summary>
public abstract partial class DocumentDbContext : IDisposable, ITransactionHolder
public abstract class DocumentDbContext : IDisposable, ITransactionHolder
{
internal readonly ChangeStreamDispatcher _cdc;
private readonly List<ICompactionAwareCollection> _compactionAwareCollections = new();
private readonly IReadOnlyDictionary<Type, object> _model;
private readonly List<IDocumentMapper> _registeredMappers = new();
private readonly IStorageEngine _storage;
internal readonly CDC.ChangeStreamDispatcher _cdc;
private readonly SemaphoreSlim _transactionLock = new(1, 1);
protected bool _disposed;
private readonly SemaphoreSlim _transactionLock = new SemaphoreSlim(1, 1);
/// <summary>
/// Gets the current active transaction, if any.
/// </summary>
public ITransaction? CurrentTransaction
{
get
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return field != null && (field.State == TransactionState.Active) ? field : null;
}
private set;
}
/// <summary>
/// Creates a new database context with default configuration
/// Creates a new database context with default configuration
/// </summary>
/// <param name="databasePath">The database file path.</param>
protected DocumentDbContext(string databasePath)
@@ -55,7 +42,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Creates a new database context with default storage configuration and custom compression settings.
/// Creates a new database context with default storage configuration and custom compression settings.
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="compressionOptions">Compression behavior options.</param>
@@ -65,7 +52,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Creates a new database context with custom configuration
/// Creates a new database context with custom configuration
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="config">The page file configuration.</param>
@@ -75,7 +62,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Creates a new database context with custom storage and compression configuration.
/// Creates a new database context with custom storage and compression configuration.
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="config">The page file configuration.</param>
@@ -91,7 +78,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
throw new ArgumentNullException(nameof(databasePath));
_storage = new StorageEngine(databasePath, config, compressionOptions, maintenanceOptions);
_cdc = new CDC.ChangeStreamDispatcher();
_cdc = new ChangeStreamDispatcher();
_storage.RegisterCdc(_cdc);
// Initialize model before collections
@@ -102,108 +89,41 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Initializes document collections for the context.
/// Gets the current active transaction, if any.
/// </summary>
protected virtual void InitializeCollections()
public ITransaction? CurrentTransaction
{
// Derived classes can override to initialize collections
get
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
return field != null && field.State == TransactionState.Active ? field : null;
}
private set;
}
private readonly IReadOnlyDictionary<Type, object> _model;
private readonly List<IDocumentMapper> _registeredMappers = new();
private readonly List<ICompactionAwareCollection> _compactionAwareCollections = new();
/// <summary>
/// Gets the concrete storage engine for advanced scenarios in derived contexts.
/// Gets the concrete storage engine for advanced scenarios in derived contexts.
/// </summary>
protected StorageEngine Engine => (StorageEngine)_storage;
/// <summary>
/// Gets compression options bound to this context's storage engine.
/// Gets compression options bound to this context's storage engine.
/// </summary>
protected CompressionOptions CompressionOptions => _storage.CompressionOptions;
/// <summary>
/// Gets the compression service for codec operations.
/// Gets the compression service for codec operations.
/// </summary>
protected CompressionService CompressionService => _storage.CompressionService;
/// <summary>
/// Gets compression telemetry counters.
/// Gets compression telemetry counters.
/// </summary>
protected CompressionTelemetry CompressionTelemetry => _storage.CompressionTelemetry;
/// <summary>
/// Override to configure the model using Fluent API.
/// </summary>
/// <param name="modelBuilder">The model builder instance.</param>
protected virtual void OnModelCreating(ModelBuilder modelBuilder)
{
}
/// <summary>
/// Helper to create a DocumentCollection instance with custom TId.
/// Used by derived classes in InitializeCollections for typed primary keys.
/// </summary>
/// <typeparam name="TId">The document identifier type.</typeparam>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="mapper">The mapper used for document serialization and key access.</param>
/// <returns>The created document collection.</returns>
protected DocumentCollection<TId, T> CreateCollection<TId, T>(IDocumentMapper<TId, T> mapper)
where T : class
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
string? customName = null;
EntityTypeBuilder<T>? builder = null;
if (_model.TryGetValue(typeof(T), out var builderObj))
{
builder = builderObj as EntityTypeBuilder<T>;
customName = builder?.CollectionName;
}
_registeredMappers.Add(mapper);
var collection = new DocumentCollection<TId, T>(_storage, this, mapper, customName);
if (collection is ICompactionAwareCollection compactionAwareCollection)
{
_compactionAwareCollections.Add(compactionAwareCollection);
}
// Apply configurations from ModelBuilder
if (builder != null)
{
foreach (var indexBuilder in builder.Indexes)
{
collection.ApplyIndexBuilder(indexBuilder);
}
}
_storage.RegisterMappers(_registeredMappers);
return collection;
}
/// <summary>
/// Gets the document collection for the specified entity type using an ObjectId as the key.
/// </summary>
/// <typeparam name="T">The type of entity to retrieve the document collection for. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;ObjectId, T&gt; instance for the specified entity type.</returns>
public DocumentCollection<ObjectId, T> Set<T>() where T : class => Set<ObjectId, T>();
/// <summary>
/// Gets a collection for managing documents of type T, identified by keys of type TId.
/// Override is generated automatically by the Source Generator for partial DbContext classes.
/// </summary>
/// <typeparam name="TId">The type of the unique identifier for documents in the collection.</typeparam>
/// <typeparam name="T">The type of the document to be managed. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;TId, T&gt; instance for performing operations on documents of type T.</returns>
public virtual DocumentCollection<TId, T> Set<TId, T>() where T : class
=> throw new InvalidOperationException($"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.");
/// <summary>
/// Releases resources used by the context.
/// Releases resources used by the context.
/// </summary>
public void Dispose()
{
@@ -220,7 +140,102 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Begins a transaction or returns the current active transaction.
/// Gets the current active transaction or starts a new one.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction GetCurrentTransactionOrStart()
{
return BeginTransaction();
}
/// <summary>
/// Gets the current active transaction or starts a new one asynchronously.
/// </summary>
/// <returns>The active transaction.</returns>
public async Task<ITransaction> GetCurrentTransactionOrStartAsync()
{
return await BeginTransactionAsync();
}
/// <summary>
/// Initializes document collections for the context.
/// </summary>
protected virtual void InitializeCollections()
{
// Derived classes can override to initialize collections
}
/// <summary>
/// Override to configure the model using Fluent API.
/// </summary>
/// <param name="modelBuilder">The model builder instance.</param>
protected virtual void OnModelCreating(ModelBuilder modelBuilder)
{
}
/// <summary>
/// Helper to create a DocumentCollection instance with custom TId.
/// Used by derived classes in InitializeCollections for typed primary keys.
/// </summary>
/// <typeparam name="TId">The document identifier type.</typeparam>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="mapper">The mapper used for document serialization and key access.</param>
/// <returns>The created document collection.</returns>
protected DocumentCollection<TId, T> CreateCollection<TId, T>(IDocumentMapper<TId, T> mapper)
where T : class
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
string? customName = null;
EntityTypeBuilder<T>? builder = null;
if (_model.TryGetValue(typeof(T), out object? builderObj))
{
builder = builderObj as EntityTypeBuilder<T>;
customName = builder?.CollectionName;
}
_registeredMappers.Add(mapper);
var collection = new DocumentCollection<TId, T>(_storage, this, mapper, customName);
if (collection is ICompactionAwareCollection compactionAwareCollection)
_compactionAwareCollections.Add(compactionAwareCollection);
// Apply configurations from ModelBuilder
if (builder != null)
foreach (var indexBuilder in builder.Indexes)
collection.ApplyIndexBuilder(indexBuilder);
_storage.RegisterMappers(_registeredMappers);
return collection;
}
/// <summary>
/// Gets the document collection for the specified entity type using an ObjectId as the key.
/// </summary>
/// <typeparam name="T">The type of entity to retrieve the document collection for. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;ObjectId, T&gt; instance for the specified entity type.</returns>
public DocumentCollection<ObjectId, T> Set<T>() where T : class
{
return Set<ObjectId, T>();
}
/// <summary>
/// Gets a collection for managing documents of type T, identified by keys of type TId.
/// Override is generated automatically by the Source Generator for partial DbContext classes.
/// </summary>
/// <typeparam name="TId">The type of the unique identifier for documents in the collection.</typeparam>
/// <typeparam name="T">The type of the document to be managed. Must be a reference type.</typeparam>
/// <returns>A DocumentCollection&lt;TId, T&gt; instance for performing operations on documents of type T.</returns>
public virtual DocumentCollection<TId, T> Set<TId, T>() where T : class
{
throw new InvalidOperationException(
$"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.");
}
/// <summary>
/// Begins a transaction or returns the current active transaction.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction BeginTransaction()
@@ -243,7 +258,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Begins a transaction asynchronously or returns the current active transaction.
/// Begins a transaction asynchronously or returns the current active transaction.
/// </summary>
/// <param name="ct">The cancellation token.</param>
/// <returns>The active transaction.</returns>
@@ -252,7 +267,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
bool lockAcquired = false;
var lockAcquired = false;
try
{
await _transactionLock.WaitAsync(ct);
@@ -271,32 +286,13 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets the current active transaction or starts a new one.
/// </summary>
/// <returns>The active transaction.</returns>
public ITransaction GetCurrentTransactionOrStart()
{
return BeginTransaction();
}
/// <summary>
/// Gets the current active transaction or starts a new one asynchronously.
/// </summary>
/// <returns>The active transaction.</returns>
public async Task<ITransaction> GetCurrentTransactionOrStartAsync()
{
return await BeginTransactionAsync();
}
/// <summary>
/// Commits the current transaction if one is active.
/// Commits the current transaction if one is active.
/// </summary>
public void SaveChanges()
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
if (CurrentTransaction != null)
{
try
{
CurrentTransaction.Commit();
@@ -305,11 +301,10 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
{
CurrentTransaction = null;
}
}
}
/// <summary>
/// Commits the current transaction asynchronously if one is active.
/// Commits the current transaction asynchronously if one is active.
/// </summary>
/// <param name="ct">The cancellation token.</param>
public async Task SaveChangesAsync(CancellationToken ct = default)
@@ -317,7 +312,6 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
if (CurrentTransaction != null)
{
try
{
await CurrentTransaction.CommitAsync(ct);
@@ -326,11 +320,10 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
{
CurrentTransaction = null;
}
}
}
/// <summary>
/// Executes a checkpoint using the requested mode.
/// Executes a checkpoint using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <returns>The checkpoint execution result.</returns>
@@ -343,12 +336,13 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Executes a checkpoint asynchronously using the requested mode.
/// Executes a checkpoint asynchronously using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The checkpoint execution result.</returns>
public Task<CheckpointResult> CheckpointAsync(CheckpointMode mode = CheckpointMode.Truncate, CancellationToken ct = default)
public Task<CheckpointResult> CheckpointAsync(CheckpointMode mode = CheckpointMode.Truncate,
CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
@@ -357,7 +351,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats()
{
@@ -368,7 +362,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// </summary>
/// <param name="options">Compaction execution options.</param>
public CompactionStats Compact(CompactionOptions? options = null)
@@ -382,7 +376,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// Runs offline compaction by default. Set options to online mode for a bounded online pass.
/// </summary>
/// <param name="options">Compaction execution options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
@@ -395,7 +389,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Alias for <see cref="Compact(CompactionOptions?)"/>.
/// Alias for <see cref="Compact(CompactionOptions?)" />.
/// </summary>
/// <param name="options">Compaction execution options.</param>
public CompactionStats Vacuum(CompactionOptions? options = null)
@@ -409,7 +403,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Async alias for <see cref="CompactAsync(CompactionOptions?, CancellationToken)"/>.
/// Async alias for <see cref="CompactAsync(CompactionOptions?, CancellationToken)" />.
/// </summary>
/// <param name="options">Compaction execution options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
@@ -437,14 +431,11 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
private void RefreshCollectionBindingsAfterCompaction()
{
foreach (var collection in _compactionAwareCollections)
{
collection.RefreshIndexBindingsAfterCompaction();
}
foreach (var collection in _compactionAwareCollections) collection.RefreshIndexBindingsAfterCompaction();
}
/// <summary>
/// Gets page usage grouped by page type.
/// Gets page usage grouped by page type.
/// </summary>
public IReadOnlyList<PageTypeUsageEntry> GetPageUsageByPageType()
{
@@ -455,7 +446,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets per-collection page usage diagnostics.
/// Gets per-collection page usage diagnostics.
/// </summary>
public IReadOnlyList<CollectionPageUsageEntry> GetPageUsageByCollection()
{
@@ -466,7 +457,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets per-collection compression ratio diagnostics.
/// Gets per-collection compression ratio diagnostics.
/// </summary>
public IReadOnlyList<CollectionCompressionRatioEntry> GetCompressionRatioByCollection()
{
@@ -477,7 +468,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets free-list summary diagnostics.
/// Gets free-list summary diagnostics.
/// </summary>
public FreeListSummary GetFreeListSummary()
{
@@ -488,7 +479,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Gets page-level fragmentation diagnostics.
/// Gets page-level fragmentation diagnostics.
/// </summary>
public FragmentationMapReport GetFragmentationMap()
{
@@ -499,7 +490,7 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs compression migration as dry-run estimation by default.
/// Runs compression migration as dry-run estimation by default.
/// </summary>
/// <param name="options">Compression migration options.</param>
public CompressionMigrationResult MigrateCompression(CompressionMigrationOptions? options = null)
@@ -511,11 +502,12 @@ public abstract partial class DocumentDbContext : IDisposable, ITransactionHolde
}
/// <summary>
/// Runs compression migration asynchronously as dry-run estimation by default.
/// Runs compression migration asynchronously as dry-run estimation by default.
/// </summary>
/// <param name="options">Compression migration options.</param>
/// <param name="ct">Cancellation token for the asynchronous operation.</param>
public Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null, CancellationToken ct = default)
public Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null,
CancellationToken ct = default)
{
if (_disposed)
throw new ObjectDisposedException(nameof(DocumentDbContext));
src/CBDD.Core/Indexing/BTreeCursor.cs
+69 -85
View File
@@ -1,27 +1,24 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Bson;
using System.Collections.Generic;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
internal sealed class BTreeCursor : IBTreeCursor
{
private readonly List<IndexEntry> _currentEntries;
private readonly BTreeIndex _index;
private readonly ulong _transactionId;
private readonly IIndexStorage _storage;
private readonly ulong _transactionId;
private int _currentEntryIndex;
private BTreeNodeHeader _currentHeader;
private uint _currentPageId;
private bool _isValid;
// State
private byte[] _pageBuffer;
private uint _currentPageId;
private int _currentEntryIndex;
private BTreeNodeHeader _currentHeader;
private List<IndexEntry> _currentEntries;
private bool _isValid;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeCursor"/> class.
/// Initializes a new instance of the <see cref="BTreeCursor" /> class.
/// </summary>
/// <param name="index">The index to traverse.</param>
/// <param name="storage">The storage engine for page access.</param>
@@ -37,7 +34,7 @@ internal sealed class BTreeCursor : IBTreeCursor
}
/// <summary>
/// Gets the current index entry at the cursor position.
/// Gets the current index entry at the cursor position.
/// </summary>
public IndexEntry Current
{
@@ -49,13 +46,13 @@ internal sealed class BTreeCursor : IBTreeCursor
}
/// <summary>
/// Moves the cursor to the first entry in the index.
/// Moves the cursor to the first entry in the index.
/// </summary>
/// <returns><see langword="true"/> if an entry is available; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if an entry is available; otherwise, <see langword="false" />.</returns>
public bool MoveToFirst()
{
// Find left-most leaf
var pageId = _index.RootPageId;
uint pageId = _index.RootPageId;
while (true)
{
LoadPage(pageId);
@@ -63,7 +60,7 @@ internal sealed class BTreeCursor : IBTreeCursor
// Go to first child (P0)
// Internal node format: [Header] [P0] [Entry1] ...
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
pageId = BitConverter.ToUInt32(_pageBuffer.AsSpan(dataOffset, 4));
}
@@ -71,13 +68,13 @@ internal sealed class BTreeCursor : IBTreeCursor
}
/// <summary>
/// Moves the cursor to the last entry in the index.
/// Moves the cursor to the last entry in the index.
/// </summary>
/// <returns><see langword="true"/> if an entry is available; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if an entry is available; otherwise, <see langword="false" />.</returns>
public bool MoveToLast()
{
// Find right-most leaf
var pageId = _index.RootPageId;
uint pageId = _index.RootPageId;
while (true)
{
LoadPage(pageId);
@@ -93,16 +90,17 @@ internal sealed class BTreeCursor : IBTreeCursor
// We want the last pointer.
// Re-read P0 just in case
uint lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(32 + 20, 4));
var lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(32 + 20, 4));
var offset = 32 + 20 + 4;
for (int i = 0; i < _currentHeader.EntryCount; i++)
int offset = 32 + 20 + 4;
for (var i = 0; i < _currentHeader.EntryCount; i++)
{
var keyLen = BitConverter.ToInt32(_pageBuffer.AsSpan(offset, 4));
offset += 4 + keyLen;
lastPointer = BitConverter.ToUInt32(_pageBuffer.AsSpan(offset, 4));
offset += 4;
}
pageId = lastPointer;
}
@@ -110,21 +108,21 @@ internal sealed class BTreeCursor : IBTreeCursor
}
/// <summary>
/// Seeks to the specified key or the next greater key.
/// Seeks to the specified key or the next greater key.
/// </summary>
/// <param name="key">The key to seek.</param>
/// <returns>
/// <see langword="true"/> if an exact key match is found; otherwise, <see langword="false"/>.
/// <see langword="true" /> if an exact key match is found; otherwise, <see langword="false" />.
/// </returns>
public bool Seek(IndexKey key)
{
// Use Index to find leaf
var leafPageId = _index.FindLeafNode(key, _transactionId);
uint leafPageId = _index.FindLeafNode(key, _transactionId);
LoadPage(leafPageId);
ParseEntries();
// Binary search in entries
var idx = _currentEntries.BinarySearch(new IndexEntry(key, default(DocumentLocation)));
int idx = _currentEntries.BinarySearch(new IndexEntry(key, default(DocumentLocation)));
if (idx >= 0)
{
@@ -133,51 +131,44 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
// Not found, ~idx is the next larger value
_currentEntryIndex = ~idx;
if (_currentEntryIndex < _currentEntries.Count)
// Not found, ~idx is the next larger value
_currentEntryIndex = ~idx;
if (_currentEntryIndex < _currentEntries.Count)
{
_isValid = true;
return false; // Positioned at next greater
}
// Key is larger than max in this page, move to next page
if (_currentHeader.NextLeafPageId != 0)
{
LoadPage(_currentHeader.NextLeafPageId);
ParseEntries();
_currentEntryIndex = 0;
if (_currentEntries.Count > 0)
{
_isValid = true;
return false; // Positioned at next greater
}
else
{
// Key is larger than max in this page, move to next page
if (_currentHeader.NextLeafPageId != 0)
{
LoadPage(_currentHeader.NextLeafPageId);
ParseEntries();
_currentEntryIndex = 0;
if (_currentEntries.Count > 0)
{
_isValid = true;
return false;
}
}
// End of index
_isValid = false;
return false;
}
}
// End of index
_isValid = false;
return false;
}
/// <summary>
/// Moves the cursor to the next entry.
/// Moves the cursor to the next entry.
/// </summary>
/// <returns><see langword="true"/> if the cursor moved to a valid entry; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the cursor moved to a valid entry; otherwise, <see langword="false" />.</returns>
public bool MoveNext()
{
if (!_isValid) return false;
_currentEntryIndex++;
if (_currentEntryIndex < _currentEntries.Count)
{
return true;
}
if (_currentEntryIndex < _currentEntries.Count) return true;
// Move to next page
if (_currentHeader.NextLeafPageId != 0)
@@ -191,18 +182,15 @@ internal sealed class BTreeCursor : IBTreeCursor
}
/// <summary>
/// Moves the cursor to the previous entry.
/// Moves the cursor to the previous entry.
/// </summary>
/// <returns><see langword="true"/> if the cursor moved to a valid entry; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the cursor moved to a valid entry; otherwise, <see langword="false" />.</returns>
public bool MovePrev()
{
if (!_isValid) return false;
_currentEntryIndex--;
if (_currentEntryIndex >= 0)
{
return true;
}
if (_currentEntryIndex >= 0) return true;
// Move to prev page
if (_currentHeader.PrevLeafPageId != 0)
@@ -215,6 +203,18 @@ internal sealed class BTreeCursor : IBTreeCursor
return false;
}
/// <summary>
/// Releases cursor resources.
/// </summary>
public void Dispose()
{
if (_pageBuffer != null)
{
ArrayPool<byte>.Shared.Return(_pageBuffer);
_pageBuffer = null!;
}
}
private void LoadPage(uint pageId)
{
if (_currentPageId == pageId && _pageBuffer != null) return;
@@ -229,9 +229,9 @@ internal sealed class BTreeCursor : IBTreeCursor
// Helper to parse entries from current page buffer
// (Similar to BTreeIndex.ReadLeafEntries)
_currentEntries.Clear();
var dataOffset = 32 + 20;
int dataOffset = 32 + 20;
for (int i = 0; i < _currentHeader.EntryCount; i++)
for (var i = 0; i < _currentHeader.EntryCount; i++)
{
// Read Key
var keyLen = BitConverter.ToInt32(_pageBuffer.AsSpan(dataOffset, 4));
@@ -257,12 +257,10 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
// Empty page? Should not happen in helper logic unless root leaf is empty
_isValid = false;
return false;
}
// Empty page? Should not happen in helper logic unless root leaf is empty
_isValid = false;
return false;
}
private bool PositionAtEnd()
@@ -274,22 +272,8 @@ internal sealed class BTreeCursor : IBTreeCursor
_isValid = true;
return true;
}
else
{
_isValid = false;
return false;
}
}
/// <summary>
/// Releases cursor resources.
/// </summary>
public void Dispose()
{
if (_pageBuffer != null)
{
ArrayPool<byte>.Shared.Return(_pageBuffer);
_pageBuffer = null!;
}
_isValid = false;
return false;
}
}
src/CBDD.Core/Indexing/BTreeIndex.cs
+209 -230
View File
File diff suppressed because it is too large Load Diff
src/CBDD.Core/Indexing/BTreeStructures.cs
+28 -32
View File
@@ -1,27 +1,26 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents an entry in an index mapping a key to a document location.
/// Implemented as struct for memory efficiency.
/// Represents an entry in an index mapping a key to a document location.
/// Implemented as struct for memory efficiency.
/// </summary>
public struct IndexEntry : IComparable<IndexEntry>, IComparable
{
/// <summary>
/// Gets or sets the index key.
/// Gets or sets the index key.
/// </summary>
public IndexKey Key { get; set; }
/// <summary>
/// Gets or sets the document location for the key.
/// Gets or sets the document location for the key.
/// </summary>
public DocumentLocation Location { get; set; }
/// <summary>
/// Initializes a new instance of the <see cref="IndexEntry"/> struct.
/// Initializes a new instance of the <see cref="IndexEntry" /> struct.
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
@@ -34,7 +33,7 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
// Backward compatibility: constructor that takes ObjectId (for migration)
// Will be removed once all code is migrated
/// <summary>
/// Initializes a legacy instance of the <see cref="IndexEntry"/> struct for migration scenarios.
/// Initializes a legacy instance of the <see cref="IndexEntry" /> struct for migration scenarios.
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="documentId">The legacy document identifier.</param>
@@ -47,12 +46,12 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
}
/// <summary>
/// Compares this entry to another entry by key.
/// Compares this entry to another entry by key.
/// </summary>
/// <param name="other">The other index entry to compare.</param>
/// <returns>
/// A value less than zero if this instance is less than <paramref name="other"/>,
/// zero if they are equal, or greater than zero if this instance is greater.
/// A value less than zero if this instance is less than <paramref name="other" />,
/// zero if they are equal, or greater than zero if this instance is greater.
/// </returns>
public int CompareTo(IndexEntry other)
{
@@ -60,14 +59,14 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
}
/// <summary>
/// Compares this entry to another object.
/// Compares this entry to another object.
/// </summary>
/// <param name="obj">The object to compare.</param>
/// <returns>
/// A value less than zero if this instance is less than <paramref name="obj"/>,
/// zero if they are equal, or greater than zero if this instance is greater.
/// A value less than zero if this instance is less than <paramref name="obj" />,
/// zero if they are equal, or greater than zero if this instance is greater.
/// </returns>
/// <exception cref="ArgumentException">Thrown when <paramref name="obj"/> is not an <see cref="IndexEntry"/>.</exception>
/// <exception cref="ArgumentException">Thrown when <paramref name="obj" /> is not an <see cref="IndexEntry" />.</exception>
public int CompareTo(object? obj)
{
if (obj is IndexEntry other) return CompareTo(other);
@@ -76,43 +75,43 @@ public struct IndexEntry : IComparable<IndexEntry>, IComparable
}
/// <summary>
/// B+Tree node for index storage.
/// Uses struct for node metadata to minimize allocations.
/// B+Tree node for index storage.
/// Uses struct for node metadata to minimize allocations.
/// </summary>
public struct BTreeNodeHeader
{
/// <summary>
/// Gets or sets the page identifier.
/// Gets or sets the page identifier.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether this node is a leaf node.
/// Gets or sets a value indicating whether this node is a leaf node.
/// </summary>
public bool IsLeaf { get; set; }
/// <summary>
/// Gets or sets the number of entries in the node.
/// Gets or sets the number of entries in the node.
/// </summary>
public ushort EntryCount { get; set; }
/// <summary>
/// Gets or sets the parent page identifier.
/// Gets or sets the parent page identifier.
/// </summary>
public uint ParentPageId { get; set; }
/// <summary>
/// Gets or sets the next leaf page identifier.
/// Gets or sets the next leaf page identifier.
/// </summary>
public uint NextLeafPageId { get; set; } // For leaf nodes only
public uint NextLeafPageId { get; set; } // For leaf nodes only
/// <summary>
/// Gets or sets the previous leaf page identifier.
/// Gets or sets the previous leaf page identifier.
/// </summary>
public uint PrevLeafPageId { get; set; } // For leaf nodes only (added for reverse scan)
public uint PrevLeafPageId { get; set; } // For leaf nodes only (added for reverse scan)
/// <summary>
/// Writes the header to a byte span.
/// Writes the header to a byte span.
/// </summary>
/// <param name="destination">The destination span.</param>
public void WriteTo(Span<byte> destination)
@@ -120,7 +119,7 @@ public struct BTreeNodeHeader
if (destination.Length < 20)
throw new ArgumentException("Destination must be at least 20 bytes");
BitConverter.TryWriteBytes(destination[0..4], PageId);
BitConverter.TryWriteBytes(destination[..4], PageId);
destination[4] = (byte)(IsLeaf ? 1 : 0);
BitConverter.TryWriteBytes(destination[5..7], EntryCount);
BitConverter.TryWriteBytes(destination[7..11], ParentPageId);
@@ -129,7 +128,7 @@ public struct BTreeNodeHeader
}
/// <summary>
/// Reads a node header from a byte span.
/// Reads a node header from a byte span.
/// </summary>
/// <param name="source">The source span.</param>
/// <returns>The parsed node header.</returns>
@@ -140,17 +139,14 @@ public struct BTreeNodeHeader
var header = new BTreeNodeHeader
{
PageId = BitConverter.ToUInt32(source[0..4]),
PageId = BitConverter.ToUInt32(source[..4]),
IsLeaf = source[4] != 0,
EntryCount = BitConverter.ToUInt16(source[5..7]),
ParentPageId = BitConverter.ToUInt32(source[7..11]),
NextLeafPageId = BitConverter.ToUInt32(source[11..15])
};
if (source.Length >= 20)
{
header.PrevLeafPageId = BitConverter.ToUInt32(source[15..19]);
}
if (source.Length >= 20) header.PrevLeafPageId = BitConverter.ToUInt32(source[15..19]);
return header;
}
src/CBDD.Core/Indexing/CollectionIndexDefinition.cs
+62 -65
View File
@@ -1,61 +1,16 @@
using System;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Bson;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// High-level metadata and configuration for a custom index on a document collection.
/// Wraps low-level IndexOptions and provides strongly-typed expression-based key extraction.
/// High-level metadata and configuration for a custom index on a document collection.
/// Wraps low-level IndexOptions and provides strongly-typed expression-based key extraction.
/// </summary>
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionIndexDefinition<T> where T : class
{
/// <summary>
/// Unique name for this index (auto-generated or user-specified)
/// </summary>
public string Name { get; }
/// <summary>
/// Property paths that make up this index key.
/// Examples: ["Age"] for simple index, ["City", "Age"] for compound index
/// </summary>
public string[] PropertyPaths { get; }
/// <summary>
/// If true, enforces uniqueness constraint on the indexed values
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Type of index structure (from existing IndexType enum)
/// </summary>
public IndexType Type { get; }
/// <summary>Vector dimensions (only for Vector index)</summary>
public int Dimensions { get; }
/// <summary>Distance metric (only for Vector index)</summary>
public VectorMetric Metric { get; }
/// <summary>
/// Compiled function to extract the index key from a document.
/// Compiled for maximum performance (10-100x faster than interpreting Expression).
/// </summary>
public Func<T, object> KeySelector { get; }
/// <summary>
/// Original expression for the key selector (for analysis and serialization)
/// </summary>
public Expression<Func<T, object>> KeySelectorExpression { get; }
/// <summary>
/// If true, this is the primary key index (_id)
/// </summary>
public bool IsPrimary { get; }
/// <summary>
/// Creates a new index definition
/// Creates a new index definition
/// </summary>
/// <param name="name">Index name</param>
/// <param name="propertyPaths">Property paths for the index</param>
@@ -93,7 +48,50 @@ public sealed class CollectionIndexDefinition<T> where T : class
}
/// <summary>
/// Converts this high-level definition to low-level IndexOptions for BTreeIndex
/// Unique name for this index (auto-generated or user-specified)
/// </summary>
public string Name { get; }
/// <summary>
/// Property paths that make up this index key.
/// Examples: ["Age"] for simple index, ["City", "Age"] for compound index
/// </summary>
public string[] PropertyPaths { get; }
/// <summary>
/// If true, enforces uniqueness constraint on the indexed values
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Type of index structure (from existing IndexType enum)
/// </summary>
public IndexType Type { get; }
/// <summary>Vector dimensions (only for Vector index)</summary>
public int Dimensions { get; }
/// <summary>Distance metric (only for Vector index)</summary>
public VectorMetric Metric { get; }
/// <summary>
/// Compiled function to extract the index key from a document.
/// Compiled for maximum performance (10-100x faster than interpreting Expression).
/// </summary>
public Func<T, object> KeySelector { get; }
/// <summary>
/// Original expression for the key selector (for analysis and serialization)
/// </summary>
public Expression<Func<T, object>> KeySelectorExpression { get; }
/// <summary>
/// If true, this is the primary key index (_id)
/// </summary>
public bool IsPrimary { get; }
/// <summary>
/// Converts this high-level definition to low-level IndexOptions for BTreeIndex
/// </summary>
public IndexOptions ToIndexOptions()
{
@@ -108,7 +106,7 @@ public sealed class CollectionIndexDefinition<T> where T : class
}
/// <summary>
/// Checks if this index can be used for a query on the specified property path
/// Checks if this index can be used for a query on the specified property path
/// </summary>
/// <param name="propertyPath">The property path to validate.</param>
public bool CanSupportQuery(string propertyPath)
@@ -123,7 +121,7 @@ public sealed class CollectionIndexDefinition<T> where T : class
}
/// <summary>
/// Checks if this index can support queries on multiple properties (compound queries)
/// Checks if this index can support queries on multiple properties (compound queries)
/// </summary>
/// <param name="propertyPaths">The ordered property paths to validate.</param>
public bool CanSupportCompoundQuery(string[] propertyPaths)
@@ -136,11 +134,9 @@ public sealed class CollectionIndexDefinition<T> where T : class
if (propertyPaths.Length > PropertyPaths.Length)
return false;
for (int i = 0; i < propertyPaths.Length; i++)
{
for (var i = 0; i < propertyPaths.Length; i++)
if (!PropertyPaths[i].Equals(propertyPaths[i], StringComparison.OrdinalIgnoreCase))
return false;
}
return true;
}
@@ -148,55 +144,56 @@ public sealed class CollectionIndexDefinition<T> where T : class
/// <inheritdoc />
public override string ToString()
{
var uniqueStr = IsUnique ? "Unique" : "Non-Unique";
var paths = string.Join(", ", PropertyPaths);
string uniqueStr = IsUnique ? "Unique" : "Non-Unique";
string paths = string.Join(", ", PropertyPaths);
return $"{Name} ({uniqueStr} {Type} on [{paths}])";
}
}
/// <summary>
/// Information about an existing index (for querying index metadata)
/// Information about an existing index (for querying index metadata)
/// </summary>
public sealed class CollectionIndexInfo
{
/// <summary>
/// Gets the index name.
/// Gets the index name.
/// </summary>
public string Name { get; init; } = string.Empty;
/// <summary>
/// Gets the indexed property paths.
/// Gets the indexed property paths.
/// </summary>
public string[] PropertyPaths { get; init; } = Array.Empty<string>();
/// <summary>
/// Gets a value indicating whether the index is unique.
/// Gets a value indicating whether the index is unique.
/// </summary>
public bool IsUnique { get; init; }
/// <summary>
/// Gets the index type.
/// Gets the index type.
/// </summary>
public IndexType Type { get; init; }
/// <summary>
/// Gets a value indicating whether this index is the primary index.
/// Gets a value indicating whether this index is the primary index.
/// </summary>
public bool IsPrimary { get; init; }
/// <summary>
/// Gets the estimated number of indexed documents.
/// Gets the estimated number of indexed documents.
/// </summary>
public long EstimatedDocumentCount { get; init; }
/// <summary>
/// Gets the estimated storage size, in bytes.
/// Gets the estimated storage size, in bytes.
/// </summary>
public long EstimatedSizeBytes { get; init; }
/// <inheritdoc />
public override string ToString()
{
return $"{Name}: {string.Join(", ", PropertyPaths)} ({EstimatedDocumentCount} docs, {EstimatedSizeBytes:N0} bytes)";
return
$"{Name}: {string.Join(", ", PropertyPaths)} ({EstimatedDocumentCount} docs, {EstimatedSizeBytes:N0} bytes)";
}
}
src/CBDD.Core/Indexing/CollectionIndexManager.cs
+122 -131
View File
@@ -1,7 +1,4 @@
using System;
using System.Collections.Generic;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
@@ -9,23 +6,23 @@ using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Manages a collection of secondary indexes on a document collection.
/// Handles index creation, deletion, automatic selection, and maintenance.
/// Manages a collection of secondary indexes on a document collection.
/// Handles index creation, deletion, automatic selection, and maintenance.
/// </summary>
/// <typeparam name="TId">Primary key type</typeparam>
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
{
private readonly Dictionary<string, CollectionSecondaryIndex<TId, T>> _indexes;
private readonly IStorageEngine _storage;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly object _lock = new();
private bool _disposed;
private readonly string _collectionName;
private readonly Dictionary<string, CollectionSecondaryIndex<TId, T>> _indexes;
private readonly object _lock = new();
private readonly IDocumentMapper<TId, T> _mapper;
private readonly IStorageEngine _storage;
private bool _disposed;
private CollectionMetadata _metadata;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}"/> class.
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}" /> class.
/// </summary>
/// <param name="storage">The storage engine used to persist index data and metadata.</param>
/// <param name="mapper">The document mapper for the collection type.</param>
@@ -36,12 +33,13 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}"/> class from the storage abstraction.
/// Initializes a new instance of the <see cref="CollectionIndexManager{TId, T}" /> class from the storage abstraction.
/// </summary>
/// <param name="storage">The storage abstraction used to persist index state.</param>
/// <param name="mapper">The document mapper for the collection.</param>
/// <param name="collectionName">An optional collection name override.</param>
internal CollectionIndexManager(IStorageEngine storage, IDocumentMapper<TId, T> mapper, string? collectionName = null)
internal CollectionIndexManager(IStorageEngine storage, IDocumentMapper<TId, T> mapper,
string? collectionName = null)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_mapper = mapper ?? throw new ArgumentNullException(nameof(mapper));
@@ -49,17 +47,53 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
_indexes = new Dictionary<string, CollectionSecondaryIndex<TId, T>>(StringComparer.OrdinalIgnoreCase);
// Load existing metadata via storage
_metadata = _storage.GetCollectionMetadata(_collectionName) ?? new CollectionMetadata { Name = _collectionName };
_metadata = _storage.GetCollectionMetadata(_collectionName) ??
new CollectionMetadata { Name = _collectionName };
// Initialize indexes from metadata
foreach (var idxMeta in _metadata.Indexes)
{
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type, idxMeta.Dimensions, idxMeta.Metric);
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type,
idxMeta.Dimensions, idxMeta.Metric);
var index = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper, idxMeta.RootPageId);
_indexes[idxMeta.Name] = index;
}
}
/// <summary>
/// Gets the root page identifier for the primary index.
/// </summary>
public uint PrimaryRootPageId => _metadata.PrimaryRootPageId;
/// <summary>
/// Releases resources used by the index manager.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// No auto-save on dispose to avoid unnecessary I/O if no changes
lock (_lock)
{
foreach (var index in _indexes.Values)
try
{
index.Dispose();
}
catch
{
/* Best effort */
}
_indexes.Clear();
_disposed = true;
}
GC.SuppressFinalize(this);
}
private void UpdateMetadata()
{
_metadata.Indexes.Clear();
@@ -80,7 +114,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Creates a new secondary index
/// Creates a new secondary index
/// </summary>
/// <param name="definition">Index definition</param>
/// <returns>The created secondary index</returns>
@@ -113,7 +147,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
// ... methods ...
/// <summary>
/// Creates a simple index on a single property
/// Creates a simple index on a single property
/// </summary>
/// <typeparam name="TKey">Key type</typeparam>
/// <param name="keySelector">Expression to extract key from document</param>
@@ -129,7 +163,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
throw new ArgumentNullException(nameof(keySelector));
// Extract property paths from expression
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
// Generate name if not provided
name ??= GenerateIndexName(propertyPaths);
@@ -150,7 +184,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Creates a vector index for a collection property.
/// Creates a vector index for a collection property.
/// </summary>
/// <typeparam name="TKey">The selected key type.</typeparam>
/// <param name="keySelector">Expression to extract the indexed field.</param>
@@ -158,10 +192,11 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
/// <param name="metric">Distance metric used by the vector index.</param>
/// <param name="name">Optional index name.</param>
/// <returns>The created or existing index.</returns>
public CollectionSecondaryIndex<TId, T> CreateVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
public CollectionSecondaryIndex<TId, T> CreateVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector,
int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
@@ -169,21 +204,19 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
// Reuse the original parameter from keySelector to avoid invalid expression trees.
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector, false, dimensions, metric);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector,
false, dimensions, metric);
return CreateIndex(definition);
}
}
/// <summary>
/// Ensures that an index exists for the specified key selector.
/// Ensures that an index exists for the specified key selector.
/// </summary>
/// <param name="keySelector">Expression to extract the indexed field.</param>
/// <param name="name">Optional index name.</param>
@@ -194,7 +227,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
string? name = null,
bool unique = false)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
name ??= GenerateIndexName(propertyPaths);
lock (_lock)
@@ -207,7 +240,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Ensures that an index exists for the specified untyped key selector.
/// Ensures that an index exists for the specified untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="name">Optional index name.</param>
@@ -220,10 +253,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
{
// Convert LambdaExpression to Expression<Func<T, object>> properly by sharing parameters
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
@@ -231,7 +261,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Creates a vector index from an untyped key selector.
/// Creates a vector index from an untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="dimensions">Vector dimensionality.</param>
@@ -244,8 +274,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
VectorMetric metric = VectorMetric.Cosine,
string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
@@ -253,20 +283,18 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector, false, dimensions, metric);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Vector,
false, dimensions, metric);
return CreateIndex(definition);
}
}
/// <summary>
/// Creates a spatial index from an untyped key selector.
/// Creates a spatial index from an untyped key selector.
/// </summary>
/// <param name="keySelector">Untyped expression that selects indexed fields.</param>
/// <param name="name">Optional index name.</param>
@@ -275,8 +303,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
LambdaExpression keySelector,
string? name = null)
{
var propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
var indexName = name ?? GenerateIndexName(propertyPaths);
string[] propertyPaths = ExpressionAnalyzer.ExtractPropertyPaths(keySelector);
string indexName = name ?? GenerateIndexName(propertyPaths);
lock (_lock)
{
@@ -284,20 +312,18 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
return existing;
var body = keySelector.Body;
if (body.Type != typeof(object))
{
body = Expression.Convert(body, typeof(object));
}
if (body.Type != typeof(object)) body = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(body, keySelector.Parameters);
var definition = new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Spatial);
var definition =
new CollectionIndexDefinition<T>(indexName, propertyPaths, lambda, false, IndexType.Spatial);
return CreateIndex(definition);
}
}
/// <summary>
/// Drops an existing index by name
/// Drops an existing index by name
/// </summary>
/// <param name="name">Index name</param>
/// <returns>True if index was found and dropped, false otherwise</returns>
@@ -324,7 +350,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Gets an index by name
/// Gets an index by name
/// </summary>
/// <param name="name">The index name.</param>
public CollectionSecondaryIndex<TId, T>? GetIndex(string name)
@@ -336,7 +362,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Gets all indexes
/// Gets all indexes
/// </summary>
public IEnumerable<CollectionSecondaryIndex<TId, T>> GetAllIndexes()
{
@@ -347,7 +373,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Gets information about all indexes
/// Gets information about all indexes
/// </summary>
public IEnumerable<CollectionIndexInfo> GetIndexInfo()
{
@@ -358,8 +384,8 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Finds the best index to use for a query on the specified property.
/// Returns null if no suitable index found (requires full scan).
/// Finds the best index to use for a query on the specified property.
/// Returns null if no suitable index found (requires full scan).
/// </summary>
/// <param name="propertyPath">Property path being queried</param>
/// <returns>Best index for the query, or null if none suitable</returns>
@@ -387,7 +413,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Finds the best index for a compound query on multiple properties
/// Finds the best index for a compound query on multiple properties
/// </summary>
/// <param name="propertyPaths">The ordered list of queried property paths.</param>
public CollectionSecondaryIndex<TId, T>? FindBestCompoundIndex(string[] propertyPaths)
@@ -413,7 +439,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Inserts a document into all indexes
/// Inserts a document into all indexes
/// </summary>
/// <param name="document">Document to insert</param>
/// <param name="location">Physical location of the document</param>
@@ -425,22 +451,20 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Insert(document, location, transaction);
}
foreach (var index in _indexes.Values) index.Insert(document, location, transaction);
}
}
/// <summary>
/// Updates a document in all indexes
/// Updates a document in all indexes
/// </summary>
/// <param name="oldDocument">Old version of document</param>
/// <param name="newDocument">New version of document</param>
/// <param name="oldLocation">Physical location of old document</param>
/// <param name="newLocation">Physical location of new document</param>
/// <param name="transaction">Transaction context</param>
public void UpdateInAll(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation, ITransaction transaction)
public void UpdateInAll(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation,
ITransaction transaction)
{
if (oldDocument == null)
throw new ArgumentNullException(nameof(oldDocument));
@@ -450,14 +474,12 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Update(oldDocument, newDocument, oldLocation, newLocation, transaction);
}
}
}
/// <summary>
/// Deletes a document from all indexes
/// Deletes a document from all indexes
/// </summary>
/// <param name="document">Document to delete</param>
/// <param name="location">Physical location of the document</param>
@@ -469,35 +491,29 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
lock (_lock)
{
foreach (var index in _indexes.Values)
{
index.Delete(document, location, transaction);
}
foreach (var index in _indexes.Values) index.Delete(document, location, transaction);
}
}
/// <summary>
/// Generates an index name from property paths
/// Generates an index name from property paths
/// </summary>
private static string GenerateIndexName(string[] propertyPaths)
{
return $"idx_{string.Join("_", propertyPaths)}";
}
private CollectionIndexDefinition<T> RebuildDefinition(string name, string[] paths, bool isUnique, IndexType type, int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
private CollectionIndexDefinition<T> RebuildDefinition(string name, string[] paths, bool isUnique, IndexType type,
int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
{
var param = Expression.Parameter(typeof(T), "u");
Expression body;
if (paths.Length == 1)
{
body = Expression.PropertyOrField(param, paths[0]);
}
else
{
body = Expression.NewArrayInit(typeof(object),
paths.Select(p => Expression.Convert(Expression.PropertyOrField(param, p), typeof(object))));
}
var objectBody = Expression.Convert(body, typeof(object));
var lambda = Expression.Lambda<Func<T, object>>(objectBody, param);
@@ -506,12 +522,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Gets the root page identifier for the primary index.
/// </summary>
public uint PrimaryRootPageId => _metadata.PrimaryRootPageId;
/// <summary>
/// Rebinds cached metadata and index instances from persisted metadata.
/// Rebinds cached metadata and index instances from persisted metadata.
/// </summary>
/// <param name="metadata">The collection metadata used to rebuild index state.</param>
internal void RebindFromMetadata(CollectionMetadata metadata)
@@ -525,16 +536,22 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
throw new ObjectDisposedException(nameof(CollectionIndexManager<TId, T>));
foreach (var index in _indexes.Values)
{
try { index.Dispose(); } catch { /* Best effort */ }
}
try
{
index.Dispose();
}
catch
{
/* Best effort */
}
_indexes.Clear();
_metadata = metadata;
foreach (var idxMeta in _metadata.Indexes)
{
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type, idxMeta.Dimensions, idxMeta.Metric);
var definition = RebuildDefinition(idxMeta.Name, idxMeta.PropertyPaths, idxMeta.IsUnique, idxMeta.Type,
idxMeta.Dimensions, idxMeta.Metric);
var index = new CollectionSecondaryIndex<TId, T>(definition, _storage, _mapper, idxMeta.RootPageId);
_indexes[idxMeta.Name] = index;
}
@@ -542,7 +559,7 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Sets the root page identifier for the primary index.
/// Sets the root page identifier for the primary index.
/// </summary>
/// <param name="pageId">The root page identifier.</param>
public void SetPrimaryRootPageId(uint pageId)
@@ -558,83 +575,57 @@ public sealed class CollectionIndexManager<TId, T> : IDisposable where T : class
}
/// <summary>
/// Gets the current collection metadata.
/// Gets the current collection metadata.
/// </summary>
/// <returns>The collection metadata.</returns>
public CollectionMetadata GetMetadata() => _metadata;
public CollectionMetadata GetMetadata()
{
return _metadata;
}
private void SaveMetadata()
{
UpdateMetadata();
_storage.SaveCollectionMetadata(_metadata);
}
/// <summary>
/// Releases resources used by the index manager.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// No auto-save on dispose to avoid unnecessary I/O if no changes
lock (_lock)
{
foreach (var index in _indexes.Values)
{
try { index.Dispose(); } catch { /* Best effort */ }
}
_indexes.Clear();
_disposed = true;
}
GC.SuppressFinalize(this);
}
}
/// <summary>
/// Helper class to analyze LINQ expressions and extract property paths
/// Helper class to analyze LINQ expressions and extract property paths
/// </summary>
public static class ExpressionAnalyzer
{
/// <summary>
/// Extracts property paths from a lambda expression.
/// Supports simple property access (p => p.Age) and anonymous types (p => new { p.City, p.Age }).
/// Extracts property paths from a lambda expression.
/// Supports simple property access (p => p.Age) and anonymous types (p => new { p.City, p.Age }).
/// </summary>
/// <param name="expression">The lambda expression to analyze.</param>
public static string[] ExtractPropertyPaths(LambdaExpression expression)
{
if (expression.Body is MemberExpression memberExpr)
{
// Simple property: p => p.Age
return new[] { memberExpr.Member.Name };
}
else if (expression.Body is NewExpression newExpr)
{
if (expression.Body is NewExpression newExpr)
// Compound key via anonymous type: p => new { p.City, p.Age }
return newExpr.Arguments
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
else if (expression.Body is UnaryExpression { NodeType: ExpressionType.Convert } unaryExpr)
if (expression.Body is UnaryExpression { NodeType: ExpressionType.Convert } unaryExpr)
{
// Handle Convert(Member) or Convert(New)
if (unaryExpr.Operand is MemberExpression innerMember)
{
// Wrapped property: p => (object)p.Age
return new[] { innerMember.Member.Name };
}
else if (unaryExpr.Operand is NewExpression innerNew)
{
if (unaryExpr.Operand is NewExpression innerNew)
// Wrapped anonymous type: p => (object)new { p.City, p.Age }
return innerNew.Arguments
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
.OfType<MemberExpression>()
.Select(m => m.Member.Name)
.ToArray();
}
throw new ArgumentException(
src/CBDD.Core/Indexing/CollectionSecondaryIndex.cs
+140 -153
View File
@@ -1,47 +1,27 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using System;
using System.Linq;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents a secondary (non-primary) index on a document collection.
/// Provides a high-level, strongly-typed wrapper around the low-level BTreeIndex.
/// Handles automatic key extraction from documents using compiled expressions.
/// Represents a secondary (non-primary) index on a document collection.
/// Provides a high-level, strongly-typed wrapper around the low-level BTreeIndex.
/// Handles automatic key extraction from documents using compiled expressions.
/// </summary>
/// <typeparam name="TId">Primary key type</typeparam>
/// <typeparam name="T">Document type</typeparam>
public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : class
{
private readonly CollectionIndexDefinition<T> _definition;
private readonly BTreeIndex? _btreeIndex;
private readonly VectorSearchIndex? _vectorIndex;
private readonly RTreeIndex? _spatialIndex;
private readonly IDocumentMapper<TId, T> _mapper;
private readonly RTreeIndex? _spatialIndex;
private readonly VectorSearchIndex? _vectorIndex;
private bool _disposed;
/// <summary>
/// Gets the index definition
/// </summary>
public CollectionIndexDefinition<T> Definition => _definition;
/// <summary>
/// Gets the underlying BTree index (for advanced scenarios)
/// </summary>
public BTreeIndex? BTreeIndex => _btreeIndex;
/// <summary>
/// Gets the root page identifier for the underlying index structure.
/// </summary>
public uint RootPageId => _btreeIndex?.RootPageId ?? _vectorIndex?.RootPageId ?? _spatialIndex?.RootPageId ?? 0;
/// <summary>
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}"/> class.
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}" /> class.
/// </summary>
/// <param name="definition">The index definition.</param>
/// <param name="storage">The storage engine.</param>
@@ -57,7 +37,8 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
}
/// <summary>
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}"/> class from index storage abstractions.
/// Initializes a new instance of the <see cref="CollectionSecondaryIndex{TId, T}" /> class from index storage
/// abstractions.
/// </summary>
/// <param name="definition">The index definition.</param>
/// <param name="storage">The index storage abstraction.</param>
@@ -69,7 +50,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
IDocumentMapper<TId, T> mapper,
uint rootPageId = 0)
{
_definition = definition ?? throw new ArgumentNullException(nameof(definition));
Definition = definition ?? throw new ArgumentNullException(nameof(definition));
_mapper = mapper ?? throw new ArgumentNullException(nameof(mapper));
var indexOptions = definition.ToIndexOptions();
@@ -77,25 +58,55 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
if (indexOptions.Type == IndexType.Vector)
{
_vectorIndex = new VectorSearchIndex(storage, indexOptions, rootPageId);
_btreeIndex = null;
BTreeIndex = null;
_spatialIndex = null;
}
else if (indexOptions.Type == IndexType.Spatial)
{
_spatialIndex = new RTreeIndex(storage, indexOptions, rootPageId);
_btreeIndex = null;
BTreeIndex = null;
_vectorIndex = null;
}
else
{
_btreeIndex = new BTreeIndex(storage, indexOptions, rootPageId);
BTreeIndex = new BTreeIndex(storage, indexOptions, rootPageId);
_vectorIndex = null;
_spatialIndex = null;
}
}
/// <summary>
/// Inserts a document into this index
/// Gets the index definition
/// </summary>
public CollectionIndexDefinition<T> Definition { get; }
/// <summary>
/// Gets the underlying BTree index (for advanced scenarios)
/// </summary>
public BTreeIndex? BTreeIndex { get; }
/// <summary>
/// Gets the root page identifier for the underlying index structure.
/// </summary>
public uint RootPageId => BTreeIndex?.RootPageId ?? _vectorIndex?.RootPageId ?? _spatialIndex?.RootPageId ?? 0;
/// <summary>
/// Releases resources used by this index wrapper.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
// BTreeIndex doesn't currently implement IDisposable
// Future: may need to flush buffers, close resources
_disposed = true;
GC.SuppressFinalize(this);
}
/// <summary>
/// Inserts a document into this index
/// </summary>
/// <param name="document">Document to index</param>
/// <param name="location">Physical location of the document</param>
@@ -106,7 +117,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
throw new ArgumentNullException(nameof(document));
// Extract key using compiled selector (fast!)
var keyValue = _definition.KeySelector(document);
object? keyValue = Definition.KeySelector(document);
if (keyValue == null)
return; // Skip null keys
@@ -114,45 +125,38 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
// Vector Index Support
if (keyValue is float[] singleVector)
{
_vectorIndex.Insert(singleVector, location, transaction);
}
else if (keyValue is IEnumerable<float[]> vectors)
{
foreach (var v in vectors)
{
foreach (float[] v in vectors)
_vectorIndex.Insert(v, location, transaction);
}
}
}
else if (_spatialIndex != null)
{
// Geospatial Index Support
if (keyValue is ValueTuple<double, double> t)
{
_spatialIndex.Insert(GeoBox.FromPoint(new GeoPoint(t.Item1, t.Item2)), location, transaction);
}
}
else if (_btreeIndex != null)
else if (BTreeIndex != null)
{
// BTree Index logic
var userKey = ConvertToIndexKey(keyValue);
var documentId = _mapper.GetId(document);
var compositeKey = CreateCompositeKey(userKey, _mapper.ToIndexKey(documentId));
_btreeIndex.Insert(compositeKey, location, transaction?.TransactionId);
BTreeIndex.Insert(compositeKey, location, transaction?.TransactionId);
}
}
/// <summary>
/// Updates a document in this index (delete old, insert new).
/// Only updates if the indexed key has changed.
/// Updates a document in this index (delete old, insert new).
/// Only updates if the indexed key has changed.
/// </summary>
/// <param name="oldDocument">Old version of document</param>
/// <param name="newDocument">New version of document</param>
/// <param name="oldLocation">Physical location of old document</param>
/// <param name="newLocation">Physical location of new document</param>
/// <param name="transaction">Optional transaction</param>
public void Update(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation, ITransaction transaction)
public void Update(T oldDocument, T newDocument, DocumentLocation oldLocation, DocumentLocation newLocation,
ITransaction transaction)
{
if (oldDocument == null)
throw new ArgumentNullException(nameof(oldDocument));
@@ -160,8 +164,8 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
throw new ArgumentNullException(nameof(newDocument));
// Extract keys from both versions
var oldKey = _definition.KeySelector(oldDocument);
var newKey = _definition.KeySelector(newDocument);
object? oldKey = Definition.KeySelector(oldDocument);
object? newKey = Definition.KeySelector(newDocument);
// If keys are the same, no index update needed (optimization)
if (Equals(oldKey, newKey))
@@ -174,7 +178,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
var oldUserKey = ConvertToIndexKey(oldKey);
var oldCompositeKey = CreateCompositeKey(oldUserKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Delete(oldCompositeKey, oldLocation, transaction?.TransactionId);
BTreeIndex?.Delete(oldCompositeKey, oldLocation, transaction?.TransactionId);
}
// Insert new entry if it has a key
@@ -182,12 +186,12 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
var newUserKey = ConvertToIndexKey(newKey);
var newCompositeKey = CreateCompositeKey(newUserKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Insert(newCompositeKey, newLocation, transaction?.TransactionId);
BTreeIndex?.Insert(newCompositeKey, newLocation, transaction?.TransactionId);
}
}
/// <summary>
/// Deletes a document from this index
/// Deletes a document from this index
/// </summary>
/// <param name="document">Document to remove from index</param>
/// <param name="location">Physical location of the document</param>
@@ -198,7 +202,7 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
throw new ArgumentNullException(nameof(document));
// Extract key
var keyValue = _definition.KeySelector(document);
object? keyValue = Definition.KeySelector(document);
if (keyValue == null)
return; // Nothing to delete
@@ -207,11 +211,11 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
// Create composite key and delete
var compositeKey = CreateCompositeKey(userKey, _mapper.ToIndexKey(documentId));
_btreeIndex?.Delete(compositeKey, location, transaction?.TransactionId);
BTreeIndex?.Delete(compositeKey, location, transaction?.TransactionId);
}
/// <summary>
/// Seeks a single document by exact key match (O(log n))
/// Seeks a single document by exact key match (O(log n))
/// </summary>
/// <param name="key">Key value to seek</param>
/// <param name="transaction">Optional transaction to read uncommitted changes</param>
@@ -222,31 +226,31 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
return null;
if (_vectorIndex != null && key is float[] query)
{
return _vectorIndex.Search(query, 1, transaction: transaction).FirstOrDefault().Location;
}
if (_btreeIndex != null)
if (BTreeIndex != null)
{
var userKey = ConvertToIndexKey(key);
var minComposite = CreateCompositeKeyBoundary(userKey, useMinObjectId: true);
var maxComposite = CreateCompositeKeyBoundary(userKey, useMinObjectId: false);
var firstEntry = _btreeIndex.Range(minComposite, maxComposite, IndexDirection.Forward, transaction?.TransactionId).FirstOrDefault();
return firstEntry.Location.PageId == 0 ? null : (DocumentLocation?)firstEntry.Location;
var minComposite = CreateCompositeKeyBoundary(userKey, true);
var maxComposite = CreateCompositeKeyBoundary(userKey, false);
var firstEntry = BTreeIndex
.Range(minComposite, maxComposite, IndexDirection.Forward, transaction?.TransactionId).FirstOrDefault();
return firstEntry.Location.PageId == 0 ? null : firstEntry.Location;
}
return null;
}
/// <summary>
/// Performs a vector nearest-neighbor search.
/// Performs a vector nearest-neighbor search.
/// </summary>
/// <param name="query">The query vector.</param>
/// <param name="k">The number of results to return.</param>
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction.</param>
/// <returns>The matching vector search results.</returns>
public IEnumerable<VectorSearchResult> VectorSearch(float[] query, int k, int efSearch = 100, ITransaction? transaction = null)
public IEnumerable<VectorSearchResult> VectorSearch(float[] query, int k, int efSearch = 100,
ITransaction? transaction = null)
{
if (_vectorIndex == null)
throw new InvalidOperationException("This index is not a vector index.");
@@ -255,30 +259,29 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
}
/// <summary>
/// Performs geospatial distance search
/// Performs geospatial distance search
/// </summary>
/// <param name="center">The center point.</param>
/// <param name="radiusKm">The search radius in kilometers.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Near((double Latitude, double Longitude) center, double radiusKm, ITransaction? transaction = null)
public IEnumerable<DocumentLocation> Near((double Latitude, double Longitude) center, double radiusKm,
ITransaction? transaction = null)
{
if (_spatialIndex == null)
throw new InvalidOperationException("This index is not a spatial index.");
var queryBox = SpatialMath.BoundingBox(center.Latitude, center.Longitude, radiusKm);
foreach (var loc in _spatialIndex.Search(queryBox, transaction))
{
yield return loc;
}
foreach (var loc in _spatialIndex.Search(queryBox, transaction)) yield return loc;
}
/// <summary>
/// Performs geospatial bounding box search
/// Performs geospatial bounding box search
/// </summary>
/// <param name="min">The minimum latitude/longitude corner.</param>
/// <param name="max">The maximum latitude/longitude corner.</param>
/// <param name="transaction">Optional transaction.</param>
public IEnumerable<DocumentLocation> Within((double Latitude, double Longitude) min, (double Latitude, double Longitude) max, ITransaction? transaction = null)
public IEnumerable<DocumentLocation> Within((double Latitude, double Longitude) min,
(double Latitude, double Longitude) max, ITransaction? transaction = null)
{
if (_spatialIndex == null)
throw new InvalidOperationException("This index is not a spatial index.");
@@ -288,16 +291,17 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
}
/// <summary>
/// Scans a range of keys (O(log n + k) where k is result count)
/// Scans a range of keys (O(log n + k) where k is result count)
/// </summary>
/// <param name="minKey">Minimum key (inclusive), null for unbounded</param>
/// <param name="maxKey">Maximum key (inclusive), null for unbounded</param>
/// <param name="direction">Scan direction.</param>
/// <param name="transaction">Optional transaction to read uncommitted changes</param>
/// <returns>Enumerable of document locations in key order</returns>
public IEnumerable<DocumentLocation> Range(object? minKey, object? maxKey, IndexDirection direction = IndexDirection.Forward, ITransaction? transaction = null)
public IEnumerable<DocumentLocation> Range(object? minKey, object? maxKey,
IndexDirection direction = IndexDirection.Forward, ITransaction? transaction = null)
{
if (_btreeIndex == null) yield break;
if (BTreeIndex == null) yield break;
// Handle unbounded ranges
IndexKey actualMinKey;
@@ -313,12 +317,12 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
{
actualMinKey = new IndexKey(new byte[0]);
var userMaxKey = ConvertToIndexKey(maxKey!);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, useMinObjectId: false); // Max boundary
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, false); // Max boundary
}
else if (maxKey == null)
{
var userMinKey = ConvertToIndexKey(minKey);
actualMinKey = CreateCompositeKeyBoundary(userMinKey, useMinObjectId: true); // Min boundary
actualMinKey = CreateCompositeKeyBoundary(userMinKey, true); // Min boundary
actualMaxKey = new IndexKey(Enumerable.Repeat((byte)0xFF, 255).ToArray());
}
else
@@ -330,91 +334,36 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
// Create composite boundaries:
// Min: (userMinKey, ObjectId.Empty) - captures all docs with key >= userMinKey
// Max: (userMaxKey, ObjectId.MaxValue) - captures all docs with key <= userMaxKey
actualMinKey = CreateCompositeKeyBoundary(userMinKey, useMinObjectId: true);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, useMinObjectId: false);
actualMinKey = CreateCompositeKeyBoundary(userMinKey, true);
actualMaxKey = CreateCompositeKeyBoundary(userMaxKey, false);
}
// Use BTreeIndex.Range with WAL-aware reads and direction
// Extract DocumentLocation from each entry
foreach (var entry in _btreeIndex.Range(actualMinKey, actualMaxKey, direction, transaction?.TransactionId))
{
foreach (var entry in BTreeIndex.Range(actualMinKey, actualMaxKey, direction, transaction?.TransactionId))
yield return entry.Location;
}
}
/// <summary>
/// Gets statistics about this index
/// Gets statistics about this index
/// </summary>
public CollectionIndexInfo GetInfo()
{
return new CollectionIndexInfo
{
Name = _definition.Name,
PropertyPaths = _definition.PropertyPaths,
IsUnique = _definition.IsUnique,
Type = _definition.Type,
IsPrimary = _definition.IsPrimary,
Name = Definition.Name,
PropertyPaths = Definition.PropertyPaths,
IsUnique = Definition.IsUnique,
Type = Definition.Type,
IsPrimary = Definition.IsPrimary,
EstimatedDocumentCount = 0, // TODO: Track or calculate document count
EstimatedSizeBytes = 0 // TODO: Calculate index size
EstimatedSizeBytes = 0 // TODO: Calculate index size
};
}
#region Composite Key Support (SQLite-style for Duplicate Keys)
/// <summary>
/// Creates a composite key by concatenating user key with document ID.
/// This allows duplicate user keys while maintaining BTree uniqueness.
/// Format: [UserKeyBytes] + [DocumentIdKey]
/// </summary>
private IndexKey CreateCompositeKey(IndexKey userKey, IndexKey documentIdKey)
{
// Allocate buffer: user key + document ID key length
var compositeBytes = new byte[userKey.Data.Length + documentIdKey.Data.Length];
// Copy user key
userKey.Data.CopyTo(compositeBytes.AsSpan(0, userKey.Data.Length));
// Append document ID key
documentIdKey.Data.CopyTo(compositeBytes.AsSpan(userKey.Data.Length));
return new IndexKey(compositeBytes);
}
/// <summary>
/// Creates a composite key for range query boundary.
/// Uses MIN or MAX ID representation to capture all documents with the user key.
/// </summary>
private IndexKey CreateCompositeKeyBoundary(IndexKey userKey, bool useMinObjectId)
{
// For range boundaries, we use an empty key for Min and a very large key for Max
// to wrap around all possible IDs for this user key.
IndexKey idBoundary = useMinObjectId
? new IndexKey(Array.Empty<byte>())
: new IndexKey(Enumerable.Repeat((byte)0xFF, 16).ToArray()); // Using 16 as a safe max for GUID/ObjectId
return CreateCompositeKey(userKey, idBoundary);
}
/// <summary>
/// Extracts the original user key from a composite key by removing the ObjectId suffix.
/// Used when we need to return the original indexed value.
/// </summary>
private IndexKey ExtractUserKey(IndexKey compositeKey)
{
// Composite key = UserKey + ObjectId(12 bytes)
var userKeyLength = compositeKey.Data.Length - 12;
if (userKeyLength <= 0)
return compositeKey; // Fallback for malformed keys
var userKeyBytes = compositeKey.Data.Slice(0, userKeyLength);
return new IndexKey(userKeyBytes);
}
#endregion
/// <summary>
/// Converts a CLR value to an IndexKey for BTree storage.
/// Supports all common .NET types.
/// Converts a CLR value to an IndexKey for BTree storage.
/// Supports all common .NET types.
/// </summary>
private IndexKey ConvertToIndexKey(object value)
{
@@ -434,18 +383,56 @@ public sealed class CollectionSecondaryIndex<TId, T> : IDisposable where T : cla
};
}
#region Composite Key Support (SQLite-style for Duplicate Keys)
/// <summary>
/// Releases resources used by this index wrapper.
/// Creates a composite key by concatenating user key with document ID.
/// This allows duplicate user keys while maintaining BTree uniqueness.
/// Format: [UserKeyBytes] + [DocumentIdKey]
/// </summary>
public void Dispose()
private IndexKey CreateCompositeKey(IndexKey userKey, IndexKey documentIdKey)
{
if (_disposed)
return;
// Allocate buffer: user key + document ID key length
var compositeBytes = new byte[userKey.Data.Length + documentIdKey.Data.Length];
// BTreeIndex doesn't currently implement IDisposable
// Future: may need to flush buffers, close resources
// Copy user key
userKey.Data.CopyTo(compositeBytes.AsSpan(0, userKey.Data.Length));
_disposed = true;
GC.SuppressFinalize(this);
// Append document ID key
documentIdKey.Data.CopyTo(compositeBytes.AsSpan(userKey.Data.Length));
return new IndexKey(compositeBytes);
}
/// <summary>
/// Creates a composite key for range query boundary.
/// Uses MIN or MAX ID representation to capture all documents with the user key.
/// </summary>
private IndexKey CreateCompositeKeyBoundary(IndexKey userKey, bool useMinObjectId)
{
// For range boundaries, we use an empty key for Min and a very large key for Max
// to wrap around all possible IDs for this user key.
var idBoundary = useMinObjectId
? new IndexKey(Array.Empty<byte>())
: new IndexKey(Enumerable.Repeat((byte)0xFF, 16).ToArray()); // Using 16 as a safe max for GUID/ObjectId
return CreateCompositeKey(userKey, idBoundary);
}
/// <summary>
/// Extracts the original user key from a composite key by removing the ObjectId suffix.
/// Used when we need to return the original indexed value.
/// </summary>
private IndexKey ExtractUserKey(IndexKey compositeKey)
{
// Composite key = UserKey + ObjectId(12 bytes)
int userKeyLength = compositeKey.Data.Length - 12;
if (userKeyLength <= 0)
return compositeKey; // Fallback for malformed keys
var userKeyBytes = compositeKey.Data.Slice(0, userKeyLength);
return new IndexKey(userKeyBytes);
}
#endregion
}
src/CBDD.Core/Indexing/GeoSpatialExtensions.cs
+8 -6
View File
@@ -3,27 +3,29 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
public static class GeoSpatialExtensions
{
/// <summary>
/// Performs a geospatial proximity search (Near) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// Performs a geospatial proximity search (Near) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// </summary>
/// <param name="point">The coordinate tuple (Latitude, Longitude) property of the entity.</param>
/// <param name="center">The center point (Latitude, Longitude) for the proximity search.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>True if the point is within the specified radius.</returns>
public static bool Near(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) center, double radiusKm)
public static bool Near(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) center,
double radiusKm)
{
return true;
}
/// <summary>
/// Performs a geospatial bounding box search (Within) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// Performs a geospatial bounding box search (Within) on a coordinate tuple property.
/// This method is a marker for the LINQ query provider and is optimized using R-Tree indexes if available.
/// </summary>
/// <param name="point">The coordinate tuple (Latitude, Longitude) property of the entity.</param>
/// <param name="min">The minimum (Latitude, Longitude) of the bounding box.</param>
/// <param name="max">The maximum (Latitude, Longitude) of the bounding box.</param>
/// <returns>True if the point is within the specified bounding box.</returns>
public static bool Within(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) min, (double Latitude, double Longitude) max)
public static bool Within(this (double Latitude, double Longitude) point, (double Latitude, double Longitude) min,
(double Latitude, double Longitude) max)
{
return true;
}
src/CBDD.Core/Indexing/HashIndex.cs
+15 -24
View File
@@ -1,13 +1,10 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Hash-based index for exact-match lookups.
/// Uses simple bucket-based hashing with collision handling.
/// Hash-based index for exact-match lookups.
/// Uses simple bucket-based hashing with collision handling.
/// </summary>
public sealed class HashIndex
{
@@ -15,7 +12,7 @@ public sealed class HashIndex
private readonly IndexOptions _options;
/// <summary>
/// Initializes a new instance of the <see cref="HashIndex"/> class.
/// Initializes a new instance of the <see cref="HashIndex" /> class.
/// </summary>
/// <param name="options">The index options.</param>
public HashIndex(IndexOptions options)
@@ -25,16 +22,16 @@ public sealed class HashIndex
}
/// <summary>
/// Inserts a key-location pair into the hash index
/// Inserts a key-location pair into the hash index
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
public void Insert(IndexKey key, DocumentLocation location)
{
if (_options.Unique && TryFind(key, out _))
throw new InvalidOperationException($"Duplicate key violation for unique index");
throw new InvalidOperationException("Duplicate key violation for unique index");
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
{
@@ -46,46 +43,43 @@ public sealed class HashIndex
}
/// <summary>
/// Finds a document location by exact key match
/// Finds a document location by exact key match
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">When this method returns, contains the matched document location if found.</param>
/// <returns><see langword="true"/> if a matching entry is found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if a matching entry is found; otherwise, <see langword="false" />.</returns>
public bool TryFind(IndexKey key, out DocumentLocation location)
{
location = default;
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
return false;
foreach (var entry in bucket)
{
if (entry.Key == key)
{
location = entry.Location;
return true;
}
}
return false;
}
/// <summary>
/// Removes an entry from the index
/// Removes an entry from the index
/// </summary>
/// <param name="key">The index key.</param>
/// <param name="location">The document location.</param>
/// <returns><see langword="true"/> if an entry is removed; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if an entry is removed; otherwise, <see langword="false" />.</returns>
public bool Remove(IndexKey key, DocumentLocation location)
{
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
return false;
for (int i = 0; i < bucket.Count; i++)
{
for (var i = 0; i < bucket.Count; i++)
if (bucket[i].Key == key &&
bucket[i].Location.PageId == location.PageId &&
bucket[i].Location.SlotIndex == location.SlotIndex)
@@ -97,27 +91,24 @@ public sealed class HashIndex
return true;
}
}
return false;
}
/// <summary>
/// Gets all entries matching the key
/// Gets all entries matching the key
/// </summary>
/// <param name="key">The index key.</param>
/// <returns>All matching index entries.</returns>
public IEnumerable<IndexEntry> FindAll(IndexKey key)
{
var hashCode = key.GetHashCode();
int hashCode = key.GetHashCode();
if (!_buckets.TryGetValue(hashCode, out var bucket))
yield break;
foreach (var entry in bucket)
{
if (entry.Key == key)
yield return entry;
}
}
}
src/CBDD.Core/Indexing/IBTreeCursor.cs
+11 -14
View File
@@ -1,49 +1,46 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
using System;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents a cursor for traversing a B+Tree index.
/// Provides low-level primitives for building complex queries.
/// Represents a cursor for traversing a B+Tree index.
/// Provides low-level primitives for building complex queries.
/// </summary>
public interface IBTreeCursor : IDisposable
{
/// <summary>
/// Gets the current entry at the cursor position.
/// Throws InvalidOperationException if cursor is invalid or uninitialized.
/// Gets the current entry at the cursor position.
/// Throws InvalidOperationException if cursor is invalid or uninitialized.
/// </summary>
IndexEntry Current { get; }
/// <summary>
/// Moves the cursor to the first entry in the index.
/// Moves the cursor to the first entry in the index.
/// </summary>
/// <returns>True if the index is not empty; otherwise, false.</returns>
bool MoveToFirst();
/// <summary>
/// Moves the cursor to the last entry in the index.
/// Moves the cursor to the last entry in the index.
/// </summary>
/// <returns>True if the index is not empty; otherwise, false.</returns>
bool MoveToLast();
/// <summary>
/// Seeks to the specified key.
/// If exact match found, positions there and returns true.
/// If not found, positions at the next greater key and returns false.
/// Seeks to the specified key.
/// If exact match found, positions there and returns true.
/// If not found, positions at the next greater key and returns false.
/// </summary>
/// <param name="key">Key to seek</param>
/// <returns>True if exact match found; false if positioned at next greater key.</returns>
bool Seek(IndexKey key);
/// <summary>
/// Advances the cursor to the next entry.
/// Advances the cursor to the next entry.
/// </summary>
/// <returns>True if successfully moved; false if end of index reached.</returns>
bool MoveNext();
/// <summary>
/// Moves the cursor to the previous entry.
/// Moves the cursor to the previous entry.
/// </summary>
/// <returns>True if successfully moved; false if start of index reached.</returns>
bool MovePrev();
src/CBDD.Core/Indexing/IndexKey.cs
+68 -37
View File
@@ -1,13 +1,12 @@
using System.Text;
using ZB.MOM.WW.CBDD.Bson;
using System;
using System.Linq;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Represents a key in an index.
/// Implemented as struct for efficient index operations.
/// Note: Contains byte array so cannot be readonly struct.
/// Represents a key in an index.
/// Implemented as struct for efficient index operations.
/// Note: Contains byte array so cannot be readonly struct.
/// </summary>
public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
{
@@ -15,17 +14,17 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
private readonly int _hashCode;
/// <summary>
/// Gets the minimum possible index key.
/// Gets the minimum possible index key.
/// </summary>
public static IndexKey MinKey => new IndexKey(Array.Empty<byte>());
public static IndexKey MinKey => new(Array.Empty<byte>());
/// <summary>
/// Gets the maximum possible index key.
/// Gets the maximum possible index key.
/// </summary>
public static IndexKey MaxKey => new IndexKey(Enumerable.Repeat((byte)0xFF, 32).ToArray());
public static IndexKey MaxKey => new(Enumerable.Repeat((byte)0xFF, 32).ToArray());
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from raw key bytes.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from raw key bytes.
/// </summary>
/// <param name="data">The key bytes.</param>
public IndexKey(ReadOnlySpan<byte> data)
@@ -35,7 +34,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from an object identifier.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from an object identifier.
/// </summary>
/// <param name="objectId">The object identifier value.</param>
public IndexKey(ObjectId objectId)
@@ -46,7 +45,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a 32-bit integer.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a 32-bit integer.
/// </summary>
/// <param name="value">The integer value.</param>
public IndexKey(int value)
@@ -56,7 +55,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a 64-bit integer.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a 64-bit integer.
/// </summary>
/// <param name="value">The integer value.</param>
public IndexKey(long value)
@@ -66,17 +65,17 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a string.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a string.
/// </summary>
/// <param name="value">The string value.</param>
public IndexKey(string value)
{
_data = System.Text.Encoding.UTF8.GetBytes(value);
_data = Encoding.UTF8.GetBytes(value);
_hashCode = ComputeHashCode(_data);
}
/// <summary>
/// Initializes a new instance of the <see cref="IndexKey"/> struct from a GUID.
/// Initializes a new instance of the <see cref="IndexKey" /> struct from a GUID.
/// </summary>
/// <param name="value">The GUID value.</param>
public IndexKey(Guid value)
@@ -86,27 +85,28 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Gets the raw byte data for this key.
/// Gets the raw byte data for this key.
/// </summary>
public readonly ReadOnlySpan<byte> Data => _data;
/// <summary>
/// Compares this key to another key.
/// Compares this key to another key.
/// </summary>
/// <param name="other">The key to compare with.</param>
/// <returns>
/// A value less than zero if this key is less than <paramref name="other"/>, zero if equal, or greater than zero if greater.
/// A value less than zero if this key is less than <paramref name="other" />, zero if equal, or greater than zero if
/// greater.
/// </returns>
public readonly int CompareTo(IndexKey other)
{
if (_data == null) return other._data == null ? 0 : -1;
if (other._data == null) return 1;
var minLength = Math.Min(_data.Length, other._data.Length);
int minLength = Math.Min(_data.Length, other._data.Length);
for (int i = 0; i < minLength; i++)
for (var i = 0; i < minLength; i++)
{
var cmp = _data[i].CompareTo(other._data[i]);
int cmp = _data[i].CompareTo(other._data[i]);
if (cmp != 0)
return cmp;
}
@@ -115,10 +115,10 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Determines whether this key equals another key.
/// Determines whether this key equals another key.
/// </summary>
/// <param name="other">The key to compare with.</param>
/// <returns><see langword="true"/> if the keys are equal; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the keys are equal; otherwise, <see langword="false" />.</returns>
public readonly bool Equals(IndexKey other)
{
if (_hashCode != other._hashCode)
@@ -131,17 +131,46 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <inheritdoc />
public override readonly bool Equals(object? obj) => obj is IndexKey other && Equals(other);
public readonly override bool Equals(object? obj)
{
return obj is IndexKey other && Equals(other);
}
/// <inheritdoc />
public override readonly int GetHashCode() => _hashCode;
public readonly override int GetHashCode()
{
return _hashCode;
}
public static bool operator ==(IndexKey left, IndexKey right) => left.Equals(right);
public static bool operator !=(IndexKey left, IndexKey right) => !left.Equals(right);
public static bool operator <(IndexKey left, IndexKey right) => left.CompareTo(right) < 0;
public static bool operator >(IndexKey left, IndexKey right) => left.CompareTo(right) > 0;
public static bool operator <=(IndexKey left, IndexKey right) => left.CompareTo(right) <= 0;
public static bool operator >=(IndexKey left, IndexKey right) => left.CompareTo(right) >= 0;
public static bool operator ==(IndexKey left, IndexKey right)
{
return left.Equals(right);
}
public static bool operator !=(IndexKey left, IndexKey right)
{
return !left.Equals(right);
}
public static bool operator <(IndexKey left, IndexKey right)
{
return left.CompareTo(right) < 0;
}
public static bool operator >(IndexKey left, IndexKey right)
{
return left.CompareTo(right) > 0;
}
public static bool operator <=(IndexKey left, IndexKey right)
{
return left.CompareTo(right) <= 0;
}
public static bool operator >=(IndexKey left, IndexKey right)
{
return left.CompareTo(right) >= 0;
}
private static int ComputeHashCode(ReadOnlySpan<byte> data)
{
@@ -151,7 +180,7 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
}
/// <summary>
/// Creates an <see cref="IndexKey"/> from a supported CLR value.
/// Creates an <see cref="IndexKey" /> from a supported CLR value.
/// </summary>
/// <typeparam name="T">The CLR type of the value.</typeparam>
/// <param name="value">The value to convert.</param>
@@ -167,11 +196,12 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
if (typeof(T) == typeof(Guid)) return new IndexKey((Guid)(object)value);
if (typeof(T) == typeof(byte[])) return new IndexKey((byte[])(object)value);
throw new NotSupportedException($"Type {typeof(T).Name} is not supported as an IndexKey. Provide a custom mapping.");
throw new NotSupportedException(
$"Type {typeof(T).Name} is not supported as an IndexKey. Provide a custom mapping.");
}
/// <summary>
/// Converts this key to a CLR value of type <typeparamref name="T"/>.
/// Converts this key to a CLR value of type <typeparamref name="T" />.
/// </summary>
/// <typeparam name="T">The CLR type to read from this key.</typeparam>
/// <returns>The converted value.</returns>
@@ -182,10 +212,11 @@ public struct IndexKey : IEquatable<IndexKey>, IComparable<IndexKey>
if (typeof(T) == typeof(ObjectId)) return (T)(object)new ObjectId(_data);
if (typeof(T) == typeof(int)) return (T)(object)BitConverter.ToInt32(_data);
if (typeof(T) == typeof(long)) return (T)(object)BitConverter.ToInt64(_data);
if (typeof(T) == typeof(string)) return (T)(object)System.Text.Encoding.UTF8.GetString(_data);
if (typeof(T) == typeof(string)) return (T)(object)Encoding.UTF8.GetString(_data);
if (typeof(T) == typeof(Guid)) return (T)(object)new Guid(_data);
if (typeof(T) == typeof(byte[])) return (T)(object)_data;
throw new NotSupportedException($"Type {typeof(T).Name} cannot be extracted from IndexKey. Provide a custom mapping.");
throw new NotSupportedException(
$"Type {typeof(T).Name} cannot be extracted from IndexKey. Provide a custom mapping.");
}
}
src/CBDD.Core/Indexing/IndexOptions.cs
+61 -45
View File
@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Types of indices supported
/// Types of indices supported
/// </summary>
public enum IndexType : byte
{
@@ -22,7 +22,7 @@ public enum IndexType : byte
}
/// <summary>
/// Distance metrics for vector search
/// Distance metrics for vector search
/// </summary>
public enum VectorMetric : byte
{
@@ -37,85 +37,94 @@ public enum VectorMetric : byte
}
/// <summary>
/// Index options and configuration.
/// Implemented as readonly struct for efficiency.
/// Index options and configuration.
/// Implemented as readonly struct for efficiency.
/// </summary>
public readonly struct IndexOptions
{
/// <summary>
/// Gets the configured index type.
/// Gets the configured index type.
/// </summary>
public IndexType Type { get; init; }
/// <summary>
/// Gets a value indicating whether the index enforces uniqueness.
/// Gets a value indicating whether the index enforces uniqueness.
/// </summary>
public bool Unique { get; init; }
/// <summary>
/// Gets the indexed field names.
/// Gets the indexed field names.
/// </summary>
public string[] Fields { get; init; }
// Vector search options
/// <summary>
/// Gets the vector dimensionality for vector indexes.
/// Gets the vector dimensionality for vector indexes.
/// </summary>
public int Dimensions { get; init; }
/// <summary>
/// Gets the distance metric used for vector similarity.
/// Gets the distance metric used for vector similarity.
/// </summary>
public VectorMetric Metric { get; init; }
/// <summary>
/// Gets the minimum number of graph connections per node.
/// Gets the minimum number of graph connections per node.
/// </summary>
public int M { get; init; } // Min number of connections per node
/// <summary>
/// Gets the size of the dynamic candidate list during index construction.
/// Gets the size of the dynamic candidate list during index construction.
/// </summary>
public int EfConstruction { get; init; } // Size of dynamic candidate list for construction
/// <summary>
/// Creates non-unique B+Tree index options.
/// Creates non-unique B+Tree index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateBTree(params string[] fields) => new()
public static IndexOptions CreateBTree(params string[] fields)
{
Type = IndexType.BTree,
Unique = false,
Fields = fields
};
return new IndexOptions
{
Type = IndexType.BTree,
Unique = false,
Fields = fields
};
}
/// <summary>
/// Creates unique B+Tree index options.
/// Creates unique B+Tree index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateUnique(params string[] fields) => new()
public static IndexOptions CreateUnique(params string[] fields)
{
Type = IndexType.BTree,
Unique = true,
Fields = fields
};
return new IndexOptions
{
Type = IndexType.BTree,
Unique = true,
Fields = fields
};
}
/// <summary>
/// Creates hash index options.
/// Creates hash index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateHash(params string[] fields) => new()
public static IndexOptions CreateHash(params string[] fields)
{
Type = IndexType.Hash,
Unique = false,
Fields = fields
};
return new IndexOptions
{
Type = IndexType.Hash,
Unique = false,
Fields = fields
};
}
/// <summary>
/// Creates vector index options.
/// Creates vector index options.
/// </summary>
/// <param name="dimensions">The vector dimensionality.</param>
/// <param name="metric">The similarity metric.</param>
@@ -123,26 +132,33 @@ public readonly struct IndexOptions
/// <param name="ef">The candidate list size used during index construction.</param>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateVector(int dimensions, VectorMetric metric = VectorMetric.Cosine, int m = 16, int ef = 200, params string[] fields) => new()
public static IndexOptions CreateVector(int dimensions, VectorMetric metric = VectorMetric.Cosine, int m = 16,
int ef = 200, params string[] fields)
{
Type = IndexType.Vector,
Unique = false,
Fields = fields,
Dimensions = dimensions,
Metric = metric,
M = m,
EfConstruction = ef
};
return new IndexOptions
{
Type = IndexType.Vector,
Unique = false,
Fields = fields,
Dimensions = dimensions,
Metric = metric,
M = m,
EfConstruction = ef
};
}
/// <summary>
/// Creates spatial index options.
/// Creates spatial index options.
/// </summary>
/// <param name="fields">The indexed field names.</param>
/// <returns>The configured index options.</returns>
public static IndexOptions CreateSpatial(params string[] fields) => new()
public static IndexOptions CreateSpatial(params string[] fields)
{
Type = IndexType.Spatial,
Unique = false,
Fields = fields
};
return new IndexOptions
{
Type = IndexType.Spatial,
Unique = false,
Fields = fields
};
}
}
src/CBDD.Core/Indexing/Internal/GeoTypes.cs
+18 -18
View File
@@ -1,33 +1,38 @@
namespace ZB.MOM.WW.CBDD.Core.Indexing.Internal;
/// <summary>
/// Basic spatial point (Latitude/Longitude)
/// Internal primitive for R-Tree logic.
/// Basic spatial point (Latitude/Longitude)
/// Internal primitive for R-Tree logic.
/// </summary>
internal record struct GeoPoint(double Latitude, double Longitude)
{
/// <summary>
/// Gets an empty point at coordinate origin.
/// Gets an empty point at coordinate origin.
/// </summary>
public static GeoPoint Empty => new(0, 0);
}
/// <summary>
/// Minimum Bounding Box (MBR) for spatial indexing
/// Internal primitive for R-Tree logic.
/// Minimum Bounding Box (MBR) for spatial indexing
/// Internal primitive for R-Tree logic.
/// </summary>
internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, double MaxLon)
{
/// <summary>
/// Gets an empty bounding box sentinel value.
/// Gets an empty bounding box sentinel value.
/// </summary>
public static GeoBox Empty => new(double.MaxValue, double.MaxValue, double.MinValue, double.MinValue);
/// <summary>
/// Determines whether this box contains the specified point.
/// Gets the area of this bounding box.
/// </summary>
public double Area => Math.Max(0, MaxLat - MinLat) * Math.Max(0, MaxLon - MinLon);
/// <summary>
/// Determines whether this box contains the specified point.
/// </summary>
/// <param name="point">The point to test.</param>
/// <returns><see langword="true"/> if the point is inside this box; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the point is inside this box; otherwise, <see langword="false" />.</returns>
public bool Contains(GeoPoint point)
{
return point.Latitude >= MinLat && point.Latitude <= MaxLat &&
@@ -35,10 +40,10 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
}
/// <summary>
/// Determines whether this box intersects another box.
/// Determines whether this box intersects another box.
/// </summary>
/// <param name="other">The other box to test.</param>
/// <returns><see langword="true"/> if the boxes intersect; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the boxes intersect; otherwise, <see langword="false" />.</returns>
public bool Intersects(GeoBox other)
{
return !(other.MinLat > MaxLat || other.MaxLat < MinLat ||
@@ -46,7 +51,7 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
}
/// <summary>
/// Creates a box that contains a single point.
/// Creates a box that contains a single point.
/// </summary>
/// <param name="point">The point to convert.</param>
/// <returns>A bounding box containing the specified point.</returns>
@@ -56,7 +61,7 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
}
/// <summary>
/// Expands this box to include the specified point.
/// Expands this box to include the specified point.
/// </summary>
/// <param name="point">The point to include.</param>
/// <returns>A new expanded bounding box.</returns>
@@ -70,7 +75,7 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
}
/// <summary>
/// Expands this box to include the specified box.
/// Expands this box to include the specified box.
/// </summary>
/// <param name="other">The box to include.</param>
/// <returns>A new expanded bounding box.</returns>
@@ -82,9 +87,4 @@ internal record struct GeoBox(double MinLat, double MinLon, double MaxLat, doubl
Math.Max(MaxLat, other.MaxLat),
Math.Max(MaxLon, other.MaxLon));
}
/// <summary>
/// Gets the area of this bounding box.
/// </summary>
public double Area => Math.Max(0, MaxLat - MinLat) * Math.Max(0, MaxLon - MinLon);
}
src/CBDD.Core/Indexing/InternalEntry.cs
+3 -5
View File
@@ -1,21 +1,19 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
public struct InternalEntry
{
/// <summary>
/// Gets or sets the separator key.
/// Gets or sets the separator key.
/// </summary>
public IndexKey Key { get; set; }
/// <summary>
/// Gets or sets the child page identifier.
/// Gets or sets the child page identifier.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Initializes a new instance of the <see cref="InternalEntry"/> struct.
/// Initializes a new instance of the <see cref="InternalEntry" /> struct.
/// </summary>
/// <param name="key">The separator key.</param>
/// <param name="pageId">The child page identifier.</param>
src/CBDD.Core/Indexing/RTreeIndex.cs
+80 -73
View File
@@ -1,25 +1,23 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
using System.Buffers;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// R-Tree Index implementation for Geospatial Indexing.
/// Uses Quadratic Split algorithm for simplicity and efficiency in paged storage.
/// R-Tree Index implementation for Geospatial Indexing.
/// Uses Quadratic Split algorithm for simplicity and efficiency in paged storage.
/// </summary>
internal class RTreeIndex : IDisposable
{
private readonly IIndexStorage _storage;
private readonly IndexOptions _options;
private uint _rootPageId;
private readonly object _lock = new();
private readonly IndexOptions _options;
private readonly int _pageSize;
private readonly IIndexStorage _storage;
/// <summary>
/// Initializes a new instance of the <see cref="RTreeIndex"/> class.
/// Initializes a new instance of the <see cref="RTreeIndex" /> class.
/// </summary>
/// <param name="storage">The storage engine used for page operations.</param>
/// <param name="options">The index options.</param>
@@ -28,46 +26,53 @@ internal class RTreeIndex : IDisposable
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
_rootPageId = rootPageId;
RootPageId = rootPageId;
_pageSize = _storage.PageSize;
if (_rootPageId == 0)
{
InitializeNewIndex();
}
if (RootPageId == 0) InitializeNewIndex();
}
/// <summary>
/// Gets the current root page identifier.
/// Gets the current root page identifier.
/// </summary>
public uint RootPageId => _rootPageId;
public uint RootPageId { get; private set; }
/// <summary>
/// Releases resources used by the index.
/// </summary>
public void Dispose()
{
}
private void InitializeNewIndex()
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_rootPageId = _storage.AllocatePage();
SpatialPage.Initialize(buffer, _rootPageId, true, 0);
_storage.WritePageImmediate(_rootPageId, buffer);
RootPageId = _storage.AllocatePage();
SpatialPage.Initialize(buffer, RootPageId, true, 0);
_storage.WritePageImmediate(RootPageId, buffer);
}
finally
{
ReturnPageBuffer(buffer);
}
finally { ReturnPageBuffer(buffer); }
}
/// <summary>
/// Searches for document locations whose minimum bounding rectangles intersect the specified area.
/// Searches for document locations whose minimum bounding rectangles intersect the specified area.
/// </summary>
/// <param name="area">The area to search.</param>
/// <param name="transaction">The optional transaction context.</param>
/// <returns>A sequence of matching document locations.</returns>
public IEnumerable<DocumentLocation> Search(GeoBox area, ITransaction? transaction = null)
{
if (_rootPageId == 0) yield break;
if (RootPageId == 0) yield break;
var stack = new Stack<uint>();
stack.Push(_rootPageId);
stack.Push(RootPageId);
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
while (stack.Count > 0)
@@ -78,29 +83,28 @@ internal class RTreeIndex : IDisposable
bool isLeaf = SpatialPage.GetIsLeaf(buffer);
ushort count = SpatialPage.GetEntryCount(buffer);
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var mbr, out var pointer);
if (area.Intersects(mbr))
{
if (isLeaf)
{
yield return pointer;
}
else
{
stack.Push(pointer.PageId);
}
}
}
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
/// <summary>
/// Inserts a bounding rectangle and document location into the index.
/// Inserts a bounding rectangle and document location into the index.
/// </summary>
/// <param name="mbr">The minimum bounding rectangle to index.</param>
/// <param name="loc">The document location associated with the rectangle.</param>
@@ -109,7 +113,7 @@ internal class RTreeIndex : IDisposable
{
lock (_lock)
{
var leafPageId = ChooseLeaf(_rootPageId, mbr, transaction);
uint leafPageId = ChooseLeaf(RootPageId, mbr, transaction);
InsertIntoNode(leafPageId, mbr, loc, transaction);
}
}
@@ -117,7 +121,7 @@ internal class RTreeIndex : IDisposable
private uint ChooseLeaf(uint rootId, GeoBox mbr, ITransaction? transaction)
{
uint currentId = rootId;
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
while (true)
@@ -127,10 +131,10 @@ internal class RTreeIndex : IDisposable
ushort count = SpatialPage.GetEntryCount(buffer);
uint bestChild = 0;
double minEnlargement = double.MaxValue;
double minArea = double.MaxValue;
var minEnlargement = double.MaxValue;
var minArea = double.MaxValue;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var childMbr, out var pointer);
@@ -156,12 +160,15 @@ internal class RTreeIndex : IDisposable
currentId = bestChild;
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void InsertIntoNode(uint pageId, GeoBox mbr, DocumentLocation pointer, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -186,17 +193,20 @@ internal class RTreeIndex : IDisposable
SplitNode(pageId, mbr, pointer, transaction);
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void UpdateMBRUpwards(uint pageId, ITransaction? transaction)
{
var buffer = RentPageBuffer();
var parentBuffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
byte[] parentBuffer = RentPageBuffer();
try
{
uint currentId = pageId;
while (currentId != _rootPageId)
while (currentId != RootPageId)
{
_storage.ReadPage(currentId, transaction?.TransactionId, buffer);
var currentMbr = SpatialPage.CalculateMBR(buffer);
@@ -206,9 +216,9 @@ internal class RTreeIndex : IDisposable
_storage.ReadPage(parentId, transaction?.TransactionId, parentBuffer);
ushort count = SpatialPage.GetEntryCount(parentBuffer);
bool changed = false;
var changed = false;
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(parentBuffer, i, out var mbr, out var pointer);
if (pointer.PageId == currentId)
@@ -218,6 +228,7 @@ internal class RTreeIndex : IDisposable
SpatialPage.WriteEntry(parentBuffer, i, currentMbr, pointer);
changed = true;
}
break;
}
}
@@ -241,8 +252,8 @@ internal class RTreeIndex : IDisposable
private void SplitNode(uint pageId, GeoBox newMbr, DocumentLocation newPointer, ITransaction? transaction)
{
var buffer = RentPageBuffer();
var newBuffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
byte[] newBuffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -253,11 +264,12 @@ internal class RTreeIndex : IDisposable
// Collect all entries including the new one
var entries = new List<(GeoBox Mbr, DocumentLocation Pointer)>();
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
SpatialPage.ReadEntry(buffer, i, out var m, out var p);
entries.Add((m, p));
}
entries.Add((newMbr, newPointer));
// Pick Seeds
@@ -277,8 +289,8 @@ internal class RTreeIndex : IDisposable
SpatialPage.WriteEntry(newBuffer, 0, seed2.Mbr, seed2.Pointer);
SpatialPage.SetEntryCount(newBuffer, 1);
GeoBox mbr1 = seed1.Mbr;
GeoBox mbr2 = seed2.Mbr;
var mbr1 = seed1.Mbr;
var mbr2 = seed2.Mbr;
// Distribute remaining entries
while (entries.Count > 0)
@@ -320,7 +332,7 @@ internal class RTreeIndex : IDisposable
}
// Propagate split upwards
if (pageId == _rootPageId)
if (pageId == RootPageId)
{
// New Root
uint newRootId = _storage.AllocatePage();
@@ -334,7 +346,7 @@ internal class RTreeIndex : IDisposable
else
_storage.WritePageImmediate(newRootId, buffer);
_rootPageId = newRootId;
RootPageId = newRootId;
// Update parent pointers
UpdateParentPointer(pageId, newRootId, transaction);
@@ -356,7 +368,7 @@ internal class RTreeIndex : IDisposable
private void UpdateParentPointer(uint pageId, uint parentId, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -366,27 +378,29 @@ internal class RTreeIndex : IDisposable
else
_storage.WritePageImmediate(pageId, buffer);
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private void PickSeeds(List<(GeoBox Mbr, DocumentLocation Pointer)> entries, out (GeoBox Mbr, DocumentLocation Pointer) s1, out (GeoBox Mbr, DocumentLocation Pointer) s2)
private void PickSeeds(List<(GeoBox Mbr, DocumentLocation Pointer)> entries,
out (GeoBox Mbr, DocumentLocation Pointer) s1, out (GeoBox Mbr, DocumentLocation Pointer) s2)
{
double maxWaste = double.MinValue;
var maxWaste = double.MinValue;
s1 = entries[0];
s2 = entries[1];
for (int i = 0; i < entries.Count; i++)
for (var i = 0; i < entries.Count; i++)
for (int j = i + 1; j < entries.Count; j++)
{
for (int j = i + 1; j < entries.Count; j++)
var combined = entries[i].Mbr.ExpandTo(entries[j].Mbr);
double waste = combined.Area - entries[i].Mbr.Area - entries[j].Mbr.Area;
if (waste > maxWaste)
{
var combined = entries[i].Mbr.ExpandTo(entries[j].Mbr);
double waste = combined.Area - entries[i].Mbr.Area - entries[j].Mbr.Area;
if (waste > maxWaste)
{
maxWaste = waste;
s1 = entries[i];
s2 = entries[j];
}
maxWaste = waste;
s1 = entries[i];
s2 = entries[j];
}
}
}
@@ -400,11 +414,4 @@ internal class RTreeIndex : IDisposable
{
ArrayPool<byte>.Shared.Return(buffer);
}
/// <summary>
/// Releases resources used by the index.
/// </summary>
public void Dispose()
{
}
}
src/CBDD.Core/Indexing/SpatialMath.cs
+27 -11
View File
@@ -7,16 +7,19 @@ public static class SpatialMath
private const double EarthRadiusKm = 6371.0;
/// <summary>
/// Calculates distance between two points on Earth using Haversine formula.
/// Result in kilometers.
/// Calculates distance between two points on Earth using Haversine formula.
/// Result in kilometers.
/// </summary>
/// <param name="p1">The first point.</param>
/// <param name="p2">The second point.</param>
/// <returns>The distance in kilometers.</returns>
internal static double DistanceKm(GeoPoint p1, GeoPoint p2) => DistanceKm(p1.Latitude, p1.Longitude, p2.Latitude, p2.Longitude);
internal static double DistanceKm(GeoPoint p1, GeoPoint p2)
{
return DistanceKm(p1.Latitude, p1.Longitude, p2.Latitude, p2.Longitude);
}
/// <summary>
/// Calculates distance between two coordinates on Earth using Haversine formula.
/// Calculates distance between two coordinates on Earth using Haversine formula.
/// </summary>
/// <param name="lat1">The latitude of the first point.</param>
/// <param name="lon1">The longitude of the first point.</param>
@@ -37,24 +40,30 @@ public static class SpatialMath
}
/// <summary>
/// Creates a Bounding Box (MBR) centered at a point with a given radius in km.
/// Creates a Bounding Box (MBR) centered at a point with a given radius in km.
/// </summary>
/// <param name="center">The center point.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>The bounding box.</returns>
internal static GeoBox BoundingBox(GeoPoint center, double radiusKm) => BoundingBox(center.Latitude, center.Longitude, radiusKm);
internal static GeoBox BoundingBox(GeoPoint center, double radiusKm)
{
return BoundingBox(center.Latitude, center.Longitude, radiusKm);
}
/// <summary>
/// Creates a bounding box from a coordinate and radius.
/// Creates a bounding box from a coordinate and radius.
/// </summary>
/// <param name="lat">The center latitude.</param>
/// <param name="lon">The center longitude.</param>
/// <param name="radiusKm">The radius in kilometers.</param>
/// <returns>The bounding box.</returns>
internal static GeoBox InternalBoundingBox(double lat, double lon, double radiusKm) => BoundingBox(lat, lon, radiusKm);
internal static GeoBox InternalBoundingBox(double lat, double lon, double radiusKm)
{
return BoundingBox(lat, lon, radiusKm);
}
/// <summary>
/// Creates a bounding box centered at a coordinate with a given radius in kilometers.
/// Creates a bounding box centered at a coordinate with a given radius in kilometers.
/// </summary>
/// <param name="lat">The center latitude.</param>
/// <param name="lon">The center longitude.</param>
@@ -72,6 +81,13 @@ public static class SpatialMath
lon + dLon);
}
private static double ToRadians(double degrees) => degrees * Math.PI / 180.0;
private static double ToDegrees(double radians) => radians * 180.0 / Math.PI;
private static double ToRadians(double degrees)
{
return degrees * Math.PI / 180.0;
}
private static double ToDegrees(double radians)
{
return radians * 180.0 / Math.PI;
}
}
src/CBDD.Core/Indexing/VectorMath.cs
+11 -19
View File
@@ -1,17 +1,15 @@
using System.Runtime.Intrinsics;
using System.Runtime.Intrinsics.X86;
using System.Runtime.InteropServices;
using System.Numerics;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// Optimized vector math utilities using SIMD if available.
/// Optimized vector math utilities using SIMD if available.
/// </summary>
public static class VectorMath
{
/// <summary>
/// Computes vector distance according to the selected metric.
/// Computes vector distance according to the selected metric.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
@@ -29,7 +27,7 @@ public static class VectorMath
}
/// <summary>
/// Computes cosine similarity between two vectors.
/// Computes cosine similarity between two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
@@ -45,7 +43,7 @@ public static class VectorMath
}
/// <summary>
/// Computes the dot product of two vectors.
/// Computes the dot product of two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
@@ -56,7 +54,7 @@ public static class VectorMath
throw new ArgumentException("Vectors must have same length");
float dot = 0;
int i = 0;
var i = 0;
// SIMD Optimization for .NET
if (Vector.IsHardwareAccelerated && v1.Length >= Vector<float>.Count)
@@ -65,26 +63,20 @@ public static class VectorMath
var v1Span = MemoryMarshal.Cast<float, Vector<float>>(v1);
var v2Span = MemoryMarshal.Cast<float, Vector<float>>(v2);
foreach (var chunk in Enumerable.Range(0, v1Span.Length))
{
vDot += v1Span[chunk] * v2Span[chunk];
}
foreach (int chunk in Enumerable.Range(0, v1Span.Length)) vDot += v1Span[chunk] * v2Span[chunk];
dot = Vector.Dot(vDot, Vector<float>.One);
i = v1Span.Length * Vector<float>.Count;
}
// Remaining elements
for (; i < v1.Length; i++)
{
dot += v1[i] * v2[i];
}
for (; i < v1.Length; i++) dot += v1[i] * v2[i];
return dot;
}
/// <summary>
/// Computes squared Euclidean distance between two vectors.
/// Computes squared Euclidean distance between two vectors.
/// </summary>
/// <param name="v1">The first vector.</param>
/// <param name="v2">The second vector.</param>
@@ -95,7 +87,7 @@ public static class VectorMath
throw new ArgumentException("Vectors must have same length");
float dist = 0;
int i = 0;
var i = 0;
if (Vector.IsHardwareAccelerated && v1.Length >= Vector<float>.Count)
{
@@ -103,7 +95,7 @@ public static class VectorMath
var v1Span = MemoryMarshal.Cast<float, Vector<float>>(v1);
var v2Span = MemoryMarshal.Cast<float, Vector<float>>(v2);
foreach (var chunk in Enumerable.Range(0, v1Span.Length))
foreach (int chunk in Enumerable.Range(0, v1Span.Length))
{
var diff = v1Span[chunk] - v2Span[chunk];
vDist += diff * diff;
src/CBDD.Core/Indexing/VectorSearchExtensions.cs
+12 -6
View File
@@ -3,26 +3,32 @@ namespace ZB.MOM.WW.CBDD.Core.Indexing;
public static class VectorSearchExtensions
{
/// <summary>
/// Performs a similarity search on a vector property.
/// This method is a marker for the LINQ query provider and is optimized using HNSW indexes if available.
/// Performs a similarity search on a vector property.
/// This method is a marker for the LINQ query provider and is optimized using HNSW indexes if available.
/// </summary>
/// <param name="vector">The vector property of the entity.</param>
/// <param name="query">The query vector to compare against.</param>
/// <param name="k">Number of nearest neighbors to return.</param>
/// <returns>True if the document is part of the top-k results (always returns true when evaluated in memory for compilation purposes).</returns>
/// <returns>
/// True if the document is part of the top-k results (always returns true when evaluated in memory for
/// compilation purposes).
/// </returns>
public static bool VectorSearch(this float[] vector, float[] query, int k)
{
return true;
}
/// <summary>
/// Performs a similarity search on a collection of vector properties.
/// Used for entities with multiple vectors per document.
/// Performs a similarity search on a collection of vector properties.
/// Used for entities with multiple vectors per document.
/// </summary>
/// <param name="vectors">The vector collection of the entity.</param>
/// <param name="query">The query vector to compare against.</param>
/// <param name="k">Number of nearest neighbors to return.</param>
/// <returns>True if the document is part of the top-k results (always returns true when evaluated in memory for compilation purposes).</returns>
/// <returns>
/// True if the document is part of the top-k results (always returns true when evaluated in memory for
/// compilation purposes).
/// </returns>
public static bool VectorSearch(this IEnumerable<float[]> vectors, float[] query, int k)
{
return true;
src/CBDD.Core/Indexing/VectorSearchIndex.cs
+94 -75
View File
@@ -1,29 +1,22 @@
using System.Buffers;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Core.Indexing;
/// <summary>
/// HNSW (Hierarchical Navigable Small World) index implementation.
/// Handles multi-vector indexing and similarity searches.
/// HNSW (Hierarchical Navigable Small World) index implementation.
/// Handles multi-vector indexing and similarity searches.
/// </summary>
public sealed class VectorSearchIndex
{
private struct NodeReference
{
public uint PageId;
public int NodeIndex;
public int MaxLevel;
}
private readonly IIndexStorage _storage;
private readonly IndexOptions _options;
private uint _rootPageId;
private readonly Random _random = new(42);
private readonly IIndexStorage _storage;
/// <summary>
/// Initializes a new vector search index.
/// Initializes a new vector search index.
/// </summary>
/// <param name="storage">The storage engine used by the index.</param>
/// <param name="options">Index configuration options.</param>
@@ -34,7 +27,7 @@ public sealed class VectorSearchIndex
}
/// <summary>
/// Initializes a new vector search index.
/// Initializes a new vector search index.
/// </summary>
/// <param name="storage">The index storage abstraction used by the index.</param>
/// <param name="options">Index configuration options.</param>
@@ -43,16 +36,16 @@ public sealed class VectorSearchIndex
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_options = options;
_rootPageId = rootPageId;
RootPageId = rootPageId;
}
/// <summary>
/// Gets the root page identifier of the index.
/// Gets the root page identifier of the index.
/// </summary>
public uint RootPageId => _rootPageId;
public uint RootPageId { get; private set; }
/// <summary>
/// Inserts a vector and its document location into the index.
/// Inserts a vector and its document location into the index.
/// </summary>
/// <param name="vector">The vector values to index.</param>
/// <param name="docLocation">The document location associated with the vector.</param>
@@ -60,28 +53,33 @@ public sealed class VectorSearchIndex
public void Insert(float[] vector, DocumentLocation docLocation, ITransaction? transaction = null)
{
if (vector.Length != _options.Dimensions)
throw new ArgumentException($"Vector dimension mismatch. Expected {_options.Dimensions}, got {vector.Length}");
throw new ArgumentException(
$"Vector dimension mismatch. Expected {_options.Dimensions}, got {vector.Length}");
// 1. Determine level for new node
int targetLevel = GetRandomLevel();
// 2. If index is empty, create first page and first node
if (_rootPageId == 0)
if (RootPageId == 0)
{
_rootPageId = CreateNewPage(transaction);
var pageBuffer = RentPageBuffer();
RootPageId = CreateNewPage(transaction);
byte[] pageBuffer = RentPageBuffer();
try
{
_storage.ReadPage(_rootPageId, transaction?.TransactionId, pageBuffer);
_storage.ReadPage(RootPageId, transaction?.TransactionId, pageBuffer);
VectorPage.WriteNode(pageBuffer, 0, docLocation, targetLevel, vector, _options.Dimensions);
VectorPage.IncrementNodeCount(pageBuffer); // Helper needs to be added or handled
if (transaction != null)
_storage.WritePage(_rootPageId, transaction.TransactionId, pageBuffer);
_storage.WritePage(RootPageId, transaction.TransactionId, pageBuffer);
else
_storage.WritePageImmediate(_rootPageId, pageBuffer);
_storage.WritePageImmediate(RootPageId, pageBuffer);
}
finally { ReturnPageBuffer(pageBuffer); }
finally
{
ReturnPageBuffer(pageBuffer);
}
return;
}
@@ -92,9 +90,7 @@ public sealed class VectorSearchIndex
// 4. Greedy search down to targetLevel+1
for (int l = entryPoint.MaxLevel; l > targetLevel; l--)
{
currentPoint = GreedySearch(currentPoint, vector, l, transaction);
}
// 5. Create the new node
var newNode = AllocateNode(vector, docLocation, targetLevel, transaction);
@@ -105,23 +101,18 @@ public sealed class VectorSearchIndex
var neighbors = SearchLayer(currentPoint, vector, _options.EfConstruction, l, transaction);
var selectedNeighbors = SelectNeighbors(neighbors, vector, _options.M, l, transaction);
foreach (var neighbor in selectedNeighbors)
{
AddBidirectionalLink(newNode, neighbor, l, transaction);
}
foreach (var neighbor in selectedNeighbors) AddBidirectionalLink(newNode, neighbor, l, transaction);
// Move currentPoint down for next level if available
currentPoint = GreedySearch(currentPoint, vector, l, transaction);
}
// 7. Update entry point if new node is higher
if (targetLevel > entryPoint.MaxLevel)
{
UpdateEntryPoint(newNode, transaction);
}
if (targetLevel > entryPoint.MaxLevel) UpdateEntryPoint(newNode, transaction);
}
private IEnumerable<NodeReference> SelectNeighbors(IEnumerable<NodeReference> candidates, float[] query, int m, int level, ITransaction? transaction)
private IEnumerable<NodeReference> SelectNeighbors(IEnumerable<NodeReference> candidates, float[] query, int m,
int level, ITransaction? transaction)
{
// Simple heuristic: just take top M nearest.
// HNSW Paper suggests more complex heuristic to maintain connectivity diversity.
@@ -136,14 +127,14 @@ public sealed class VectorSearchIndex
private void Link(NodeReference from, NodeReference to, int level, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(from.PageId, transaction?.TransactionId, buffer);
var links = VectorPage.GetLinksSpan(buffer, from.NodeIndex, level, _options.Dimensions, _options.M);
// Find first empty slot (PageId == 0)
for (int i = 0; i < links.Length; i += 6)
for (var i = 0; i < links.Length; i += 6)
{
var existing = DocumentLocation.ReadFrom(links.Slice(i, 6));
if (existing.PageId == 0)
@@ -160,7 +151,10 @@ public sealed class VectorSearchIndex
// If full, we should technically prune or redistribute links as per HNSW paper.
// For now, we assume M is large enough or we skip (limited connectivity).
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private NodeReference AllocateNode(float[] vector, DocumentLocation docLoc, int level, ITransaction? transaction)
@@ -168,10 +162,10 @@ public sealed class VectorSearchIndex
// Find a page with space or create new
// For simplicity, we search for a page with available slots or append to a new one.
// Implementation omitted for brevity but required for full persistence.
uint pageId = _rootPageId; // Placeholder: need allocation strategy
int index = 0;
uint pageId = RootPageId; // Placeholder: need allocation strategy
var index = 0;
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(pageId, transaction?.TransactionId, buffer);
@@ -184,7 +178,10 @@ public sealed class VectorSearchIndex
else
_storage.WritePageImmediate(pageId, buffer);
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
return new NodeReference { PageId = pageId, NodeIndex = index, MaxLevel = level };
}
@@ -197,7 +194,7 @@ public sealed class VectorSearchIndex
private NodeReference GreedySearch(NodeReference entryPoint, float[] query, int level, ITransaction? transaction)
{
bool changed = true;
var changed = true;
var current = entryPoint;
float currentDist = VectorMath.Distance(query, LoadVector(current, transaction), _options.Metric);
@@ -215,10 +212,12 @@ public sealed class VectorSearchIndex
}
}
}
return current;
}
private IEnumerable<NodeReference> SearchLayer(NodeReference entryPoint, float[] query, int ef, int level, ITransaction? transaction)
private IEnumerable<NodeReference> SearchLayer(NodeReference entryPoint, float[] query, int ef, int level,
ITransaction? transaction)
{
var visited = new HashSet<NodeReference>();
var candidates = new PriorityQueue<NodeReference, float>();
@@ -233,14 +232,13 @@ public sealed class VectorSearchIndex
{
float d_c = 0;
candidates.TryPeek(out var c, out d_c);
result.TryPeek(out var f, out var d_f);
result.TryPeek(out var f, out float d_f);
if (d_c > -d_f) break;
candidates.Dequeue();
foreach (var e in GetNeighbors(c, level, transaction))
{
if (!visited.Contains(e))
{
visited.Add(e);
@@ -254,7 +252,6 @@ public sealed class VectorSearchIndex
if (result.Count > ef) result.Dequeue();
}
}
}
}
// Convert result to list (ordered by distance)
@@ -268,48 +265,49 @@ public sealed class VectorSearchIndex
{
// For now, assume a fixed location or track it in page 0 of index
// TODO: Real implementation
return new NodeReference { PageId = _rootPageId, NodeIndex = 0, MaxLevel = 0 };
return new NodeReference { PageId = RootPageId, NodeIndex = 0, MaxLevel = 0 };
}
private float[] LoadVector(NodeReference node, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
float[] vector = new float[_options.Dimensions];
var vector = new float[_options.Dimensions];
VectorPage.ReadNodeData(buffer, node.NodeIndex, out _, out _, vector);
return vector;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
/// <summary>
/// Searches the index for the nearest vectors to the query.
/// Searches the index for the nearest vectors to the query.
/// </summary>
/// <param name="query">The query vector.</param>
/// <param name="k">The number of nearest results to return.</param>
/// <param name="efSearch">The search breadth parameter.</param>
/// <param name="transaction">Optional transaction context.</param>
/// <returns>The nearest vector search results.</returns>
public IEnumerable<VectorSearchResult> Search(float[] query, int k, int efSearch = 100, ITransaction? transaction = null)
public IEnumerable<VectorSearchResult> Search(float[] query, int k, int efSearch = 100,
ITransaction? transaction = null)
{
if (_rootPageId == 0) yield break;
if (RootPageId == 0) yield break;
var entryPoint = GetEntryPoint();
var currentPoint = entryPoint;
// 1. Greedy search through higher layers to find entry point for level 0
for (int l = entryPoint.MaxLevel; l > 0; l--)
{
currentPoint = GreedySearch(currentPoint, query, l, transaction);
}
for (int l = entryPoint.MaxLevel; l > 0; l--) currentPoint = GreedySearch(currentPoint, query, l, transaction);
// 2. Comprehensive search on level 0
var nearest = SearchLayer(currentPoint, query, Math.Max(efSearch, k), 0, transaction);
// 3. Return top-k results
int count = 0;
var count = 0;
foreach (var node in nearest)
{
if (count++ >= k) break;
@@ -322,26 +320,29 @@ public sealed class VectorSearchIndex
private DocumentLocation LoadDocumentLocation(NodeReference node, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
VectorPage.ReadNodeData(buffer, node.NodeIndex, out var loc, out _, new float[0]); // Vector not needed here
return loc;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private IEnumerable<NodeReference> GetNeighbors(NodeReference node, int level, ITransaction? transaction)
{
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
var results = new List<NodeReference>();
try
{
_storage.ReadPage(node.PageId, transaction?.TransactionId, buffer);
var links = VectorPage.GetLinksSpan(buffer, node.NodeIndex, level, _options.Dimensions, _options.M);
for (int i = 0; i < links.Length; i += 6)
for (var i = 0; i < links.Length; i += 6)
{
var loc = DocumentLocation.ReadFrom(links.Slice(i, 6));
if (loc.PageId == 0) break; // End of links
@@ -349,7 +350,11 @@ public sealed class VectorSearchIndex
results.Add(new NodeReference { PageId = loc.PageId, NodeIndex = loc.SlotIndex });
}
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
return results;
}
@@ -357,29 +362,43 @@ public sealed class VectorSearchIndex
{
// Probability p = 1/M for each level
double p = 1.0 / _options.M;
int level = 0;
while (_random.NextDouble() < p && level < 15)
{
level++;
}
var level = 0;
while (_random.NextDouble() < p && level < 15) level++;
return level;
}
private uint CreateNewPage(ITransaction? transaction)
{
uint pageId = _storage.AllocatePage();
var buffer = RentPageBuffer();
byte[] buffer = RentPageBuffer();
try
{
VectorPage.Initialize(buffer, pageId, _options.Dimensions, _options.M);
_storage.WritePageImmediate(pageId, buffer);
return pageId;
}
finally { ReturnPageBuffer(buffer); }
finally
{
ReturnPageBuffer(buffer);
}
}
private byte[] RentPageBuffer() => System.Buffers.ArrayPool<byte>.Shared.Rent(_storage.PageSize);
private void ReturnPageBuffer(byte[] buffer) => System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
private byte[] RentPageBuffer()
{
return ArrayPool<byte>.Shared.Rent(_storage.PageSize);
}
private void ReturnPageBuffer(byte[] buffer)
{
ArrayPool<byte>.Shared.Return(buffer);
}
private struct NodeReference
{
public uint PageId;
public int NodeIndex;
public int MaxLevel;
}
}
public record struct VectorSearchResult(DocumentLocation Location, float Distance);
src/CBDD.Core/Metadata/EntityTypeBuilder.cs
+58 -64
View File
@@ -1,42 +1,42 @@
using ZB.MOM.WW.CBDD.Core.Indexing;
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Metadata;
public class EntityTypeBuilder<T> where T : class
{
/// <summary>
/// Gets the configured collection name for the entity type.
/// Gets the configured collection name for the entity type.
/// </summary>
public string? CollectionName { get; private set; }
/// <summary>
/// Gets the configured indexes for the entity type.
/// Gets the configured indexes for the entity type.
/// </summary>
public List<IndexBuilder<T>> Indexes { get; } = new();
/// <summary>
/// Gets the primary key selector expression.
/// Gets the primary key selector expression.
/// </summary>
public LambdaExpression? PrimaryKeySelector { get; private set; }
/// <summary>
/// Gets a value indicating whether the primary key value is generated on add.
/// Gets a value indicating whether the primary key value is generated on add.
/// </summary>
public bool ValueGeneratedOnAdd { get; private set; }
/// <summary>
/// Gets the configured primary key property name.
/// Gets the configured primary key property name.
/// </summary>
public string? PrimaryKeyName { get; private set; }
/// <summary>
/// Gets the configured property converter types keyed by property name.
/// Gets the configured property converter types keyed by property name.
/// </summary>
public Dictionary<string, Type> PropertyConverters { get; } = new();
/// <summary>
/// Sets the collection name for the entity type.
/// Sets the collection name for the entity type.
/// </summary>
/// <param name="name">The collection name.</param>
/// <returns>The current entity type builder.</returns>
@@ -47,21 +47,22 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Adds an index for the specified key selector.
/// Adds an index for the specified key selector.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The key selector expression.</param>
/// <param name="name">The optional index name.</param>
/// <param name="unique">A value indicating whether the index is unique.</param>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasIndex<TKey>(Expression<Func<T, TKey>> keySelector, string? name = null, bool unique = false)
public EntityTypeBuilder<T> HasIndex<TKey>(Expression<Func<T, TKey>> keySelector, string? name = null,
bool unique = false)
{
Indexes.Add(new IndexBuilder<T>(keySelector, name, unique));
return this;
}
/// <summary>
/// Adds a vector index for the specified key selector.
/// Adds a vector index for the specified key selector.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The key selector expression.</param>
@@ -69,14 +70,15 @@ public class EntityTypeBuilder<T> where T : class
/// <param name="metric">The vector similarity metric.</param>
/// <param name="name">The optional index name.</param>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions, VectorMetric metric = VectorMetric.Cosine, string? name = null)
public EntityTypeBuilder<T> HasVectorIndex<TKey>(Expression<Func<T, TKey>> keySelector, int dimensions,
VectorMetric metric = VectorMetric.Cosine, string? name = null)
{
Indexes.Add(new IndexBuilder<T>(keySelector, name, false, IndexType.Vector, dimensions, metric));
return this;
}
/// <summary>
/// Adds a spatial index for the specified key selector.
/// Adds a spatial index for the specified key selector.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The key selector expression.</param>
@@ -89,7 +91,7 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Sets the primary key selector for the entity type.
/// Sets the primary key selector for the entity type.
/// </summary>
/// <typeparam name="TKey">The key type.</typeparam>
/// <param name="keySelector">The primary key selector expression.</param>
@@ -102,28 +104,25 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Configures a converter for the primary key property.
/// Configures a converter for the primary key property.
/// </summary>
/// <typeparam name="TConverter">The converter type.</typeparam>
/// <returns>The current entity type builder.</returns>
public EntityTypeBuilder<T> HasConversion<TConverter>()
{
if (!string.IsNullOrEmpty(PrimaryKeyName))
{
PropertyConverters[PrimaryKeyName] = typeof(TConverter);
}
if (!string.IsNullOrEmpty(PrimaryKeyName)) PropertyConverters[PrimaryKeyName] = typeof(TConverter);
return this;
}
/// <summary>
/// Configures a specific property on the entity type.
/// Configures a specific property on the entity type.
/// </summary>
/// <typeparam name="TProperty">The property type.</typeparam>
/// <param name="propertyExpression">The property expression.</param>
/// <returns>A builder for the selected property.</returns>
public PropertyBuilder Property<TProperty>(Expression<Func<T, TProperty>> propertyExpression)
{
var propertyName = ExpressionAnalyzer.ExtractPropertyPaths(propertyExpression).FirstOrDefault();
string? propertyName = ExpressionAnalyzer.ExtractPropertyPaths(propertyExpression).FirstOrDefault();
return new PropertyBuilder(this, propertyName);
}
@@ -133,7 +132,7 @@ public class EntityTypeBuilder<T> where T : class
private readonly string? _propertyName;
/// <summary>
/// Initializes a new instance of the <see cref="PropertyBuilder"/> class.
/// Initializes a new instance of the <see cref="PropertyBuilder" /> class.
/// </summary>
/// <param name="parent">The parent entity type builder.</param>
/// <param name="propertyName">The property name.</param>
@@ -144,29 +143,23 @@ public class EntityTypeBuilder<T> where T : class
}
/// <summary>
/// Marks the configured property as value generated on add.
/// Marks the configured property as value generated on add.
/// </summary>
/// <returns>The current property builder.</returns>
public PropertyBuilder ValueGeneratedOnAdd()
{
if (_propertyName == _parent.PrimaryKeyName)
{
_parent.ValueGeneratedOnAdd = true;
}
if (_propertyName == _parent.PrimaryKeyName) _parent.ValueGeneratedOnAdd = true;
return this;
}
/// <summary>
/// Configures a converter for the configured property.
/// Configures a converter for the configured property.
/// </summary>
/// <typeparam name="TConverter">The converter type.</typeparam>
/// <returns>The current property builder.</returns>
public PropertyBuilder HasConversion<TConverter>()
{
if (!string.IsNullOrEmpty(_propertyName))
{
_parent.PropertyConverters[_propertyName] = typeof(TConverter);
}
if (!string.IsNullOrEmpty(_propertyName)) _parent.PropertyConverters[_propertyName] = typeof(TConverter);
return this;
}
}
@@ -175,37 +168,7 @@ public class EntityTypeBuilder<T> where T : class
public class IndexBuilder<T>
{
/// <summary>
/// Gets the index key selector expression.
/// </summary>
public LambdaExpression KeySelector { get; }
/// <summary>
/// Gets the configured index name.
/// </summary>
public string? Name { get; }
/// <summary>
/// Gets a value indicating whether the index is unique.
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Gets the index type.
/// </summary>
public IndexType Type { get; }
/// <summary>
/// Gets the vector dimensions.
/// </summary>
public int Dimensions { get; }
/// <summary>
/// Gets the vector metric.
/// </summary>
public VectorMetric Metric { get; }
/// <summary>
/// Initializes a new instance of the <see cref="IndexBuilder{T}"/> class.
/// Initializes a new instance of the <see cref="IndexBuilder{T}" /> class.
/// </summary>
/// <param name="keySelector">The index key selector expression.</param>
/// <param name="name">The optional index name.</param>
@@ -213,7 +176,8 @@ public class IndexBuilder<T>
/// <param name="type">The index type.</param>
/// <param name="dimensions">The vector dimensions.</param>
/// <param name="metric">The vector metric.</param>
public IndexBuilder(LambdaExpression keySelector, string? name, bool unique, IndexType type = IndexType.BTree, int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
public IndexBuilder(LambdaExpression keySelector, string? name, bool unique, IndexType type = IndexType.BTree,
int dimensions = 0, VectorMetric metric = VectorMetric.Cosine)
{
KeySelector = keySelector;
Name = name;
@@ -222,4 +186,34 @@ public class IndexBuilder<T>
Dimensions = dimensions;
Metric = metric;
}
/// <summary>
/// Gets the index key selector expression.
/// </summary>
public LambdaExpression KeySelector { get; }
/// <summary>
/// Gets the configured index name.
/// </summary>
public string? Name { get; }
/// <summary>
/// Gets a value indicating whether the index is unique.
/// </summary>
public bool IsUnique { get; }
/// <summary>
/// Gets the index type.
/// </summary>
public IndexType Type { get; }
/// <summary>
/// Gets the vector dimensions.
/// </summary>
public int Dimensions { get; }
/// <summary>
/// Gets the vector metric.
/// </summary>
public VectorMetric Metric { get; }
}
src/CBDD.Core/Metadata/ModelBuilder.cs
+9 -8
View File
@@ -1,6 +1,3 @@
using System.Linq.Expressions;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Metadata;
public class ModelBuilder
@@ -8,23 +5,27 @@ public class ModelBuilder
private readonly Dictionary<Type, object> _entityBuilders = new();
/// <summary>
/// Gets or creates the entity builder for the specified entity type.
/// Gets or creates the entity builder for the specified entity type.
/// </summary>
/// <typeparam name="T">The entity type.</typeparam>
/// <returns>The entity builder for <typeparamref name="T"/>.</returns>
/// <returns>The entity builder for <typeparamref name="T" />.</returns>
public EntityTypeBuilder<T> Entity<T>() where T : class
{
if (!_entityBuilders.TryGetValue(typeof(T), out var builder))
if (!_entityBuilders.TryGetValue(typeof(T), out object? builder))
{
builder = new EntityTypeBuilder<T>();
_entityBuilders[typeof(T)] = builder;
}
return (EntityTypeBuilder<T>)builder;
}
/// <summary>
/// Gets all registered entity builders.
/// Gets all registered entity builders.
/// </summary>
/// <returns>A read-only dictionary of entity builders keyed by entity type.</returns>
public IReadOnlyDictionary<Type, object> GetEntityBuilders() => _entityBuilders;
public IReadOnlyDictionary<Type, object> GetEntityBuilders()
{
return _entityBuilders;
}
}
src/CBDD.Core/Metadata/ValueConverter.cs
+4 -4
View File
@@ -1,19 +1,19 @@
namespace ZB.MOM.WW.CBDD.Core.Metadata;
/// <summary>
/// Defines a bidirectional conversion between a model type (e.g. ValueObject)
/// and a provider type supported by the storage engine (e.g. string, int, Guid, ObjectId).
/// Defines a bidirectional conversion between a model type (e.g. ValueObject)
/// and a provider type supported by the storage engine (e.g. string, int, Guid, ObjectId).
/// </summary>
public abstract class ValueConverter<TModel, TProvider>
{
/// <summary>
/// Converts the model value to the provider value.
/// Converts the model value to the provider value.
/// </summary>
/// <param name="model">The model value to convert.</param>
public abstract TProvider ConvertToProvider(TModel model);
/// <summary>
/// Converts the provider value back to the model value.
/// Converts the provider value back to the model value.
/// </summary>
/// <param name="provider">The provider value to convert.</param>
public abstract TModel ConvertFromProvider(TProvider provider);
src/CBDD.Core/Query/BTreeExpressionVisitor.cs
+5 -4
View File
@@ -7,15 +7,17 @@ internal class BTreeExpressionVisitor : ExpressionVisitor
private readonly QueryModel _model = new();
/// <summary>
/// Gets the query model built while visiting an expression tree.
/// Gets the query model built while visiting an expression tree.
/// </summary>
public QueryModel GetModel() => _model;
public QueryModel GetModel()
{
return _model;
}
/// <inheritdoc />
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Queryable))
{
switch (node.Method.Name)
{
case "Where":
@@ -35,7 +37,6 @@ internal class BTreeExpressionVisitor : ExpressionVisitor
VisitSkip(node);
break;
}
}
return base.VisitMethodCall(node);
}
src/CBDD.Core/Query/BTreeQueryProvider.cs
+15 -35
View File
@@ -1,6 +1,6 @@
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using static ZB.MOM.WW.CBDD.Core.Query.IndexOptimizer;
@@ -11,7 +11,7 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
private readonly DocumentCollection<TId, T> _collection;
/// <summary>
/// Initializes a new instance of the <see cref="BTreeQueryProvider{TId, T}"/> class.
/// Initializes a new instance of the <see cref="BTreeQueryProvider{TId, T}" /> class.
/// </summary>
/// <param name="collection">The backing document collection.</param>
public BTreeQueryProvider(DocumentCollection<TId, T> collection)
@@ -20,18 +20,17 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
}
/// <summary>
/// Creates a query from the specified expression.
/// Creates a query from the specified expression.
/// </summary>
/// <param name="expression">The query expression.</param>
/// <returns>An <see cref="IQueryable"/> representing the query.</returns>
/// <returns>An <see cref="IQueryable" /> representing the query.</returns>
public IQueryable CreateQuery(Expression expression)
{
var elementType = expression.Type.GetGenericArguments()[0];
try
{
return (IQueryable)Activator.CreateInstance(
typeof(BTreeQueryable<>).MakeGenericType(elementType),
new object[] { this, expression })!;
typeof(BTreeQueryable<>).MakeGenericType(elementType), this, expression)!;
}
catch (TargetInvocationException ex)
{
@@ -40,18 +39,18 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
}
/// <summary>
/// Creates a strongly typed query from the specified expression.
/// Creates a strongly typed query from the specified expression.
/// </summary>
/// <typeparam name="TElement">The element type of the query.</typeparam>
/// <param name="expression">The query expression.</param>
/// <returns>An <see cref="IQueryable{T}"/> representing the query.</returns>
/// <returns>An <see cref="IQueryable{T}" /> representing the query.</returns>
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
return new BTreeQueryable<TElement>(this, expression);
}
/// <summary>
/// Executes a query expression.
/// Executes a query expression.
/// </summary>
/// <param name="expression">The query expression.</param>
/// <returns>The query result.</returns>
@@ -61,7 +60,7 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
}
/// <summary>
/// Executes a query expression and returns a strongly typed result.
/// Executes a query expression and returns a strongly typed result.
/// </summary>
/// <typeparam name="TResult">The result type.</typeparam>
/// <param name="expression">The query expression.</param>
@@ -80,45 +79,30 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
IEnumerable<T> sourceData = null!;
// A. Try Index Optimization (Only if Where clause exists)
var indexOpt = IndexOptimizer.TryOptimize<T>(model, _collection.GetIndexes());
var indexOpt = TryOptimize<T>(model, _collection.GetIndexes());
if (indexOpt != null)
{
if (indexOpt.IsVectorSearch)
{
sourceData = _collection.VectorSearch(indexOpt.IndexName, indexOpt.VectorQuery!, indexOpt.K);
}
else if (indexOpt.IsSpatialSearch)
{
sourceData = indexOpt.SpatialType == SpatialQueryType.Near
? _collection.Near(indexOpt.IndexName, indexOpt.SpatialPoint, indexOpt.RadiusKm)
: _collection.Within(indexOpt.IndexName, indexOpt.SpatialMin, indexOpt.SpatialMax);
}
else
{
sourceData = _collection.QueryIndex(indexOpt.IndexName, indexOpt.MinValue, indexOpt.MaxValue);
}
}
// B. Try Scan Optimization (if no index used)
if (sourceData == null)
{
Func<ZB.MOM.WW.CBDD.Bson.BsonSpanReader, bool>? bsonPredicate = null;
if (model.WhereClause != null)
{
bsonPredicate = BsonExpressionEvaluator.TryCompile<T>(model.WhereClause);
}
Func<BsonSpanReader, bool>? bsonPredicate = null;
if (model.WhereClause != null) bsonPredicate = BsonExpressionEvaluator.TryCompile<T>(model.WhereClause);
if (bsonPredicate != null)
{
sourceData = _collection.Scan(bsonPredicate);
}
if (bsonPredicate != null) sourceData = _collection.Scan(bsonPredicate);
}
// C. Fallback to Full Scan
if (sourceData == null)
{
sourceData = _collection.FindAll();
}
if (sourceData == null) sourceData = _collection.FindAll();
// 3. Rewrite Expression Tree to use Enumerable
// Replace the "Root" IQueryable with our sourceData IEnumerable
@@ -140,10 +124,8 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
// We need to turn it into a Func<TResult> and invoke it.
if (rewrittenExpression.Type != typeof(TResult))
{
// If TResult is object (non-generic Execute), we need to cast
rewrittenExpression = Expression.Convert(rewrittenExpression, typeof(TResult));
}
var lambda = Expression.Lambda<Func<TResult>>(rewrittenExpression);
var compiled = lambda.Compile();
@@ -153,7 +135,7 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
private class RootFinder : ExpressionVisitor
{
/// <summary>
/// Gets the root queryable found in the expression tree.
/// Gets the root queryable found in the expression tree.
/// </summary>
public IQueryable? Root { get; private set; }
@@ -162,11 +144,9 @@ public class BTreeQueryProvider<TId, T> : IQueryProvider where T : class
{
// If we found a Queryable, that's our root source
if (Root == null && node.Value is IQueryable q)
{
// We typically want the "base" queryable (the BTreeQueryable instance)
// In a chain like Coll.Where.Select, the root is Coll.
Root = q;
}
return base.VisitConstant(node);
}
}
src/CBDD.Core/Query/BTreeQueryable.cs
+5 -6
View File
@@ -1,5 +1,4 @@
using System.Collections;
using System.Linq;
using System.Linq.Expressions;
namespace ZB.MOM.WW.CBDD.Core.Query;
@@ -7,7 +6,7 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal class BTreeQueryable<T> : IOrderedQueryable<T>
{
/// <summary>
/// Initializes a new queryable wrapper for the specified provider and expression.
/// Initializes a new queryable wrapper for the specified provider and expression.
/// </summary>
/// <param name="provider">The query provider.</param>
/// <param name="expression">The expression tree.</param>
@@ -18,7 +17,7 @@ internal class BTreeQueryable<T> : IOrderedQueryable<T>
}
/// <summary>
/// Initializes a new queryable wrapper for the specified provider.
/// Initializes a new queryable wrapper for the specified provider.
/// </summary>
/// <param name="provider">The query provider.</param>
public BTreeQueryable(IQueryProvider provider)
@@ -28,17 +27,17 @@ internal class BTreeQueryable<T> : IOrderedQueryable<T>
}
/// <summary>
/// Gets the element type returned by this query.
/// Gets the element type returned by this query.
/// </summary>
public Type ElementType => typeof(T);
/// <summary>
/// Gets the expression tree associated with this query.
/// Gets the expression tree associated with this query.
/// </summary>
public Expression Expression { get; }
/// <summary>
/// Gets the query provider for this query.
/// Gets the query provider for this query.
/// </summary>
public IQueryProvider Provider { get; }
src/CBDD.Core/Query/BsonExpressionEvaluator.cs
+21 -22
View File
@@ -6,11 +6,11 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal static class BsonExpressionEvaluator
{
/// <summary>
/// Attempts to compile a LINQ predicate expression into a BSON reader predicate.
/// Attempts to compile a LINQ predicate expression into a BSON reader predicate.
/// </summary>
/// <typeparam name="T">The entity type of the original expression.</typeparam>
/// <param name="expression">The lambda expression to compile.</param>
/// <returns>A compiled predicate when supported; otherwise, <see langword="null"/>.</returns>
/// <returns>A compiled predicate when supported; otherwise, <see langword="null" />.</returns>
public static Func<BsonSpanReader, bool>? TryCompile<T>(LambdaExpression expression)
{
// Simple optimization for: x => x.Prop op Constant
@@ -29,12 +29,11 @@ internal static class BsonExpressionEvaluator
}
if (left is MemberExpression member && right is ConstantExpression constant)
{
// Check if member is property of parameter
if (member.Expression == expression.Parameters[0])
{
var propertyName = member.Member.Name.ToLowerInvariant();
var value = constant.Value;
string propertyName = member.Member.Name.ToLowerInvariant();
object? value = constant.Value;
// Handle Id mapping?
// If property is "id", Bson field is "_id"
@@ -42,22 +41,25 @@ internal static class BsonExpressionEvaluator
return CreatePredicate(propertyName, value, nodeType);
}
}
}
return null;
}
private static ExpressionType Flip(ExpressionType type) => type switch
private static ExpressionType Flip(ExpressionType type)
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
return type switch
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}
private static Func<BsonSpanReader, bool>? CreatePredicate(string propertyName, object? targetValue, ExpressionType op)
private static Func<BsonSpanReader, bool>? CreatePredicate(string propertyName, object? targetValue,
ExpressionType op)
{
// We need to return a delegate that searches for propertyName in BsonSpanReader and compares
@@ -71,13 +73,11 @@ internal static class BsonExpressionEvaluator
var type = reader.ReadBsonType();
if (type == 0) break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
if (name == propertyName)
{
// Found -> read value and compare
return Compare(ref reader, type, targetValue, op);
}
reader.SkipValue(type);
}
@@ -86,6 +86,7 @@ internal static class BsonExpressionEvaluator
{
return false;
}
return false; // Not found
};
}
@@ -97,9 +98,8 @@ internal static class BsonExpressionEvaluator
if (type == BsonType.Int32)
{
var val = reader.ReadInt32();
int val = reader.ReadInt32();
if (target is int targetInt)
{
return op switch
{
ExpressionType.Equal => val == targetInt,
@@ -110,14 +110,13 @@ internal static class BsonExpressionEvaluator
ExpressionType.LessThanOrEqual => val <= targetInt,
_ => false
};
}
}
else if (type == BsonType.String)
{
var val = reader.ReadString();
string val = reader.ReadString();
if (target is string targetStr)
{
var cmp = string.Compare(val, targetStr, StringComparison.Ordinal);
int cmp = string.Compare(val, targetStr, StringComparison.Ordinal);
return op switch
{
ExpressionType.Equal => cmp == 0,
src/CBDD.Core/Query/EnumerableRewriter.cs
+5 -10
View File
@@ -1,6 +1,3 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
@@ -12,7 +9,7 @@ internal class EnumerableRewriter : ExpressionVisitor
private readonly object _target;
/// <summary>
/// Initializes a new instance of the <see cref="EnumerableRewriter"/> class.
/// Initializes a new instance of the <see cref="EnumerableRewriter" /> class.
/// </summary>
/// <param name="source">The original queryable source to replace.</param>
/// <param name="target">The target enumerable-backed object.</param>
@@ -26,10 +23,7 @@ internal class EnumerableRewriter : ExpressionVisitor
protected override Expression VisitConstant(ConstantExpression node)
{
// Replace the IQueryable source with the materialized IEnumerable
if (node.Value == _source)
{
return Expression.Constant(_target);
}
if (node.Value == _source) return Expression.Constant(_target);
return base.VisitConstant(node);
}
@@ -38,11 +32,11 @@ internal class EnumerableRewriter : ExpressionVisitor
{
if (node.Method.DeclaringType == typeof(Queryable))
{
var methodName = node.Method.Name;
string methodName = node.Method.Name;
var typeArgs = node.Method.GetGenericArguments();
var args = new Expression[node.Arguments.Count];
for (int i = 0; i < node.Arguments.Count; i++)
for (var i = 0; i < node.Arguments.Count; i++)
{
var arg = Visit(node.Arguments[i]);
@@ -52,6 +46,7 @@ internal class EnumerableRewriter : ExpressionVisitor
var lambda = (LambdaExpression)quote.Operand;
arg = Expression.Constant(lambda.Compile());
}
args[i] = arg;
}
src/CBDD.Core/Query/IndexOptimizer.cs
+108 -108
View File
@@ -5,86 +5,19 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal static class IndexOptimizer
{
/// <summary>
/// Represents the selected index and bounds for an optimized query.
/// </summary>
public class OptimizationResult
public enum SpatialQueryType
{
/// <summary>
/// Gets or sets the selected index name.
/// </summary>
public string IndexName { get; set; } = "";
/// <summary>
/// Gets or sets the minimum bound value.
/// </summary>
public object? MinValue { get; set; }
/// <summary>
/// Gets or sets the maximum bound value.
/// </summary>
public object? MaxValue { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses a range.
/// </summary>
public bool IsRange { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses vector search.
/// </summary>
public bool IsVectorSearch { get; set; }
/// <summary>
/// Gets or sets the vector query values.
/// </summary>
public float[]? VectorQuery { get; set; }
/// <summary>
/// Gets or sets the number of nearest neighbors for vector search.
/// </summary>
public int K { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses spatial search.
/// </summary>
public bool IsSpatialSearch { get; set; }
/// <summary>
/// Gets or sets the center point for near queries.
/// </summary>
public (double Latitude, double Longitude) SpatialPoint { get; set; }
/// <summary>
/// Gets or sets the search radius in kilometers.
/// </summary>
public double RadiusKm { get; set; }
/// <summary>
/// Gets or sets the minimum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMin { get; set; }
/// <summary>
/// Gets or sets the maximum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMax { get; set; }
/// <summary>
/// Gets or sets the spatial query type.
/// </summary>
public SpatialQueryType SpatialType { get; set; }
Near,
Within
}
public enum SpatialQueryType { Near, Within }
/// <summary>
/// Attempts to optimize a query model using available indexes.
/// Attempts to optimize a query model using available indexes.
/// </summary>
/// <typeparam name="T">The document type.</typeparam>
/// <param name="model">The query model.</param>
/// <param name="indexes">The available collection indexes.</param>
/// <returns>An optimization result when optimization is possible; otherwise, <see langword="null"/>.</returns>
/// <returns>An optimization result when optimization is possible; otherwise, <see langword="null" />.</returns>
public static OptimizationResult? TryOptimize<T>(QueryModel model, IEnumerable<CollectionIndexInfo> indexes)
{
if (model.WhereClause == null) return null;
@@ -92,7 +25,8 @@ internal static class IndexOptimizer
return OptimizeExpression(model.WhereClause.Body, model.WhereClause.Parameters[0], indexes);
}
private static OptimizationResult? OptimizeExpression(Expression expression, ParameterExpression parameter, IEnumerable<CollectionIndexInfo> indexes)
private static OptimizationResult? OptimizeExpression(Expression expression, ParameterExpression parameter,
IEnumerable<CollectionIndexInfo> indexes)
{
// ... (Existing AndAlso logic remains the same) ...
if (expression is BinaryExpression binary && binary.NodeType == ExpressionType.AndAlso)
@@ -101,7 +35,6 @@ internal static class IndexOptimizer
var right = OptimizeExpression(binary.Right, parameter, indexes);
if (left != null && right != null && left.IndexName == right.IndexName)
{
return new OptimizationResult
{
IndexName = left.IndexName,
@@ -109,12 +42,11 @@ internal static class IndexOptimizer
MaxValue = left.MaxValue ?? right.MaxValue,
IsRange = true
};
}
return left ?? right;
}
// Handle Simple Binary Predicates
var (propertyName, value, op) = ParseSimplePredicate(expression, parameter);
(string? propertyName, object? value, var op) = ParseSimplePredicate(expression, parameter);
if (propertyName != null)
{
var index = indexes.FirstOrDefault(i => Matches(i, propertyName));
@@ -141,19 +73,21 @@ internal static class IndexOptimizer
result.IsRange = true;
break;
}
return result;
}
}
// Handle StartsWith
if (expression is MethodCallExpression call && call.Method.Name == "StartsWith" && call.Object is MemberExpression member)
{
if (member.Expression == parameter && call.Arguments[0] is ConstantExpression constant && constant.Value is string prefix)
if (expression is MethodCallExpression call && call.Method.Name == "StartsWith" &&
call.Object is MemberExpression member)
if (member.Expression == parameter && call.Arguments[0] is ConstantExpression constant &&
constant.Value is string prefix)
{
var index = indexes.FirstOrDefault(i => Matches(i, member.Member.Name));
if (index != null && index.Type == IndexType.BTree)
{
var nextPrefix = IncrementPrefix(prefix);
string nextPrefix = IncrementPrefix(prefix);
return new OptimizationResult
{
IndexName = index.Name,
@@ -163,20 +97,19 @@ internal static class IndexOptimizer
};
}
}
}
// Handle Method Calls (VectorSearch, Near, Within)
if (expression is MethodCallExpression mcall)
{
// VectorSearch(this float[] vector, float[] query, int k)
if (mcall.Method.Name == "VectorSearch" && mcall.Arguments[0] is MemberExpression vMember && vMember.Expression == parameter)
if (mcall.Method.Name == "VectorSearch" && mcall.Arguments[0] is MemberExpression vMember &&
vMember.Expression == parameter)
{
var query = EvaluateExpression<float[]>(mcall.Arguments[1]);
float[] query = EvaluateExpression<float[]>(mcall.Arguments[1]);
var k = EvaluateExpression<int>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Vector && Matches(i, vMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
@@ -184,18 +117,17 @@ internal static class IndexOptimizer
VectorQuery = query,
K = k
};
}
}
// Near(this (double, double) point, (double, double) center, double radiusKm)
if (mcall.Method.Name == "Near" && mcall.Arguments[0] is MemberExpression nMember && nMember.Expression == parameter)
if (mcall.Method.Name == "Near" && mcall.Arguments[0] is MemberExpression nMember &&
nMember.Expression == parameter)
{
var center = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var radius = EvaluateExpression<double>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, nMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
@@ -204,18 +136,17 @@ internal static class IndexOptimizer
SpatialPoint = center,
RadiusKm = radius
};
}
}
// Within(this (double, double) point, (double, double) min, (double, double) max)
if (mcall.Method.Name == "Within" && mcall.Arguments[0] is MemberExpression wMember && wMember.Expression == parameter)
if (mcall.Method.Name == "Within" && mcall.Arguments[0] is MemberExpression wMember &&
wMember.Expression == parameter)
{
var min = EvaluateExpression<(double, double)>(mcall.Arguments[1]);
var max = EvaluateExpression<(double, double)>(mcall.Arguments[2]);
var index = indexes.FirstOrDefault(i => i.Type == IndexType.Spatial && Matches(i, wMember.Member.Name));
if (index != null)
{
return new OptimizationResult
{
IndexName = index.Name,
@@ -224,7 +155,6 @@ internal static class IndexOptimizer
SpatialMin = min,
SpatialMax = max
};
}
}
}
@@ -241,10 +171,7 @@ internal static class IndexOptimizer
private static T EvaluateExpression<T>(Expression expression)
{
if (expression is ConstantExpression constant)
{
return (T)constant.Value!;
}
if (expression is ConstantExpression constant) return (T)constant.Value!;
// Evaluate more complex expressions (closures, properties, etc.)
var lambda = Expression.Lambda(expression);
@@ -258,7 +185,8 @@ internal static class IndexOptimizer
return index.PropertyPaths[0].Equals(propertyName, StringComparison.OrdinalIgnoreCase);
}
private static (string? propertyName, object? value, ExpressionType op) ParseSimplePredicate(Expression expression, ParameterExpression parameter)
private static (string? propertyName, object? value, ExpressionType op) ParseSimplePredicate(Expression expression,
ParameterExpression parameter)
{
if (expression is BinaryExpression binary)
{
@@ -273,27 +201,99 @@ internal static class IndexOptimizer
}
if (left is MemberExpression member && right is ConstantExpression constant)
{
if (member.Expression == parameter)
return (member.Member.Name, constant.Value, nodeType);
}
// Handle Convert
if (left is UnaryExpression unary && unary.Operand is MemberExpression member2 && right is ConstantExpression constant2)
{
if (left is UnaryExpression unary && unary.Operand is MemberExpression member2 &&
right is ConstantExpression constant2)
if (member2.Expression == parameter)
return (member2.Member.Name, constant2.Value, nodeType);
}
}
return (null, null, ExpressionType.Default);
}
private static ExpressionType Flip(ExpressionType type) => type switch
private static ExpressionType Flip(ExpressionType type)
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
return type switch
{
ExpressionType.GreaterThan => ExpressionType.LessThan,
ExpressionType.LessThan => ExpressionType.GreaterThan,
ExpressionType.GreaterThanOrEqual => ExpressionType.LessThanOrEqual,
ExpressionType.LessThanOrEqual => ExpressionType.GreaterThanOrEqual,
_ => type
};
}
/// <summary>
/// Represents the selected index and bounds for an optimized query.
/// </summary>
public class OptimizationResult
{
/// <summary>
/// Gets or sets the selected index name.
/// </summary>
public string IndexName { get; set; } = "";
/// <summary>
/// Gets or sets the minimum bound value.
/// </summary>
public object? MinValue { get; set; }
/// <summary>
/// Gets or sets the maximum bound value.
/// </summary>
public object? MaxValue { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses a range.
/// </summary>
public bool IsRange { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses vector search.
/// </summary>
public bool IsVectorSearch { get; set; }
/// <summary>
/// Gets or sets the vector query values.
/// </summary>
public float[]? VectorQuery { get; set; }
/// <summary>
/// Gets or sets the number of nearest neighbors for vector search.
/// </summary>
public int K { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the query uses spatial search.
/// </summary>
public bool IsSpatialSearch { get; set; }
/// <summary>
/// Gets or sets the center point for near queries.
/// </summary>
public (double Latitude, double Longitude) SpatialPoint { get; set; }
/// <summary>
/// Gets or sets the search radius in kilometers.
/// </summary>
public double RadiusKm { get; set; }
/// <summary>
/// Gets or sets the minimum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMin { get; set; }
/// <summary>
/// Gets or sets the maximum point for within queries.
/// </summary>
public (double Latitude, double Longitude) SpatialMax { get; set; }
/// <summary>
/// Gets or sets the spatial query type.
/// </summary>
public SpatialQueryType SpatialType { get; set; }
}
}
src/CBDD.Core/Query/QueryModel.cs
+6 -6
View File
@@ -5,32 +5,32 @@ namespace ZB.MOM.WW.CBDD.Core.Query;
internal class QueryModel
{
/// <summary>
/// Gets or sets the filter expression.
/// Gets or sets the filter expression.
/// </summary>
public LambdaExpression? WhereClause { get; set; }
/// <summary>
/// Gets or sets the projection expression.
/// Gets or sets the projection expression.
/// </summary>
public LambdaExpression? SelectClause { get; set; }
/// <summary>
/// Gets or sets the ordering expression.
/// Gets or sets the ordering expression.
/// </summary>
public LambdaExpression? OrderByClause { get; set; }
/// <summary>
/// Gets or sets the maximum number of results to return.
/// Gets or sets the maximum number of results to return.
/// </summary>
public int? Take { get; set; }
/// <summary>
/// Gets or sets the number of results to skip.
/// Gets or sets the number of results to skip.
/// </summary>
public int? Skip { get; set; }
/// <summary>
/// Gets or sets a value indicating whether ordering is descending.
/// Gets or sets a value indicating whether ordering is descending.
/// </summary>
public bool OrderDescending { get; set; }
}
src/CBDD.Core/Storage/DictionaryPage.cs
+65 -69
View File
@@ -1,13 +1,13 @@
using System.Runtime.InteropServices;
using System.Buffers;
using System.Buffers.Binary;
using System.Text;
using ZB.MOM.WW.CBDD.Core;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page for storing dictionary entries (Key -> Value map).
/// Uses a sorted list of keys for binary search within the page.
/// Supports chaining via PageHeader.NextPageId for dictionaries larger than one page.
/// Page for storing dictionary entries (Key -> Value map).
/// Uses a sorted list of keys for binary search within the page.
/// Supports chaining via PageHeader.NextPageId for dictionaries larger than one page.
/// </summary>
public struct DictionaryPage
{
@@ -25,12 +25,12 @@ public struct DictionaryPage
private const int OffsetsStart = 36;
/// <summary>
/// Values 0-100 are reserved for internal system keys (e.g. _id, _v).
/// Values 0-100 are reserved for internal system keys (e.g. _id, _v).
/// </summary>
public const ushort ReservedValuesEnd = 100;
/// <summary>
/// Initialize a new dictionary page
/// Initialize a new dictionary page
/// </summary>
/// <param name="page">The page buffer to initialize.</param>
/// <param name="pageId">The page identifier.</param>
@@ -49,43 +49,40 @@ public struct DictionaryPage
header.WriteTo(page);
// 2. Initialize Counts
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), 0);
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), (ushort)page.Length);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), 0);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), (ushort)page.Length);
}
/// <summary>
/// Inserts a key-value pair into the page.
/// Returns false if there is not enough space.
/// Inserts a key-value pair into the page.
/// Returns false if there is not enough space.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="key">The dictionary key.</param>
/// <param name="value">The value mapped to the key.</param>
/// <returns><see langword="true"/> if the entry was inserted; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the entry was inserted; otherwise, <see langword="false" />.</returns>
public static bool Insert(Span<byte> page, string key, ushort value)
{
var keyByteCount = Encoding.UTF8.GetByteCount(key);
int keyByteCount = Encoding.UTF8.GetByteCount(key);
if (keyByteCount > 255) throw new ArgumentException("Key length must be <= 255 bytes");
// Entry Size: KeyLen(1) + Key(N) + Value(2)
var entrySize = 1 + keyByteCount + 2;
var requiredSpace = entrySize + 2; // +2 for Offset entry
int entrySize = 1 + keyByteCount + 2;
int requiredSpace = entrySize + 2; // +2 for Offset entry
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
var freeSpaceEnd = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(FreeSpaceEndOffset));
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort freeSpaceEnd = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(FreeSpaceEndOffset));
var offsetsEnd = OffsetsStart + (count * 2);
var freeSpace = freeSpaceEnd - offsetsEnd;
int offsetsEnd = OffsetsStart + count * 2;
int freeSpace = freeSpaceEnd - offsetsEnd;
if (freeSpace < requiredSpace)
{
return false; // Page Full
}
if (freeSpace < requiredSpace) return false; // Page Full
// 1. Prepare Data
var insertionOffset = (ushort)(freeSpaceEnd - entrySize);
page[insertionOffset] = (byte)keyByteCount; // Write Key Length
Encoding.UTF8.GetBytes(key, page.Slice(insertionOffset + 1, keyByteCount)); // Write Key
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(insertionOffset + 1 + keyByteCount), value); // Write Value
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(insertionOffset + 1 + keyByteCount), value); // Write Value
// 2. Insert Offset into Sorted List
// Find insert Index using spans
@@ -95,57 +92,57 @@ public struct DictionaryPage
// Shift offsets if needed
if (insertIndex < count)
{
var src = page.Slice(OffsetsStart + (insertIndex * 2), (count - insertIndex) * 2);
var dest = page.Slice(OffsetsStart + ((insertIndex + 1) * 2));
var src = page.Slice(OffsetsStart + insertIndex * 2, (count - insertIndex) * 2);
var dest = page.Slice(OffsetsStart + (insertIndex + 1) * 2);
src.CopyTo(dest);
}
// Write new offset
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(OffsetsStart + (insertIndex * 2)), insertionOffset);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(OffsetsStart + insertIndex * 2), insertionOffset);
// 3. Update Metadata
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), (ushort)(count + 1));
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), insertionOffset);
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(CountOffset), (ushort)(count + 1));
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(FreeSpaceEndOffset), insertionOffset);
// Update FreeBytes in header (approximate)
var pageHeader = PageHeader.ReadFrom(page);
pageHeader.FreeBytes = (ushort)(insertionOffset - (OffsetsStart + ((count + 1) * 2)));
pageHeader.FreeBytes = (ushort)(insertionOffset - (OffsetsStart + (count + 1) * 2));
pageHeader.WriteTo(page);
return true;
}
/// <summary>
/// Tries to find a value for the given key in THIS page.
/// Tries to find a value for the given key in THIS page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="keyBytes">The UTF-8 encoded key bytes.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <returns><see langword="true"/> if the key was found; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if the key was found; otherwise, <see langword="false" />.</returns>
public static bool TryFind(ReadOnlySpan<byte> page, ReadOnlySpan<byte> keyBytes, out ushort value)
{
value = 0;
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
if (count == 0) return false;
// Binary Search
int low = 0;
var low = 0;
int high = count - 1;
while (low <= high)
{
int mid = low + (high - low) / 2;
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (mid * 2)));
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + mid * 2));
// Read Key at Offset
var keyLen = page[offset];
byte keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
int comparison = entryKeySpan.SequenceCompareTo(keyBytes);
if (comparison == 0)
{
value = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
value = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
return true;
}
@@ -159,22 +156,23 @@ public struct DictionaryPage
}
/// <summary>
/// Tries to find a value for the given key across a chain of DictionaryPages.
/// Tries to find a value for the given key across a chain of DictionaryPages.
/// </summary>
/// <param name="storage">The storage engine used to read pages.</param>
/// <param name="startPageId">The first page in the dictionary chain.</param>
/// <param name="key">The key to search for.</param>
/// <param name="value">When this method returns, contains the found value.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns><see langword="true"/> if the key was found; otherwise, <see langword="false"/>.</returns>
public static bool TryFindGlobal(StorageEngine storage, uint startPageId, string key, out ushort value, ulong? transactionId = null)
/// <returns><see langword="true" /> if the key was found; otherwise, <see langword="false" />.</returns>
public static bool TryFindGlobal(StorageEngine storage, uint startPageId, string key, out ushort value,
ulong? transactionId = null)
{
var keyByteCount = Encoding.UTF8.GetByteCount(key);
Span<byte> keyBytes = keyByteCount <= 256 ? stackalloc byte[keyByteCount] : new byte[keyByteCount];
int keyByteCount = Encoding.UTF8.GetByteCount(key);
var keyBytes = keyByteCount <= 256 ? stackalloc byte[keyByteCount] : new byte[keyByteCount];
Encoding.UTF8.GetBytes(key, keyBytes);
var pageId = startPageId;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(storage.PageSize);
uint pageId = startPageId;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(storage.PageSize);
try
{
while (pageId != 0)
@@ -183,10 +181,7 @@ public struct DictionaryPage
storage.ReadPage(pageId, transactionId, pageBuffer);
// TryFind in this page
if (TryFind(pageBuffer, keyBytes, out value))
{
return true;
}
if (TryFind(pageBuffer, keyBytes, out value)) return true;
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
@@ -195,7 +190,7 @@ public struct DictionaryPage
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
value = 0;
@@ -204,15 +199,15 @@ public struct DictionaryPage
private static int FindInsertIndex(ReadOnlySpan<byte> page, int count, ReadOnlySpan<byte> keyBytes)
{
int low = 0;
var low = 0;
int high = count - 1;
while (low <= high)
{
int mid = low + (high - low) / 2;
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (mid * 2)));
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + mid * 2));
var keyLen = page[offset];
byte keyLen = page[offset];
var entryKeySpan = page.Slice(offset + 1, keyLen);
int comparison = entryKeySpan.SequenceCompareTo(keyBytes);
@@ -223,40 +218,44 @@ public struct DictionaryPage
else
high = mid - 1;
}
return low;
}
/// <summary>
/// Gets all entries in the page (for debugging/dumping)
/// Gets all entries in the page (for debugging/dumping)
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>All key-value pairs in the page.</returns>
public static IEnumerable<(string Key, ushort Value)> GetAll(ReadOnlySpan<byte> page)
{
var count = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
ushort count = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(CountOffset));
var list = new List<(string Key, ushort Value)>();
for (int i = 0; i < count; i++)
for (var i = 0; i < count; i++)
{
var offset = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + (i * 2)));
var keyLen = page[offset];
var keyStr = Encoding.UTF8.GetString(page.Slice(offset + 1, keyLen));
var val = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
ushort offset = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(OffsetsStart + i * 2));
byte keyLen = page[offset];
string keyStr = Encoding.UTF8.GetString(page.Slice(offset + 1, keyLen));
ushort val = BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(offset + 1 + keyLen));
list.Add((keyStr, val));
}
return list;
}
/// <summary>
/// Retrieves all key-value pairs across a chain of DictionaryPages.
/// Used for rebuilding the in-memory cache.
/// Retrieves all key-value pairs across a chain of DictionaryPages.
/// Used for rebuilding the in-memory cache.
/// </summary>
/// <param name="storage">The storage engine used to read pages.</param>
/// <param name="startPageId">The first page in the dictionary chain.</param>
/// <param name="transactionId">Optional transaction identifier for isolated reads.</param>
/// <returns>All key-value pairs across the page chain.</returns>
public static IEnumerable<(string Key, ushort Value)> FindAllGlobal(StorageEngine storage, uint startPageId, ulong? transactionId = null)
public static IEnumerable<(string Key, ushort Value)> FindAllGlobal(StorageEngine storage, uint startPageId,
ulong? transactionId = null)
{
var pageId = startPageId;
var pageBuffer = System.Buffers.ArrayPool<byte>.Shared.Rent(storage.PageSize);
uint pageId = startPageId;
byte[] pageBuffer = ArrayPool<byte>.Shared.Rent(storage.PageSize);
try
{
while (pageId != 0)
@@ -265,10 +264,7 @@ public struct DictionaryPage
storage.ReadPage(pageId, transactionId, pageBuffer);
// Get all entries in this page
foreach (var entry in GetAll(pageBuffer))
{
yield return entry;
}
foreach (var entry in GetAll(pageBuffer)) yield return entry;
// Move to next page
var header = PageHeader.ReadFrom(pageBuffer);
@@ -277,7 +273,7 @@ public struct DictionaryPage
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(pageBuffer);
ArrayPool<byte>.Shared.Return(pageBuffer);
}
}
}
src/CBDD.Core/Storage/IIndexStorage.cs
+12 -7
View File
@@ -1,39 +1,44 @@
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Narrow storage port for index structures (page operations + allocation only).
/// Narrow storage port for index structures (page operations + allocation only).
/// </summary>
internal interface IIndexStorage
{
/// <summary>
/// Gets or sets the PageSize.
/// Gets or sets the PageSize.
/// </summary>
int PageSize { get; }
/// <summary>
/// Executes AllocatePage.
/// Executes AllocatePage.
/// </summary>
uint AllocatePage();
/// <summary>
/// Executes FreePage.
/// Executes FreePage.
/// </summary>
/// <param name="pageId">The page identifier.</param>
void FreePage(uint pageId);
/// <summary>
/// Executes ReadPage.
/// Executes ReadPage.
/// </summary>
/// <param name="pageId">The page identifier.</param>
/// <param name="transactionId">The optional transaction identifier.</param>
/// <param name="destination">The destination buffer.</param>
void ReadPage(uint pageId, ulong? transactionId, Span<byte> destination);
/// <summary>
/// Executes WritePage.
/// Executes WritePage.
/// </summary>
/// <param name="pageId">The page identifier.</param>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="data">The source page data.</param>
void WritePage(uint pageId, ulong transactionId, ReadOnlySpan<byte> data);
/// <summary>
/// Executes WritePageImmediate.
/// Executes WritePageImmediate.
/// </summary>
/// <param name="pageId">The page identifier.</param>
/// <param name="data">The source page data.</param>
src/CBDD.Core/Storage/IStorageEngine.cs
+21 -20
View File
@@ -8,110 +8,111 @@ using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Storage port used by collection/index orchestration to avoid concrete engine coupling.
/// Storage port used by collection/index orchestration to avoid concrete engine coupling.
/// </summary>
internal interface IStorageEngine : IIndexStorage, IDisposable
{
/// <summary>
/// Gets the current page count.
/// Gets the current page count.
/// </summary>
uint PageCount { get; }
/// <summary>
/// Gets the active change stream dispatcher.
/// Gets the active change stream dispatcher.
/// </summary>
ChangeStreamDispatcher? Cdc { get; }
/// <summary>
/// Gets compression options used by the storage engine.
/// Gets compression options used by the storage engine.
/// </summary>
CompressionOptions CompressionOptions { get; }
/// <summary>
/// Gets the compression service.
/// Gets the compression service.
/// </summary>
CompressionService CompressionService { get; }
/// <summary>
/// Gets compression telemetry for the storage engine.
/// Gets compression telemetry for the storage engine.
/// </summary>
CompressionTelemetry CompressionTelemetry { get; }
/// <summary>
/// Determines whether a page is locked.
/// Determines whether a page is locked.
/// </summary>
/// <param name="pageId">The page identifier to inspect.</param>
/// <param name="excludingTxId">A transaction identifier to exclude from lock checks.</param>
bool IsPageLocked(uint pageId, ulong excludingTxId);
/// <summary>
/// Registers the change stream dispatcher.
/// Registers the change stream dispatcher.
/// </summary>
/// <param name="cdc">The change stream dispatcher instance.</param>
void RegisterCdc(ChangeStreamDispatcher cdc);
/// <summary>
/// Begins a transaction.
/// Begins a transaction.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
Transaction BeginTransaction(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted);
/// <summary>
/// Begins a transaction asynchronously.
/// Begins a transaction asynchronously.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">A cancellation token.</param>
Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted, CancellationToken ct = default);
Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted,
CancellationToken ct = default);
/// <summary>
/// Gets collection metadata by name.
/// Gets collection metadata by name.
/// </summary>
/// <param name="name">The collection name.</param>
CollectionMetadata? GetCollectionMetadata(string name);
/// <summary>
/// Saves collection metadata.
/// Saves collection metadata.
/// </summary>
/// <param name="metadata">The metadata to persist.</param>
void SaveCollectionMetadata(CollectionMetadata metadata);
/// <summary>
/// Registers document mappers.
/// Registers document mappers.
/// </summary>
/// <param name="mappers">The mapper instances to register.</param>
void RegisterMappers(IEnumerable<IDocumentMapper> mappers);
/// <summary>
/// Gets schema chain entries for the specified root page.
/// Gets schema chain entries for the specified root page.
/// </summary>
/// <param name="rootPageId">The schema root page identifier.</param>
List<BsonSchema> GetSchemas(uint rootPageId);
/// <summary>
/// Appends a schema to the specified schema chain.
/// Appends a schema to the specified schema chain.
/// </summary>
/// <param name="rootPageId">The schema root page identifier.</param>
/// <param name="schema">The schema to append.</param>
uint AppendSchema(uint rootPageId, BsonSchema schema);
/// <summary>
/// Gets the key-to-token mapping.
/// Gets the key-to-token mapping.
/// </summary>
ConcurrentDictionary<string, ushort> GetKeyMap();
/// <summary>
/// Gets the token-to-key mapping.
/// Gets the token-to-key mapping.
/// </summary>
ConcurrentDictionary<ushort, string> GetKeyReverseMap();
/// <summary>
/// Gets or creates a dictionary token for the specified key.
/// Gets or creates a dictionary token for the specified key.
/// </summary>
/// <param name="key">The key value.</param>
ushort GetOrAddDictionaryEntry(string key);
/// <summary>
/// Registers key values in the dictionary mapping.
/// Registers key values in the dictionary mapping.
/// </summary>
/// <param name="keys">The keys to register.</param>
void RegisterKeys(IEnumerable<string> keys);
src/CBDD.Core/Storage/PageFile.cs
+177 -197
View File
@@ -3,27 +3,27 @@ using System.IO.MemoryMappedFiles;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Configuration for page-based storage
/// Configuration for page-based storage
/// </summary>
public readonly struct PageFileConfig
{
/// <summary>
/// Gets the size of each page in bytes.
/// Gets the size of each page in bytes.
/// </summary>
public int PageSize { get; init; }
/// <summary>
/// Gets the initial file size in bytes.
/// Gets the initial file size in bytes.
/// </summary>
public long InitialFileSize { get; init; }
/// <summary>
/// Gets the memory-mapped file access mode.
/// Gets the memory-mapped file access mode.
/// </summary>
public MemoryMappedFileAccess Access { get; init; }
/// <summary>
/// Small pages for embedded scenarios with many tiny documents
/// Small pages for embedded scenarios with many tiny documents
/// </summary>
public static PageFileConfig Small => new()
{
@@ -33,7 +33,7 @@ public readonly struct PageFileConfig
};
/// <summary>
/// Default balanced configuration for document databases (16KB like MySQL InnoDB)
/// Default balanced configuration for document databases (16KB like MySQL InnoDB)
/// </summary>
public static PageFileConfig Default => new()
{
@@ -43,7 +43,7 @@ public readonly struct PageFileConfig
};
/// <summary>
/// Large pages for databases with big documents (32KB like MongoDB WiredTiger)
/// Large pages for databases with big documents (32KB like MongoDB WiredTiger)
/// </summary>
public static PageFileConfig Large => new()
{
@@ -54,54 +54,51 @@ public readonly struct PageFileConfig
}
/// <summary>
/// Page-based file storage with memory-mapped I/O.
/// Manages fixed-size pages for efficient storage and retrieval.
/// Page-based file storage with memory-mapped I/O.
/// Manages fixed-size pages for efficient storage and retrieval.
/// </summary>
public sealed class PageFile : IDisposable
{
private readonly string _filePath;
private readonly PageFileConfig _config;
private FileStream? _fileStream;
private MemoryMappedFile? _mappedFile;
private readonly object _lock = new();
private bool _disposed;
private bool _wasCreated;
private uint _nextPageId;
private FileStream? _fileStream;
private uint _firstFreePageId;
private MemoryMappedFile? _mappedFile;
/// <summary>
/// Gets the next page identifier that will be allocated.
/// </summary>
public uint NextPageId => _nextPageId;
/// <summary>
/// Indicates whether this file was newly created on the current open call.
/// </summary>
public bool WasCreated => _wasCreated;
/// <summary>
/// Initializes a new instance of the <see cref="PageFile"/> class.
/// Initializes a new instance of the <see cref="PageFile" /> class.
/// </summary>
/// <param name="filePath">The file path for the page file.</param>
/// <param name="config">The page file configuration.</param>
public PageFile(string filePath, PageFileConfig config)
{
_filePath = filePath ?? throw new ArgumentNullException(nameof(filePath));
FilePath = filePath ?? throw new ArgumentNullException(nameof(filePath));
_config = config;
}
/// <summary>
/// Gets the configured page size in bytes.
/// Gets the next page identifier that will be allocated.
/// </summary>
public uint NextPageId { get; private set; }
/// <summary>
/// Indicates whether this file was newly created on the current open call.
/// </summary>
public bool WasCreated { get; private set; }
/// <summary>
/// Gets the configured page size in bytes.
/// </summary>
public int PageSize => _config.PageSize;
/// <summary>
/// Gets the underlying file path.
/// Gets the underlying file path.
/// </summary>
public string FilePath => _filePath;
public string FilePath { get; }
/// <summary>
/// Gets the current physical file length in bytes.
/// Gets the current physical file length in bytes.
/// </summary>
public long FileLengthBytes
{
@@ -116,12 +113,37 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Gets the effective page-file configuration.
/// Gets the effective page-file configuration.
/// </summary>
public PageFileConfig Config => _config;
/// <summary>
/// Opens the page file, creating it if it doesn't exist
/// Releases resources used by the page file.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
lock (_lock)
{
// 1. Flush any pending writes from memory-mapped file
if (_fileStream != null) _fileStream.Flush(true);
// 2. Close memory-mapped file first
_mappedFile?.Dispose();
// 3. Then close file stream
_fileStream?.Dispose();
_disposed = true;
}
GC.SuppressFinalize(this);
}
/// <summary>
/// Opens the page file, creating it if it doesn't exist
/// </summary>
public void Open()
{
@@ -130,26 +152,28 @@ public sealed class PageFile : IDisposable
if (_fileStream != null)
return; // Already open
var fileExists = File.Exists(_filePath);
bool fileExists = File.Exists(FilePath);
_fileStream = new FileStream(
_filePath,
FilePath,
FileMode.OpenOrCreate,
FileAccess.ReadWrite,
FileShare.None,
bufferSize: 4096,
4096,
FileOptions.RandomAccess);
_wasCreated = !fileExists || _fileStream.Length == 0;
if (_wasCreated)
WasCreated = !fileExists || _fileStream.Length == 0;
if (WasCreated)
{
// Initialize new file with 2 pages (Header + Collection Metadata)
_fileStream.SetLength(_config.InitialFileSize < _config.PageSize * 2 ? _config.PageSize * 2 : _config.InitialFileSize);
_fileStream.SetLength(_config.InitialFileSize < _config.PageSize * 2
? _config.PageSize * 2
: _config.InitialFileSize);
InitializeHeader();
}
// Initialize next page ID based on file length
_nextPageId = (uint)(_fileStream.Length / _config.PageSize);
NextPageId = (uint)(_fileStream.Length / _config.PageSize);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -157,7 +181,7 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
// Read free list head from Page 0
if (_fileStream.Length >= _config.PageSize)
@@ -172,7 +196,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Initializes the file header (page 0) and collection metadata (page 1)
/// Initializes the file header (page 0) and collection metadata (page 1)
/// </summary>
private void InitializeHeader()
{
@@ -216,7 +240,7 @@ public sealed class PageFile : IDisposable
// ... (ReadPage / WritePage unchanged) ...
/// <summary>
/// Reads a page by ID into the provided span
/// Reads a page by ID into the provided span
/// </summary>
/// <param name="pageId">The page identifier to read.</param>
/// <param name="destination">The destination span that receives page bytes.</param>
@@ -228,7 +252,7 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var offset = (long)pageId * _config.PageSize;
long offset = pageId * _config.PageSize;
using var accessor = _mappedFile.CreateViewAccessor(offset, _config.PageSize, MemoryMappedFileAccess.Read);
var temp = new byte[_config.PageSize];
@@ -237,7 +261,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Writes a page at the specified ID from the provided span
/// Writes a page at the specified ID from the provided span
/// </summary>
/// <param name="pageId">The page identifier to write.</param>
/// <param name="source">The source span that contains page bytes.</param>
@@ -249,16 +273,15 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var offset = (long)pageId * _config.PageSize;
long offset = pageId * _config.PageSize;
// Ensure file is large enough
if (offset + _config.PageSize > _fileStream!.Length)
{
lock (_lock)
{
if (offset + _config.PageSize > _fileStream.Length)
{
var newSize = Math.Max(offset + _config.PageSize, _fileStream.Length * 2);
long newSize = Math.Max(offset + _config.PageSize, _fileStream.Length * 2);
_fileStream.SetLength(newSize);
// Recreate memory-mapped file with new size
@@ -269,10 +292,9 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
}
}
}
// Write to memory-mapped file
using (var accessor = _mappedFile.CreateViewAccessor(offset, _config.PageSize, MemoryMappedFileAccess.Write))
@@ -282,7 +304,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Allocates a new page (reuses free page if available) and returns its ID
/// Allocates a new page (reuses free page if available) and returns its ID
/// </summary>
public uint AllocatePage()
{
@@ -294,7 +316,7 @@ public sealed class PageFile : IDisposable
// 1. Try to reuse a free page
if (_firstFreePageId != 0)
{
var recycledPageId = _firstFreePageId;
uint recycledPageId = _firstFreePageId;
// Read the recycled page to update the free list head
var buffer = new byte[_config.PageSize];
@@ -311,13 +333,13 @@ public sealed class PageFile : IDisposable
}
// 2. No free pages, append new one
var pageId = _nextPageId++;
uint pageId = NextPageId++;
// Extend file if necessary
var requiredLength = (long)(pageId + 1) * _config.PageSize;
long requiredLength = (pageId + 1) * _config.PageSize;
if (requiredLength > _fileStream.Length)
{
var newSize = Math.Max(requiredLength, _fileStream.Length * 2);
long newSize = Math.Max(requiredLength, _fileStream.Length * 2);
_fileStream.SetLength(newSize);
// Recreate memory-mapped file with new size
@@ -328,7 +350,7 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
}
return pageId;
@@ -336,7 +358,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Marks a page as free and adds it to the free list
/// Marks a page as free and adds it to the free list
/// </summary>
/// <param name="pageId">The page identifier to mark as free.</param>
public void FreePage(uint pageId)
@@ -386,7 +408,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Reads bytes from the page 0 extension region (immediately after the 32-byte file header).
/// Reads bytes from the page 0 extension region (immediately after the 32-byte file header).
/// </summary>
/// <param name="extensionOffset">Offset into the extension region, relative to byte 32.</param>
/// <param name="destination">Destination span receiving bytes.</param>
@@ -401,18 +423,20 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var absoluteOffset = 32 + extensionOffset;
int absoluteOffset = 32 + extensionOffset;
if (absoluteOffset + destination.Length > _config.PageSize)
throw new ArgumentOutOfRangeException(nameof(destination), "Requested range exceeds page 0 extension region.");
throw new ArgumentOutOfRangeException(nameof(destination),
"Requested range exceeds page 0 extension region.");
using var accessor = _mappedFile.CreateViewAccessor(absoluteOffset, destination.Length, MemoryMappedFileAccess.Read);
using var accessor =
_mappedFile.CreateViewAccessor(absoluteOffset, destination.Length, MemoryMappedFileAccess.Read);
var temp = new byte[destination.Length];
accessor.ReadArray(0, temp, 0, temp.Length);
temp.CopyTo(destination);
}
/// <summary>
/// Writes bytes to the page 0 extension region (immediately after the 32-byte file header).
/// Writes bytes to the page 0 extension region (immediately after the 32-byte file header).
/// </summary>
/// <param name="extensionOffset">Offset into the extension region, relative to byte 32.</param>
/// <param name="source">Source bytes to write.</param>
@@ -427,28 +451,29 @@ public sealed class PageFile : IDisposable
if (_mappedFile == null)
throw new InvalidOperationException("File not open");
var absoluteOffset = 32 + extensionOffset;
int absoluteOffset = 32 + extensionOffset;
if (absoluteOffset + source.Length > _config.PageSize)
throw new ArgumentOutOfRangeException(nameof(source), "Requested range exceeds page 0 extension region.");
using var accessor = _mappedFile.CreateViewAccessor(absoluteOffset, source.Length, MemoryMappedFileAccess.Write);
using var accessor =
_mappedFile.CreateViewAccessor(absoluteOffset, source.Length, MemoryMappedFileAccess.Write);
accessor.WriteArray(0, source.ToArray(), 0, source.Length);
}
/// <summary>
/// Flushes all pending writes to disk.
/// Called by CheckpointManager after applying WAL changes.
/// Flushes all pending writes to disk.
/// Called by CheckpointManager after applying WAL changes.
/// </summary>
public void Flush()
{
lock (_lock)
{
_fileStream?.Flush(flushToDisk: true);
_fileStream?.Flush(true);
}
}
/// <summary>
/// Writes a durable snapshot of the currently opened page file to a separate path.
/// Writes a durable snapshot of the currently opened page file to a separate path.
/// </summary>
/// <param name="destinationPath">Destination file path for the snapshot.</param>
public void SnapshotToFile(string destinationPath)
@@ -460,14 +485,11 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
var directory = Path.GetDirectoryName(destinationPath);
if (!string.IsNullOrWhiteSpace(directory))
{
Directory.CreateDirectory(directory);
}
string? directory = Path.GetDirectoryName(destinationPath);
if (!string.IsNullOrWhiteSpace(directory)) Directory.CreateDirectory(directory);
_fileStream!.Flush(flushToDisk: true);
var originalPosition = _fileStream.Position;
_fileStream!.Flush(true);
long originalPosition = _fileStream.Position;
try
{
_fileStream.Position = 0;
@@ -476,10 +498,10 @@ public sealed class PageFile : IDisposable
FileMode.Create,
FileAccess.Write,
FileShare.None,
bufferSize: 128 * 1024,
128 * 1024,
FileOptions.SequentialScan | FileOptions.WriteThrough);
_fileStream.CopyTo(destination);
destination.Flush(flushToDisk: true);
destination.Flush(true);
}
finally
{
@@ -489,7 +511,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Replaces the current file bytes with bytes from a source file and remaps the memory-mapped view.
/// Replaces the current file bytes with bytes from a source file and remaps the memory-mapped view.
/// </summary>
/// <param name="sourcePath">The source file path used as replacement content.</param>
public void ReplaceFromFile(string sourcePath)
@@ -508,11 +530,12 @@ public sealed class PageFile : IDisposable
FileMode.Open,
FileAccess.Read,
FileShare.Read,
bufferSize: 128 * 1024,
128 * 1024,
FileOptions.SequentialScan);
if (source.Length <= 0 || source.Length % _config.PageSize != 0)
throw new InvalidDataException($"Replacement file length must be a positive multiple of page size ({_config.PageSize}).");
throw new InvalidDataException(
$"Replacement file length must be a positive multiple of page size ({_config.PageSize}).");
_mappedFile?.Dispose();
_mappedFile = null;
@@ -520,7 +543,7 @@ public sealed class PageFile : IDisposable
_fileStream!.SetLength(source.Length);
_fileStream.Position = 0;
source.CopyTo(_fileStream);
_fileStream.Flush(flushToDisk: true);
_fileStream.Flush(true);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -528,16 +551,17 @@ public sealed class PageFile : IDisposable
_fileStream.Length,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
_nextPageId = (uint)(_fileStream.Length / _config.PageSize);
NextPageId = (uint)(_fileStream.Length / _config.PageSize);
_firstFreePageId = 0;
if (_fileStream.Length >= _config.PageSize)
{
const int pageHeaderSizeBytes = 32;
var headerSpan = new byte[pageHeaderSizeBytes];
using var accessor = _mappedFile.CreateViewAccessor(0, pageHeaderSizeBytes, MemoryMappedFileAccess.Read);
using var accessor =
_mappedFile.CreateViewAccessor(0, pageHeaderSizeBytes, MemoryMappedFileAccess.Read);
accessor.ReadArray(0, headerSpan, 0, pageHeaderSizeBytes);
var header = PageHeader.ReadFrom(headerSpan);
_firstFreePageId = header.NextPageId;
@@ -546,9 +570,9 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Enumerates all free pages, combining explicit free-list entries and reclaimable empty pages.
/// Enumerates all free pages, combining explicit free-list entries and reclaimable empty pages.
/// </summary>
/// <param name="includeEmptyPages">If set to <see langword="true"/>, includes all-zero pages as reclaimable.</param>
/// <param name="includeEmptyPages">If set to <see langword="true" />, includes all-zero pages as reclaimable.</param>
/// <returns>A sorted list of free page identifiers.</returns>
public IReadOnlyList<uint> EnumerateFreePages(bool includeEmptyPages = true)
{
@@ -561,9 +585,12 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Normalizes the free-list by rebuilding it from a deterministic sorted free page set.
/// Normalizes the free-list by rebuilding it from a deterministic sorted free page set.
/// </summary>
/// <param name="includeEmptyPages">If set to <see langword="true"/>, all-zero pages are converted into explicit free-list pages.</param>
/// <param name="includeEmptyPages">
/// If set to <see langword="true" />, all-zero pages are converted into explicit free-list
/// pages.
/// </param>
/// <returns>The number of pages in the normalized free-list.</returns>
public int NormalizeFreeList(bool includeEmptyPages = true)
{
@@ -577,10 +604,13 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Truncates contiguous reclaimable pages at the end of the file.
/// Reclaimable tail pages include explicit free pages and truly empty pages.
/// Truncates contiguous reclaimable pages at the end of the file.
/// Reclaimable tail pages include explicit free pages and truly empty pages.
/// </summary>
/// <param name="minimumPageCount">Minimum number of pages that must remain after truncation (defaults to header + metadata pages).</param>
/// <param name="minimumPageCount">
/// Minimum number of pages that must remain after truncation (defaults to header + metadata
/// pages).
/// </param>
/// <returns>Details about the truncation operation.</returns>
public TailTruncationResult TruncateReclaimableTailPages(uint minimumPageCount = 2)
{
@@ -588,31 +618,22 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
if (_nextPageId <= minimumPageCount)
{
return TailTruncationResult.None(_nextPageId);
}
if (NextPageId <= minimumPageCount) return TailTruncationResult.None(NextPageId);
var freePages = new HashSet<uint>(CollectFreePageIds(includeEmptyPages: true));
var originalPageCount = _nextPageId;
var newPageCount = _nextPageId;
var freePages = new HashSet<uint>(CollectFreePageIds(true));
uint originalPageCount = NextPageId;
uint newPageCount = NextPageId;
var pageBuffer = new byte[_config.PageSize];
while (newPageCount > minimumPageCount)
{
var candidatePageId = newPageCount - 1;
if (!IsReclaimableTailPage(candidatePageId, freePages, pageBuffer))
{
break;
}
uint candidatePageId = newPageCount - 1;
if (!IsReclaimableTailPage(candidatePageId, freePages, pageBuffer)) break;
newPageCount--;
}
if (newPageCount == originalPageCount)
{
return TailTruncationResult.None(originalPageCount);
}
if (newPageCount == originalPageCount) return TailTruncationResult.None(originalPageCount);
freePages.RemoveWhere(pageId => pageId >= newPageCount);
var remainingFreePages = freePages.ToList();
@@ -622,10 +643,10 @@ public sealed class PageFile : IDisposable
_mappedFile?.Dispose();
_mappedFile = null;
var previousLengthBytes = _fileStream!.Length;
var newLengthBytes = (long)newPageCount * _config.PageSize;
long previousLengthBytes = _fileStream!.Length;
long newLengthBytes = newPageCount * _config.PageSize;
_fileStream.SetLength(newLengthBytes);
_fileStream.Flush(flushToDisk: true);
_fileStream.Flush(true);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -633,9 +654,9 @@ public sealed class PageFile : IDisposable
newLengthBytes,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
_nextPageId = newPageCount;
NextPageId = newPageCount;
return new TailTruncationResult(
originalPageCount,
@@ -646,7 +667,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Trims excess physical file capacity beyond the current logical page count.
/// Trims excess physical file capacity beyond the current logical page count.
/// </summary>
/// <returns>The number of bytes removed from the file.</returns>
public long TrimExcessCapacityToLogicalPageCount()
@@ -655,8 +676,8 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
var targetLengthBytes = (long)_nextPageId * _config.PageSize;
var currentLengthBytes = _fileStream!.Length;
long targetLengthBytes = NextPageId * _config.PageSize;
long currentLengthBytes = _fileStream!.Length;
if (currentLengthBytes <= targetLengthBytes)
return 0;
@@ -664,7 +685,7 @@ public sealed class PageFile : IDisposable
_mappedFile = null;
_fileStream.SetLength(targetLengthBytes);
_fileStream.Flush(flushToDisk: true);
_fileStream.Flush(true);
_mappedFile = MemoryMappedFile.CreateFromFile(
_fileStream,
@@ -672,18 +693,18 @@ public sealed class PageFile : IDisposable
targetLengthBytes,
_config.Access,
HandleInheritability.None,
leaveOpen: true);
true);
return currentLengthBytes - targetLengthBytes;
}
}
/// <summary>
/// Defragments a slotted page in place by packing live slot payloads densely at the end of the page.
/// Defragments a slotted page in place by packing live slot payloads densely at the end of the page.
/// </summary>
/// <param name="pageId">The page identifier to defragment.</param>
/// <param name="reclaimedBytes">The number of free bytes reclaimed by defragmentation.</param>
/// <returns><see langword="true"/> when the page layout changed; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when the page layout changed; otherwise, <see langword="false" />.</returns>
public bool DefragmentSlottedPage(uint pageId, out int reclaimedBytes)
{
var result = DefragmentSlottedPageWithStats(pageId);
@@ -692,7 +713,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Defragments a slotted page in place and returns detailed relocation stats.
/// Defragments a slotted page in place and returns detailed relocation stats.
/// </summary>
/// <param name="pageId">The page identifier to defragment.</param>
public SlottedPageDefragmentationResult DefragmentSlottedPageWithStats(uint pageId)
@@ -701,7 +722,7 @@ public sealed class PageFile : IDisposable
{
EnsureFileOpen();
if (pageId >= _nextPageId)
if (pageId >= NextPageId)
throw new ArgumentOutOfRangeException(nameof(pageId));
var pageBuffer = new byte[_config.PageSize];
@@ -717,11 +738,11 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Defragments a slotted-page buffer in memory by rewriting live slots densely.
/// Defragments a slotted-page buffer in memory by rewriting live slots densely.
/// </summary>
/// <param name="pageBuffer">The page buffer to compact in place.</param>
/// <param name="reclaimedBytes">The number of free bytes reclaimed by compaction.</param>
/// <returns><see langword="true"/> when compaction modified the page; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> when compaction modified the page; otherwise, <see langword="false" />.</returns>
public static bool TryDefragmentSlottedPage(Span<byte> pageBuffer, out int reclaimedBytes)
{
var result = TryDefragmentSlottedPageWithStats(pageBuffer);
@@ -730,7 +751,7 @@ public sealed class PageFile : IDisposable
}
/// <summary>
/// Defragments a slotted-page buffer in memory and returns detailed relocation stats.
/// Defragments a slotted-page buffer in memory and returns detailed relocation stats.
/// </summary>
/// <param name="pageBuffer">The page buffer to compact in place.</param>
public static SlottedPageDefragmentationResult TryDefragmentSlottedPageWithStats(Span<byte> pageBuffer)
@@ -742,7 +763,7 @@ public sealed class PageFile : IDisposable
if (!IsSlottedPageType(header.PageType))
return SlottedPageDefragmentationResult.None;
var slotArrayEnd = SlottedPageHeader.Size + (header.SlotCount * SlotEntry.Size);
int slotArrayEnd = SlottedPageHeader.Size + header.SlotCount * SlotEntry.Size;
if (slotArrayEnd > pageBuffer.Length)
return SlottedPageDefragmentationResult.None;
@@ -750,29 +771,29 @@ public sealed class PageFile : IDisposable
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.Slice(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0 || slot.Length == 0)
continue;
var dataEnd = slot.Offset + slot.Length;
int dataEnd = slot.Offset + slot.Length;
if (slot.Offset < slotArrayEnd || dataEnd > pageBuffer.Length)
return SlottedPageDefragmentationResult.None;
var slotData = pageBuffer.Slice(slot.Offset, slot.Length).ToArray();
byte[] slotData = pageBuffer.Slice(slot.Offset, slot.Length).ToArray();
activeSlots.Add((i, slot, slotData));
}
var newFreeSpaceStart = (ushort)slotArrayEnd;
var writeCursor = pageBuffer.Length;
int writeCursor = pageBuffer.Length;
var changed = false;
var relocatedSlots = 0;
var oldFreeBytes = header.AvailableFreeSpace;
int oldFreeBytes = header.AvailableFreeSpace;
for (var i = 0; i < activeSlots.Count; i++)
{
var (slotIndex, slot, slotData) = activeSlots[i];
(ushort slotIndex, var slot, byte[] slotData) = activeSlots[i];
writeCursor -= slotData.Length;
if (writeCursor < newFreeSpaceStart)
return SlottedPageDefragmentationResult.None;
@@ -786,57 +807,24 @@ public sealed class PageFile : IDisposable
changed = true;
}
var slotOffset = SlottedPageHeader.Size + (slotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + slotIndex * SlotEntry.Size;
slot.WriteTo(pageBuffer.Slice(slotOffset, SlotEntry.Size));
}
if (writeCursor > newFreeSpaceStart)
{
pageBuffer.Slice(newFreeSpaceStart, writeCursor - newFreeSpaceStart).Clear();
}
if (header.FreeSpaceStart != newFreeSpaceStart || header.FreeSpaceEnd != writeCursor)
{
changed = true;
}
if (header.FreeSpaceStart != newFreeSpaceStart || header.FreeSpaceEnd != writeCursor) changed = true;
header.FreeSpaceStart = newFreeSpaceStart;
header.FreeSpaceEnd = (ushort)writeCursor;
header.WriteTo(pageBuffer);
var newFreeBytes = header.AvailableFreeSpace;
var reclaimedBytes = Math.Max(0, newFreeBytes - oldFreeBytes);
int newFreeBytes = header.AvailableFreeSpace;
int reclaimedBytes = Math.Max(0, newFreeBytes - oldFreeBytes);
return new SlottedPageDefragmentationResult(changed, reclaimedBytes, relocatedSlots);
}
/// <summary>
/// Releases resources used by the page file.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
lock (_lock)
{
// 1. Flush any pending writes from memory-mapped file
if (_fileStream != null)
{
_fileStream.Flush(flushToDisk: true);
}
// 2. Close memory-mapped file first
_mappedFile?.Dispose();
// 3. Then close file stream
_fileStream?.Dispose();
_disposed = true;
}
GC.SuppressFinalize(this);
}
private void EnsureFileOpen()
{
if (_fileStream == null || _mappedFile == null)
@@ -846,26 +834,23 @@ public sealed class PageFile : IDisposable
private List<uint> CollectFreePageIds(bool includeEmptyPages)
{
var freePages = new HashSet<uint>();
if (_nextPageId <= 2)
if (NextPageId <= 2)
return [];
var pageBuffer = new byte[_config.PageSize];
var seen = new HashSet<uint>();
var current = _firstFreePageId;
while (current != 0 && current < _nextPageId && seen.Add(current))
uint current = _firstFreePageId;
while (current != 0 && current < NextPageId && seen.Add(current))
{
if (current > 1)
{
freePages.Add(current);
}
if (current > 1) freePages.Add(current);
ReadPage(current, pageBuffer);
var header = PageHeader.ReadFrom(pageBuffer);
current = header.NextPageId;
}
for (uint pageId = 2; pageId < _nextPageId; pageId++)
for (uint pageId = 2; pageId < NextPageId; pageId++)
{
ReadPage(pageId, pageBuffer);
var header = PageHeader.ReadFrom(pageBuffer);
@@ -876,10 +861,7 @@ public sealed class PageFile : IDisposable
continue;
}
if (includeEmptyPages && IsTrulyEmptyPage(pageBuffer))
{
freePages.Add(pageId);
}
if (includeEmptyPages && IsTrulyEmptyPage(pageBuffer)) freePages.Add(pageId);
}
var ordered = freePages.ToList();
@@ -900,8 +882,8 @@ public sealed class PageFile : IDisposable
for (var i = 0; i < sortedFreePageIds.Count; i++)
{
var pageId = sortedFreePageIds[i];
var nextPageId = i + 1 < sortedFreePageIds.Count ? sortedFreePageIds[i + 1] : 0;
uint pageId = sortedFreePageIds[i];
uint nextPageId = i + 1 < sortedFreePageIds.Count ? sortedFreePageIds[i + 1] : 0;
Array.Clear(pageBuffer, 0, pageBuffer.Length);
var freeHeader = new PageHeader
@@ -928,7 +910,7 @@ public sealed class PageFile : IDisposable
private bool IsReclaimableTailPage(uint pageId, HashSet<uint> explicitFreePages, byte[] pageBuffer)
{
if (pageId <= 1 || pageId >= _nextPageId)
if (pageId <= 1 || pageId >= NextPageId)
return false;
if (explicitFreePages.Contains(pageId))
@@ -945,27 +927,25 @@ public sealed class PageFile : IDisposable
private static bool IsTrulyEmptyPage(ReadOnlySpan<byte> pageBuffer)
{
for (var i = 0; i < pageBuffer.Length; i++)
{
if (pageBuffer[i] != 0)
return false;
}
return true;
}
}
/// <summary>
/// Detailed result from slotted-page defragmentation.
/// Detailed result from slotted-page defragmentation.
/// </summary>
public readonly struct SlottedPageDefragmentationResult
{
/// <summary>
/// No-op result for non-slotted or invalid buffers.
/// No-op result for non-slotted or invalid buffers.
/// </summary>
public static SlottedPageDefragmentationResult None => new(false, 0, 0);
/// <summary>
/// Initializes a new instance of the <see cref="SlottedPageDefragmentationResult"/> struct.
/// Initializes a new instance of the <see cref="SlottedPageDefragmentationResult" /> struct.
/// </summary>
/// <param name="changed">Indicates whether the page layout changed.</param>
/// <param name="reclaimedBytes">The number of bytes reclaimed.</param>
@@ -978,28 +958,28 @@ public readonly struct SlottedPageDefragmentationResult
}
/// <summary>
/// Gets a value indicating whether the page layout changed.
/// Gets a value indicating whether the page layout changed.
/// </summary>
public bool Changed { get; }
/// <summary>
/// Gets reclaimed free bytes after defragmentation.
/// Gets reclaimed free bytes after defragmentation.
/// </summary>
public int ReclaimedBytes { get; }
/// <summary>
/// Gets the number of live slots that were moved to a new offset.
/// Gets the number of live slots that were moved to a new offset.
/// </summary>
public int RelocatedSlotCount { get; }
}
/// <summary>
/// Result of a reclaimable tail truncation operation.
/// Result of a reclaimable tail truncation operation.
/// </summary>
public readonly struct TailTruncationResult
{
/// <summary>
/// Initializes a new instance of the <see cref="TailTruncationResult"/> struct.
/// Initializes a new instance of the <see cref="TailTruncationResult" /> struct.
/// </summary>
/// <param name="prePageCount">The page count before truncation.</param>
/// <param name="postPageCount">The page count after truncation.</param>
@@ -1014,27 +994,27 @@ public readonly struct TailTruncationResult
}
/// <summary>
/// Gets the page count before truncation.
/// Gets the page count before truncation.
/// </summary>
public uint PrePageCount { get; }
/// <summary>
/// Gets the page count after truncation.
/// Gets the page count after truncation.
/// </summary>
public uint PostPageCount { get; }
/// <summary>
/// Gets the number of truncated pages.
/// Gets the number of truncated pages.
/// </summary>
public uint TruncatedPages { get; }
/// <summary>
/// Gets the number of truncated bytes.
/// Gets the number of truncated bytes.
/// </summary>
public long TruncatedBytes { get; }
/// <summary>
/// Creates a no-op truncation result.
/// Creates a no-op truncation result.
/// </summary>
/// <param name="pageCount">The page count to assign before and after truncation.</param>
public static TailTruncationResult None(uint pageCount)
src/CBDD.Core/Storage/PageHeader.cs
+17 -29
View File
@@ -3,54 +3,42 @@ using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Represents a page header in the database file.
/// Fixed 32-byte structure at the start of each page.
/// Implemented as struct for efficient memory layout.
/// Represents a page header in the database file.
/// Fixed 32-byte structure at the start of each page.
/// Implemented as struct for efficient memory layout.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 32)]
public struct PageHeader
{
/// <summary>Page ID (offset in pages from start of file)</summary>
[FieldOffset(0)]
public uint PageId;
[FieldOffset(0)] public uint PageId;
/// <summary>Type of this page</summary>
[FieldOffset(4)]
public PageType PageType;
[FieldOffset(4)] public PageType PageType;
/// <summary>Number of free bytes in this page</summary>
[FieldOffset(5)]
public ushort FreeBytes;
[FieldOffset(5)] public ushort FreeBytes;
/// <summary>ID of next page in linked list (0 if none). For Page 0 (Header), this points to the First Free Page.</summary>
[FieldOffset(7)]
public uint NextPageId;
[FieldOffset(7)] public uint NextPageId;
/// <summary>Transaction ID that last modified this page</summary>
[FieldOffset(11)]
public ulong TransactionId;
[FieldOffset(11)] public ulong TransactionId;
/// <summary>Checksum for data integrity (CRC32)</summary>
[FieldOffset(19)]
public uint Checksum;
[FieldOffset(19)] public uint Checksum;
/// <summary>Dictionary Root Page ID (Only used in Page 0 / File Header)</summary>
[FieldOffset(23)]
public uint DictionaryRootPageId;
[FieldOffset(23)] public uint DictionaryRootPageId;
[FieldOffset(27)]
private byte _reserved5;
[FieldOffset(28)]
private byte _reserved6;
[FieldOffset(29)]
private byte _reserved7;
[FieldOffset(30)]
private byte _reserved8;
[FieldOffset(31)]
private byte _reserved9;
[FieldOffset(27)] private byte _reserved5;
[FieldOffset(28)] private byte _reserved6;
[FieldOffset(29)] private byte _reserved7;
[FieldOffset(30)] private byte _reserved8;
[FieldOffset(31)] private byte _reserved9;
/// <summary>
/// Writes the header to a span
/// Writes the header to a span
/// </summary>
/// <param name="destination">The destination span that receives the serialized header.</param>
public readonly void WriteTo(Span<byte> destination)
@@ -62,7 +50,7 @@ public struct PageHeader
}
/// <summary>
/// Reads a header from a span
/// Reads a header from a span
/// </summary>
/// <param name="source">The source span containing a serialized header.</param>
public static PageHeader ReadFrom(ReadOnlySpan<byte> source)
src/CBDD.Core/Storage/PageType.cs
+1 -1
View File
@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page types in the database file
/// Page types in the database file
/// </summary>
public enum PageType : byte
{
src/CBDD.Core/Storage/SlottedPage.cs
+37 -46
View File
@@ -1,50 +1,43 @@
using System.Buffers.Binary;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Header for slotted pages supporting multiple variable-size documents per page.
/// Fixed 24-byte structure at start of each data page.
/// Header for slotted pages supporting multiple variable-size documents per page.
/// Fixed 24-byte structure at start of each data page.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 24)]
public struct SlottedPageHeader
{
/// <summary>Page ID</summary>
[FieldOffset(0)]
public uint PageId;
[FieldOffset(0)] public uint PageId;
/// <summary>Type of page (Data, Overflow, Index, Metadata)</summary>
[FieldOffset(4)]
public PageType PageType;
[FieldOffset(4)] public PageType PageType;
/// <summary>Number of slot entries in this page</summary>
[FieldOffset(8)]
public ushort SlotCount;
[FieldOffset(8)] public ushort SlotCount;
/// <summary>Offset where free space starts (grows down with slots)</summary>
[FieldOffset(10)]
public ushort FreeSpaceStart;
[FieldOffset(10)] public ushort FreeSpaceStart;
/// <summary>Offset where free space ends (grows up with data)</summary>
[FieldOffset(12)]
public ushort FreeSpaceEnd;
[FieldOffset(12)] public ushort FreeSpaceEnd;
/// <summary>Next overflow page ID (0 if none)</summary>
[FieldOffset(14)]
public uint NextOverflowPage;
[FieldOffset(14)] public uint NextOverflowPage;
/// <summary>Transaction ID that last modified this page</summary>
[FieldOffset(18)]
public uint TransactionId;
[FieldOffset(18)] public uint TransactionId;
/// <summary>Reserved for future use</summary>
[FieldOffset(22)]
public ushort Reserved;
[FieldOffset(22)] public ushort Reserved;
public const int Size = 24;
/// <summary>
/// Initializes a header with the current slotted-page format marker.
/// Initializes a header with the current slotted-page format marker.
/// </summary>
public SlottedPageHeader()
{
@@ -53,12 +46,12 @@ public struct SlottedPageHeader
}
/// <summary>
/// Gets available free space in bytes
/// Gets available free space in bytes
/// </summary>
public readonly int AvailableFreeSpace => FreeSpaceEnd - FreeSpaceStart;
/// <summary>
/// Writes header to span
/// Writes header to span
/// </summary>
/// <param name="destination">The destination span that receives the serialized header.</param>
public readonly void WriteTo(Span<byte> destination)
@@ -70,7 +63,7 @@ public struct SlottedPageHeader
}
/// <summary>
/// Reads header from span
/// Reads header from span
/// </summary>
/// <param name="source">The source span containing the serialized header.</param>
public static SlottedPageHeader ReadFrom(ReadOnlySpan<byte> source)
@@ -83,28 +76,25 @@ public struct SlottedPageHeader
}
/// <summary>
/// Slot entry pointing to a document within a page.
/// Fixed 8-byte structure in slot array.
/// Slot entry pointing to a document within a page.
/// Fixed 8-byte structure in slot array.
/// </summary>
[StructLayout(LayoutKind.Explicit, Size = 8)]
public struct SlotEntry
{
/// <summary>Offset to document data within page</summary>
[FieldOffset(0)]
public ushort Offset;
[FieldOffset(0)] public ushort Offset;
/// <summary>Length of document data in bytes</summary>
[FieldOffset(2)]
public ushort Length;
[FieldOffset(2)] public ushort Length;
/// <summary>Slot flags (deleted, overflow, etc.)</summary>
[FieldOffset(4)]
public SlotFlags Flags;
[FieldOffset(4)] public SlotFlags Flags;
public const int Size = 8;
/// <summary>
/// Writes slot entry to span
/// Writes slot entry to span
/// </summary>
/// <param name="destination">The destination span that receives the serialized slot entry.</param>
public readonly void WriteTo(Span<byte> destination)
@@ -116,7 +106,7 @@ public struct SlotEntry
}
/// <summary>
/// Reads slot entry from span
/// Reads slot entry from span
/// </summary>
/// <param name="source">The source span containing the serialized slot entry.</param>
public static SlotEntry ReadFrom(ReadOnlySpan<byte> source)
@@ -129,7 +119,7 @@ public struct SlotEntry
}
/// <summary>
/// Flags for slot entries
/// Flags for slot entries
/// </summary>
[Flags]
public enum SlotFlags : uint
@@ -144,26 +134,27 @@ public enum SlotFlags : uint
HasOverflow = 1 << 1,
/// <summary>Document data is compressed</summary>
Compressed = 1 << 2,
Compressed = 1 << 2
}
/// <summary>
/// Location of a document within the database.
/// Maps ObjectId to specific page and slot.
/// Location of a document within the database.
/// Maps ObjectId to specific page and slot.
/// </summary>
public readonly struct DocumentLocation
{
/// <summary>
/// Gets the page identifier containing the document.
/// Gets the page identifier containing the document.
/// </summary>
public uint PageId { get; init; }
/// <summary>
/// Gets the slot index within the page.
/// Gets the slot index within the page.
/// </summary>
public ushort SlotIndex { get; init; }
/// <summary>
/// Initializes a new instance of the <see cref="DocumentLocation"/> struct.
/// Initializes a new instance of the <see cref="DocumentLocation" /> struct.
/// </summary>
/// <param name="pageId">The page identifier containing the document.</param>
/// <param name="slotIndex">The slot index within the page.</param>
@@ -174,7 +165,7 @@ public readonly struct DocumentLocation
}
/// <summary>
/// Serializes DocumentLocation to a byte span (6 bytes: 4 for PageId + 2 for SlotIndex)
/// Serializes DocumentLocation to a byte span (6 bytes: 4 for PageId + 2 for SlotIndex)
/// </summary>
/// <param name="destination">The destination span that receives the serialized value.</param>
public void WriteTo(Span<byte> destination)
@@ -182,12 +173,12 @@ public readonly struct DocumentLocation
if (destination.Length < 6)
throw new ArgumentException("Destination must be at least 6 bytes", nameof(destination));
System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(destination, PageId);
System.Buffers.Binary.BinaryPrimitives.WriteUInt16LittleEndian(destination.Slice(4), SlotIndex);
BinaryPrimitives.WriteUInt32LittleEndian(destination, PageId);
BinaryPrimitives.WriteUInt16LittleEndian(destination.Slice(4), SlotIndex);
}
/// <summary>
/// Deserializes DocumentLocation from a byte span (6 bytes)
/// Deserializes DocumentLocation from a byte span (6 bytes)
/// </summary>
/// <param name="source">The source span containing the serialized value.</param>
public static DocumentLocation ReadFrom(ReadOnlySpan<byte> source)
@@ -195,14 +186,14 @@ public readonly struct DocumentLocation
if (source.Length < 6)
throw new ArgumentException("Source must be at least 6 bytes", nameof(source));
var pageId = System.Buffers.Binary.BinaryPrimitives.ReadUInt32LittleEndian(source);
var slotIndex = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian(source.Slice(4));
uint pageId = BinaryPrimitives.ReadUInt32LittleEndian(source);
ushort slotIndex = BinaryPrimitives.ReadUInt16LittleEndian(source.Slice(4));
return new DocumentLocation(pageId, slotIndex);
}
/// <summary>
/// Size in bytes when serialized
/// Size in bytes when serialized
/// </summary>
public const int SerializedSize = 6;
}
src/CBDD.Core/Storage/SpatialPage.cs
+47 -26
View File
@@ -1,12 +1,11 @@
using System.Buffers.Binary;
using System.Runtime.InteropServices;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Indexing.Internal;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page for storing R-Tree nodes for Geospatial Indexing.
/// Page for storing R-Tree nodes for Geospatial Indexing.
/// </summary>
internal struct SpatialPage
{
@@ -30,7 +29,7 @@ internal struct SpatialPage
public const int EntrySize = 38; // 32 (GeoBox) + 6 (Pointer)
/// <summary>
/// Initializes a spatial page.
/// Initializes a spatial page.
/// </summary>
/// <param name="page">The page buffer to initialize.</param>
/// <param name="pageId">The page identifier.</param>
@@ -55,56 +54,77 @@ internal struct SpatialPage
}
/// <summary>
/// Gets a value indicating whether the page is a leaf node.
/// Gets a value indicating whether the page is a leaf node.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns><see langword="true"/> if the page is a leaf node; otherwise, <see langword="false"/>.</returns>
public static bool GetIsLeaf(ReadOnlySpan<byte> page) => page[IsLeafOffset] == 1;
/// <returns><see langword="true" /> if the page is a leaf node; otherwise, <see langword="false" />.</returns>
public static bool GetIsLeaf(ReadOnlySpan<byte> page)
{
return page[IsLeafOffset] == 1;
}
/// <summary>
/// Gets the tree level stored in the page.
/// Gets the tree level stored in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The level value.</returns>
public static byte GetLevel(ReadOnlySpan<byte> page) => page[LevelOffset];
public static byte GetLevel(ReadOnlySpan<byte> page)
{
return page[LevelOffset];
}
/// <summary>
/// Gets the number of entries in the page.
/// Gets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The number of entries.</returns>
public static ushort GetEntryCount(ReadOnlySpan<byte> page) => BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(EntryCountOffset));
public static ushort GetEntryCount(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadUInt16LittleEndian(page.Slice(EntryCountOffset));
}
/// <summary>
/// Sets the number of entries in the page.
/// Sets the number of entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="count">The entry count to set.</param>
public static void SetEntryCount(Span<byte> page, ushort count) => BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(EntryCountOffset), count);
public static void SetEntryCount(Span<byte> page, ushort count)
{
BinaryPrimitives.WriteUInt16LittleEndian(page.Slice(EntryCountOffset), count);
}
/// <summary>
/// Gets the parent page identifier.
/// Gets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The parent page identifier.</returns>
public static uint GetParentPageId(ReadOnlySpan<byte> page) => BinaryPrimitives.ReadUInt32LittleEndian(page.Slice(ParentPageIdOffset));
public static uint GetParentPageId(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadUInt32LittleEndian(page.Slice(ParentPageIdOffset));
}
/// <summary>
/// Sets the parent page identifier.
/// Sets the parent page identifier.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="parentId">The parent page identifier.</param>
public static void SetParentPageId(Span<byte> page, uint parentId) => BinaryPrimitives.WriteUInt32LittleEndian(page.Slice(ParentPageIdOffset), parentId);
public static void SetParentPageId(Span<byte> page, uint parentId)
{
BinaryPrimitives.WriteUInt32LittleEndian(page.Slice(ParentPageIdOffset), parentId);
}
/// <summary>
/// Gets the maximum number of entries that can fit in a page.
/// Gets the maximum number of entries that can fit in a page.
/// </summary>
/// <param name="pageSize">The page size in bytes.</param>
/// <returns>The maximum number of entries.</returns>
public static int GetMaxEntries(int pageSize) => (pageSize - DataOffset) / EntrySize;
public static int GetMaxEntries(int pageSize)
{
return (pageSize - DataOffset) / EntrySize;
}
/// <summary>
/// Writes an entry at the specified index.
/// Writes an entry at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="index">The entry index.</param>
@@ -112,7 +132,7 @@ internal struct SpatialPage
/// <param name="pointer">The document location pointer.</param>
public static void WriteEntry(Span<byte> page, int index, GeoBox mbr, DocumentLocation pointer)
{
int offset = DataOffset + (index * EntrySize);
int offset = DataOffset + index * EntrySize;
var entrySpan = page.Slice(offset, EntrySize);
// Write MBR (4 doubles)
@@ -127,7 +147,7 @@ internal struct SpatialPage
}
/// <summary>
/// Reads an entry at the specified index.
/// Reads an entry at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="index">The entry index.</param>
@@ -135,7 +155,7 @@ internal struct SpatialPage
/// <param name="pointer">When this method returns, contains the entry document location.</param>
public static void ReadEntry(ReadOnlySpan<byte> page, int index, out GeoBox mbr, out DocumentLocation pointer)
{
int offset = DataOffset + (index * EntrySize);
int offset = DataOffset + index * EntrySize;
var entrySpan = page.Slice(offset, EntrySize);
var doubles = MemoryMarshal.Cast<byte, double>(entrySpan.Slice(0, 32));
@@ -144,22 +164,23 @@ internal struct SpatialPage
}
/// <summary>
/// Calculates the combined MBR of all entries in the page.
/// Calculates the combined MBR of all entries in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The combined MBR, or <see cref="GeoBox.Empty"/> when the page has no entries.</returns>
/// <returns>The combined MBR, or <see cref="GeoBox.Empty" /> when the page has no entries.</returns>
public static GeoBox CalculateMBR(ReadOnlySpan<byte> page)
{
ushort count = GetEntryCount(page);
if (count == 0) return GeoBox.Empty;
GeoBox result = GeoBox.Empty;
for (int i = 0; i < count; i++)
var result = GeoBox.Empty;
for (var i = 0; i < count; i++)
{
ReadEntry(page, i, out var mbr, out _);
if (i == 0) result = mbr;
else result = result.ExpandTo(mbr);
}
return result;
}
}
src/CBDD.Core/Storage/StorageEngine.Collections.cs
+71 -81
View File
@@ -1,31 +1,27 @@
using System;
using System.Collections.Generic;
using System.IO;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Indexing;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public class CollectionMetadata
{
/// <summary>
/// Gets or sets the collection name.
/// Gets or sets the collection name.
/// </summary>
public string Name { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the root page identifier of the primary index.
/// Gets or sets the root page identifier of the primary index.
/// </summary>
public uint PrimaryRootPageId { get; set; }
/// <summary>
/// Gets or sets the root page identifier of the schema chain.
/// Gets or sets the root page identifier of the schema chain.
/// </summary>
public uint SchemaRootPageId { get; set; }
/// <summary>
/// Gets the collection index metadata list.
/// Gets the collection index metadata list.
/// </summary>
public List<IndexMetadata> Indexes { get; } = new();
}
@@ -33,37 +29,37 @@ public class CollectionMetadata
public class IndexMetadata
{
/// <summary>
/// Gets or sets the index name.
/// Gets or sets the index name.
/// </summary>
public string Name { get; set; } = string.Empty;
/// <summary>
/// Gets or sets a value indicating whether this index enforces uniqueness.
/// Gets or sets a value indicating whether this index enforces uniqueness.
/// </summary>
public bool IsUnique { get; set; }
/// <summary>
/// Gets or sets the index type.
/// Gets or sets the index type.
/// </summary>
public IndexType Type { get; set; }
/// <summary>
/// Gets or sets indexed property paths.
/// Gets or sets indexed property paths.
/// </summary>
public string[] PropertyPaths { get; set; } = Array.Empty<string>();
/// <summary>
/// Gets or sets vector dimensions for vector indexes.
/// Gets or sets vector dimensions for vector indexes.
/// </summary>
public int Dimensions { get; set; }
/// <summary>
/// Gets or sets the vector similarity metric for vector indexes.
/// Gets or sets the vector similarity metric for vector indexes.
/// </summary>
public VectorMetric Metric { get; set; }
/// <summary>
/// Gets or sets the root page identifier of the index structure.
/// Gets or sets the root page identifier of the index structure.
/// </summary>
public uint RootPageId { get; set; }
}
@@ -71,7 +67,7 @@ public class IndexMetadata
public sealed partial class StorageEngine
{
/// <summary>
/// Gets collection metadata by name.
/// Gets collection metadata by name.
/// </summary>
/// <param name="name">The collection name.</param>
/// <returns>The collection metadata if found; otherwise, null.</returns>
@@ -82,39 +78,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Returns all collection metadata entries currently registered in page 1.
/// </summary>
public IReadOnlyList<CollectionMetadata> GetAllCollectionMetadata()
{
var result = new List<CollectionMetadata>();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
if (header.PageType != PageType.Collection || header.SlotCount == 0)
return result;
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0)
continue;
if (slot.Offset < SlottedPageHeader.Size || slot.Offset + slot.Length > buffer.Length)
continue;
if (TryDeserializeCollectionMetadata(buffer.AsSpan(slot.Offset, slot.Length), out var metadata) && metadata != null)
{
result.Add(metadata);
}
}
return result;
}
/// <summary>
/// Saves collection metadata to the metadata page.
/// Saves collection metadata to the metadata page.
/// </summary>
/// <param name="metadata">The metadata to save.</param>
public void SaveCollectionMetadata(CollectionMetadata metadata)
@@ -133,10 +97,7 @@ public sealed partial class StorageEngine
writer.Write((byte)idx.Type);
writer.Write(idx.RootPageId);
writer.Write(idx.PropertyPaths.Length);
foreach (var path in idx.PropertyPaths)
{
writer.Write(path);
}
foreach (string path in idx.PropertyPaths) writer.Write(path);
if (idx.Type == IndexType.Vector)
{
@@ -145,7 +106,7 @@ public sealed partial class StorageEngine
}
}
var newData = stream.ToArray();
byte[] newData = stream.ToArray();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
@@ -155,7 +116,7 @@ public sealed partial class StorageEngine
for (ushort i = 0; i < header.SlotCount; i++)
{
var slotOffset = SlottedPageHeader.Size + (i * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0) continue;
@@ -163,7 +124,7 @@ public sealed partial class StorageEngine
{
using var ms = new MemoryStream(buffer, slot.Offset, slot.Length, false);
using var reader = new BinaryReader(ms);
var name = reader.ReadString();
string name = reader.ReadString();
if (string.Equals(name, metadata.Name, StringComparison.OrdinalIgnoreCase))
{
@@ -171,22 +132,23 @@ public sealed partial class StorageEngine
break;
}
}
catch { }
catch
{
}
}
if (existingSlotIndex >= 0)
{
var slotOffset = SlottedPageHeader.Size + (existingSlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + existingSlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
slot.Flags |= SlotFlags.Deleted;
slot.WriteTo(buffer.AsSpan(slotOffset));
}
if (header.AvailableFreeSpace < newData.Length + SlotEntry.Size)
{
// Compact logic omitted as per current architecture
throw new InvalidOperationException("Not enough space in Metadata Page (Page 1) to save collection metadata.");
}
throw new InvalidOperationException(
"Not enough space in Metadata Page (Page 1) to save collection metadata.");
int docOffset = header.FreeSpaceEnd - newData.Length;
newData.CopyTo(buffer.AsSpan(docOffset));
@@ -202,7 +164,7 @@ public sealed partial class StorageEngine
header.SlotCount++;
}
var newSlotEntryOffset = SlottedPageHeader.Size + (slotIndex * SlotEntry.Size);
int newSlotEntryOffset = SlottedPageHeader.Size + slotIndex * SlotEntry.Size;
var newSlot = new SlotEntry
{
Offset = (ushort)docOffset,
@@ -213,14 +175,52 @@ public sealed partial class StorageEngine
header.FreeSpaceEnd = (ushort)docOffset;
if (existingSlotIndex == -1)
{
header.FreeSpaceStart = (ushort)(SlottedPageHeader.Size + (header.SlotCount * SlotEntry.Size));
}
header.FreeSpaceStart = (ushort)(SlottedPageHeader.Size + header.SlotCount * SlotEntry.Size);
header.WriteTo(buffer);
WritePageImmediate(1, buffer);
}
/// <summary>
/// Registers all BSON keys used by a set of mappers into the global dictionary.
/// </summary>
/// <param name="mappers">The mappers whose keys should be registered.</param>
public void RegisterMappers(IEnumerable<IDocumentMapper> mappers)
{
var allKeys = mappers.SelectMany(m => m.UsedKeys).Distinct();
RegisterKeys(allKeys);
}
/// <summary>
/// Returns all collection metadata entries currently registered in page 1.
/// </summary>
public IReadOnlyList<CollectionMetadata> GetAllCollectionMetadata()
{
var result = new List<CollectionMetadata>();
var buffer = new byte[PageSize];
ReadPage(1, null, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
if (header.PageType != PageType.Collection || header.SlotCount == 0)
return result;
for (ushort i = 0; i < header.SlotCount; i++)
{
int slotOffset = SlottedPageHeader.Size + i * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(buffer.AsSpan(slotOffset));
if ((slot.Flags & SlotFlags.Deleted) != 0)
continue;
if (slot.Offset < SlottedPageHeader.Size || slot.Offset + slot.Length > buffer.Length)
continue;
if (TryDeserializeCollectionMetadata(buffer.AsSpan(slot.Offset, slot.Length), out var metadata) &&
metadata != null) result.Add(metadata);
}
return result;
}
private static bool TryDeserializeCollectionMetadata(ReadOnlySpan<byte> rawBytes, out CollectionMetadata? metadata)
{
metadata = null;
@@ -230,16 +230,16 @@ public sealed partial class StorageEngine
using var ms = new MemoryStream(rawBytes.ToArray());
using var reader = new BinaryReader(ms);
var collName = reader.ReadString();
string collName = reader.ReadString();
var parsed = new CollectionMetadata { Name = collName };
parsed.PrimaryRootPageId = reader.ReadUInt32();
parsed.SchemaRootPageId = reader.ReadUInt32();
var indexCount = reader.ReadInt32();
int indexCount = reader.ReadInt32();
if (indexCount < 0)
return false;
for (int j = 0; j < indexCount; j++)
for (var j = 0; j < indexCount; j++)
{
var idx = new IndexMetadata
{
@@ -249,12 +249,12 @@ public sealed partial class StorageEngine
RootPageId = reader.ReadUInt32()
};
var pathCount = reader.ReadInt32();
int pathCount = reader.ReadInt32();
if (pathCount < 0)
return false;
idx.PropertyPaths = new string[pathCount];
for (int k = 0; k < pathCount; k++)
for (var k = 0; k < pathCount; k++)
idx.PropertyPaths[k] = reader.ReadString();
if (idx.Type == IndexType.Vector)
@@ -274,14 +274,4 @@ public sealed partial class StorageEngine
return false;
}
}
/// <summary>
/// Registers all BSON keys used by a set of mappers into the global dictionary.
/// </summary>
/// <param name="mappers">The mappers whose keys should be registered.</param>
public void RegisterMappers(IEnumerable<IDocumentMapper> mappers)
{
var allKeys = mappers.SelectMany(m => m.UsedKeys).Distinct();
RegisterKeys(allKeys);
}
}
src/CBDD.Core/Storage/StorageEngine.Diagnostics.cs
+70 -85
View File
@@ -5,172 +5,173 @@ using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Aggregated page counts grouped by page type.
/// Aggregated page counts grouped by page type.
/// </summary>
public sealed class PageTypeUsageEntry
{
/// <summary>
/// Gets the page type.
/// Gets the page type.
/// </summary>
public PageType PageType { get; init; }
/// <summary>
/// Gets the number of pages of this type.
/// Gets the number of pages of this type.
/// </summary>
public int PageCount { get; init; }
}
/// <summary>
/// Per-collection page usage summary.
/// Per-collection page usage summary.
/// </summary>
public sealed class CollectionPageUsageEntry
{
/// <summary>
/// Gets the collection name.
/// Gets the collection name.
/// </summary>
public string CollectionName { get; init; } = string.Empty;
/// <summary>
/// Gets the total number of distinct pages referenced by the collection.
/// Gets the total number of distinct pages referenced by the collection.
/// </summary>
public int TotalDistinctPages { get; init; }
/// <summary>
/// Gets the number of data pages.
/// Gets the number of data pages.
/// </summary>
public int DataPages { get; init; }
/// <summary>
/// Gets the number of overflow pages.
/// Gets the number of overflow pages.
/// </summary>
public int OverflowPages { get; init; }
/// <summary>
/// Gets the number of index pages.
/// Gets the number of index pages.
/// </summary>
public int IndexPages { get; init; }
/// <summary>
/// Gets the number of other page types.
/// Gets the number of other page types.
/// </summary>
public int OtherPages { get; init; }
}
/// <summary>
/// Per-collection compression ratio summary.
/// Per-collection compression ratio summary.
/// </summary>
public sealed class CollectionCompressionRatioEntry
{
/// <summary>
/// Gets the collection name.
/// Gets the collection name.
/// </summary>
public string CollectionName { get; init; } = string.Empty;
/// <summary>
/// Gets the number of documents.
/// Gets the number of documents.
/// </summary>
public long DocumentCount { get; init; }
/// <summary>
/// Gets the number of compressed documents.
/// Gets the number of compressed documents.
/// </summary>
public long CompressedDocumentCount { get; init; }
/// <summary>
/// Gets the total uncompressed byte count.
/// Gets the total uncompressed byte count.
/// </summary>
public long BytesBeforeCompression { get; init; }
/// <summary>
/// Gets the total stored byte count.
/// Gets the total stored byte count.
/// </summary>
public long BytesAfterCompression { get; init; }
/// <summary>
/// Gets the compression ratio.
/// Gets the compression ratio.
/// </summary>
public double CompressionRatio => BytesAfterCompression <= 0 ? 1.0 : (double)BytesBeforeCompression / BytesAfterCompression;
public double CompressionRatio =>
BytesAfterCompression <= 0 ? 1.0 : (double)BytesBeforeCompression / BytesAfterCompression;
}
/// <summary>
/// Summary of free-list and reclaimable tail information.
/// Summary of free-list and reclaimable tail information.
/// </summary>
public sealed class FreeListSummary
{
/// <summary>
/// Gets the total page count.
/// Gets the total page count.
/// </summary>
public uint PageCount { get; init; }
/// <summary>
/// Gets the free page count.
/// Gets the free page count.
/// </summary>
public int FreePageCount { get; init; }
/// <summary>
/// Gets the total free bytes.
/// Gets the total free bytes.
/// </summary>
public long FreeBytes { get; init; }
/// <summary>
/// Gets the fragmentation percentage.
/// Gets the fragmentation percentage.
/// </summary>
public double FragmentationPercent { get; init; }
/// <summary>
/// Gets the number of reclaimable pages at the file tail.
/// Gets the number of reclaimable pages at the file tail.
/// </summary>
public uint TailReclaimablePages { get; init; }
}
/// <summary>
/// Single page entry in fragmentation reporting.
/// Single page entry in fragmentation reporting.
/// </summary>
public sealed class FragmentationPageEntry
{
/// <summary>
/// Gets the page identifier.
/// Gets the page identifier.
/// </summary>
public uint PageId { get; init; }
/// <summary>
/// Gets the page type.
/// Gets the page type.
/// </summary>
public PageType PageType { get; init; }
/// <summary>
/// Gets a value indicating whether this page is free.
/// Gets a value indicating whether this page is free.
/// </summary>
public bool IsFreePage { get; init; }
/// <summary>
/// Gets the free bytes on the page.
/// Gets the free bytes on the page.
/// </summary>
public int FreeBytes { get; init; }
}
/// <summary>
/// Detailed fragmentation map and totals.
/// Detailed fragmentation map and totals.
/// </summary>
public sealed class FragmentationMapReport
{
/// <summary>
/// Gets the page entries.
/// Gets the page entries.
/// </summary>
public IReadOnlyList<FragmentationPageEntry> Pages { get; init; } = Array.Empty<FragmentationPageEntry>();
/// <summary>
/// Gets the total free bytes across all pages.
/// Gets the total free bytes across all pages.
/// </summary>
public long TotalFreeBytes { get; init; }
/// <summary>
/// Gets the fragmentation percentage.
/// Gets the fragmentation percentage.
/// </summary>
public double FragmentationPercent { get; init; }
/// <summary>
/// Gets the number of reclaimable pages at the file tail.
/// Gets the number of reclaimable pages at the file tail.
/// </summary>
public uint TailReclaimablePages { get; init; }
}
@@ -178,11 +179,11 @@ public sealed class FragmentationMapReport
public sealed partial class StorageEngine
{
/// <summary>
/// Gets page usage grouped by page type.
/// Gets page usage grouped by page type.
/// </summary>
public IReadOnlyList<PageTypeUsageEntry> GetPageUsageByPageType()
{
var pageCount = _pageFile.NextPageId;
uint pageCount = _pageFile.NextPageId;
var buffer = new byte[_pageFile.PageSize];
var counts = new Dictionary<PageType, int>();
@@ -190,7 +191,7 @@ public sealed partial class StorageEngine
{
_pageFile.ReadPage(pageId, buffer);
var pageType = PageHeader.ReadFrom(buffer).PageType;
counts[pageType] = counts.TryGetValue(pageType, out var count) ? count + 1 : 1;
counts[pageType] = counts.TryGetValue(pageType, out int count) ? count + 1 : 1;
}
return counts
@@ -204,7 +205,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets per-collection page usage by resolving primary-index locations and related index roots.
/// Gets per-collection page usage by resolving primary-index locations and related index roots.
/// </summary>
public IReadOnlyList<CollectionPageUsageEntry> GetPageUsageByCollection()
{
@@ -221,27 +222,23 @@ public sealed partial class StorageEngine
pageIds.Add(metadata.SchemaRootPageId);
foreach (var indexMetadata in metadata.Indexes)
{
if (indexMetadata.RootPageId != 0)
pageIds.Add(indexMetadata.RootPageId);
}
foreach (var location in EnumeratePrimaryLocations(metadata))
{
pageIds.Add(location.PageId);
if (TryReadFirstOverflowPage(location, out var firstOverflowPage))
{
if (TryReadFirstOverflowPage(location, out uint firstOverflowPage))
AddOverflowChainPages(pageIds, firstOverflowPage);
}
}
int data = 0;
int overflow = 0;
int indexPages = 0;
int other = 0;
var data = 0;
var overflow = 0;
var indexPages = 0;
var other = 0;
var pageBuffer = new byte[_pageFile.PageSize];
foreach (var pageId in pageIds)
foreach (uint pageId in pageIds)
{
if (pageId >= _pageFile.NextPageId)
continue;
@@ -250,21 +247,13 @@ public sealed partial class StorageEngine
var pageType = PageHeader.ReadFrom(pageBuffer).PageType;
if (pageType == PageType.Data)
{
data++;
}
else if (pageType == PageType.Overflow)
{
overflow++;
}
else if (pageType == PageType.Index || pageType == PageType.Vector || pageType == PageType.Spatial)
{
indexPages++;
}
else
{
other++;
}
}
results.Add(new CollectionPageUsageEntry
@@ -282,7 +271,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets per-collection logical-vs-stored compression ratios.
/// Gets per-collection logical-vs-stored compression ratios.
/// </summary>
public IReadOnlyList<CollectionCompressionRatioEntry> GetCompressionRatioByCollection()
{
@@ -298,7 +287,8 @@ public sealed partial class StorageEngine
foreach (var location in EnumeratePrimaryLocations(metadata))
{
if (!TryReadSlotPayloadStats(location, out var isCompressed, out var originalBytes, out var storedBytes))
if (!TryReadSlotPayloadStats(location, out bool isCompressed, out int originalBytes,
out int storedBytes))
continue;
docs++;
@@ -323,7 +313,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets free-list summary for diagnostics.
/// Gets free-list summary for diagnostics.
/// </summary>
public FreeListSummary GetFreeListSummary()
{
@@ -339,12 +329,12 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets detailed page-level fragmentation diagnostics.
/// Gets detailed page-level fragmentation diagnostics.
/// </summary>
public FragmentationMapReport GetFragmentationMap()
{
var freePageSet = new HashSet<uint>(_pageFile.EnumerateFreePages(includeEmptyPages: true));
var pageCount = _pageFile.NextPageId;
var freePageSet = new HashSet<uint>(_pageFile.EnumerateFreePages());
uint pageCount = _pageFile.NextPageId;
var buffer = new byte[_pageFile.PageSize];
var pages = new List<FragmentationPageEntry>((int)pageCount);
@@ -354,17 +344,12 @@ public sealed partial class StorageEngine
{
_pageFile.ReadPage(pageId, buffer);
var pageHeader = PageHeader.ReadFrom(buffer);
var isFreePage = freePageSet.Contains(pageId);
bool isFreePage = freePageSet.Contains(pageId);
int freeBytes = 0;
var freeBytes = 0;
if (isFreePage)
{
freeBytes = _pageFile.PageSize;
}
else if (TryReadSlottedFreeSpace(buffer, out var slottedFreeBytes))
{
freeBytes = slottedFreeBytes;
}
else if (TryReadSlottedFreeSpace(buffer, out int slottedFreeBytes)) freeBytes = slottedFreeBytes;
totalFreeBytes += freeBytes;
@@ -378,7 +363,7 @@ public sealed partial class StorageEngine
}
uint tailReclaimablePages = 0;
for (var i = pageCount; i > 2; i--)
for (uint i = pageCount; i > 2; i--)
{
if (!freePageSet.Contains(i - 1))
break;
@@ -386,12 +371,12 @@ public sealed partial class StorageEngine
tailReclaimablePages++;
}
var fileBytes = Math.Max(1L, _pageFile.FileLengthBytes);
long fileBytes = Math.Max(1L, _pageFile.FileLengthBytes);
return new FragmentationMapReport
{
Pages = pages,
TotalFreeBytes = totalFreeBytes,
FragmentationPercent = (totalFreeBytes * 100d) / fileBytes,
FragmentationPercent = totalFreeBytes * 100d / fileBytes,
TailReclaimablePages = tailReclaimablePages
};
}
@@ -403,10 +388,8 @@ public sealed partial class StorageEngine
var index = new BTreeIndex(this, IndexOptions.CreateUnique("_id"), metadata.PrimaryRootPageId);
foreach (var entry in index.Range(IndexKey.MinKey, IndexKey.MaxKey, IndexDirection.Forward, transactionId: 0))
{
foreach (var entry in index.Range(IndexKey.MinKey, IndexKey.MaxKey, IndexDirection.Forward, 0))
yield return entry.Location;
}
}
private bool TryReadFirstOverflowPage(in DocumentLocation location, out uint firstOverflowPage)
@@ -419,7 +402,7 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
@@ -441,7 +424,7 @@ public sealed partial class StorageEngine
var buffer = new byte[_pageFile.PageSize];
var visited = new HashSet<uint>();
var current = firstOverflowPage;
uint current = firstOverflowPage;
while (current != 0 && current < _pageFile.NextPageId && visited.Add(current))
{
@@ -472,12 +455,12 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
var hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
isCompressed = (slot.Flags & SlotFlags.Compressed) != 0;
if (!hasOverflow)
@@ -492,7 +475,8 @@ public sealed partial class StorageEngine
if (slot.Length < CompressedPayloadHeader.Size)
return false;
var compressedHeader = CompressedPayloadHeader.ReadFrom(pageBuffer.AsSpan(slot.Offset, CompressedPayloadHeader.Size));
var compressedHeader =
CompressedPayloadHeader.ReadFrom(pageBuffer.AsSpan(slot.Offset, CompressedPayloadHeader.Size));
originalBytes = compressedHeader.OriginalLength;
return true;
}
@@ -501,7 +485,7 @@ public sealed partial class StorageEngine
return false;
var primaryPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
var totalStoredBytes = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
int totalStoredBytes = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
if (totalStoredBytes < 0)
return false;
@@ -522,8 +506,8 @@ public sealed partial class StorageEngine
else
{
storedPrefix.CopyTo(headerBuffer);
var copied = storedPrefix.Length;
var nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
int copied = storedPrefix.Length;
uint nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
var overflowBuffer = new byte[_pageFile.PageSize];
while (copied < CompressedPayloadHeader.Size && nextOverflow != 0 && nextOverflow < _pageFile.NextPageId)
@@ -533,7 +517,8 @@ public sealed partial class StorageEngine
if (overflowHeader.PageType != PageType.Overflow)
return false;
var available = Math.Min(CompressedPayloadHeader.Size - copied, _pageFile.PageSize - SlottedPageHeader.Size);
int available = Math.Min(CompressedPayloadHeader.Size - copied,
_pageFile.PageSize - SlottedPageHeader.Size);
overflowBuffer.AsSpan(SlottedPageHeader.Size, available).CopyTo(headerBuffer.Slice(copied));
copied += available;
nextOverflow = overflowHeader.NextOverflowPage;
src/CBDD.Core/Storage/StorageEngine.Dictionary.cs
+87 -92
View File
@@ -1,17 +1,92 @@
using System.Collections.Concurrent;
using System.Text;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
private readonly ConcurrentDictionary<string, ushort> _dictionaryCache = new(StringComparer.OrdinalIgnoreCase);
// Lock for dictionary modifications (simple lock for now, could be RW lock)
private readonly object _dictionaryLock = new();
private readonly ConcurrentDictionary<ushort, string> _dictionaryReverseCache = new();
private uint _dictionaryRootPageId;
private ushort _nextDictionaryId;
// Lock for dictionary modifications (simple lock for now, could be RW lock)
private readonly object _dictionaryLock = new();
/// <summary>
/// Gets the key-to-id dictionary cache.
/// </summary>
/// <returns>The key-to-id map.</returns>
public ConcurrentDictionary<string, ushort> GetKeyMap()
{
return _dictionaryCache;
}
/// <summary>
/// Gets the id-to-key dictionary cache.
/// </summary>
/// <returns>The id-to-key map.</returns>
public ConcurrentDictionary<ushort, string> GetKeyReverseMap()
{
return _dictionaryReverseCache;
}
/// <summary>
/// Gets the ID for a dictionary key, creating it if it doesn't exist.
/// Thread-safe.
/// </summary>
/// <param name="key">The dictionary key.</param>
/// <returns>The dictionary identifier for the key.</returns>
public ushort GetOrAddDictionaryEntry(string key)
{
key = key.ToLowerInvariant();
if (_dictionaryCache.TryGetValue(key, out ushort id)) return id;
lock (_dictionaryLock)
{
// Double checked locking
if (_dictionaryCache.TryGetValue(key, out id)) return id;
// Try to find in storage (in case cache is incomplete or another process?)
// Note: FindAllGlobal loaded everything, so strict cache miss means it's not in DB.
// BUT if we support concurrent writers (multiple processed), we should re-check DB.
// Current CBDD seems to be single-process exclusive lock (FileShare.None).
// So in-memory cache is authoritative after load.
// Generate New ID
ushort nextId = _nextDictionaryId;
if (nextId == 0) nextId = DictionaryPage.ReservedValuesEnd + 1; // Should be init, but safety
// Insert into Page
// usage of default(ulong) or null transaction?
// Dictionary updates should ideally be transactional or immediate?
// "Immediate" for now to simplify, as dictionary is cross-collection.
// If we use transaction, we need to pass it in. For now, immediate write.
// We need to support "Insert Global" which handles overflow.
// DictionaryPage.Insert only handles single page.
// We need logic here to traverse chain and find space.
if (InsertDictionaryEntryGlobal(key, nextId))
{
_dictionaryCache[key] = nextId;
_dictionaryReverseCache[nextId] = key;
_nextDictionaryId++;
return nextId;
}
throw new InvalidOperationException("Failed to insert dictionary entry (Storage Full?)");
}
}
/// <summary>
/// Registers a set of keys in the global dictionary.
/// Ensures all keys are assigned an ID and persisted.
/// </summary>
/// <param name="keys">The keys to register.</param>
public void RegisterKeys(IEnumerable<string> keys)
{
foreach (string key in keys) GetOrAddDictionaryEntry(key.ToLowerInvariant());
}
private void InitializeDictionary()
{
@@ -57,13 +132,14 @@ public sealed partial class StorageEngine
// Warm cache
ushort maxId = DictionaryPage.ReservedValuesEnd;
foreach (var (key, val) in DictionaryPage.FindAllGlobal(this, _dictionaryRootPageId))
foreach ((string key, ushort val) in DictionaryPage.FindAllGlobal(this, _dictionaryRootPageId))
{
var lowerKey = key.ToLowerInvariant();
string lowerKey = key.ToLowerInvariant();
_dictionaryCache[lowerKey] = val;
_dictionaryReverseCache[val] = lowerKey;
if (val > maxId) maxId = val;
}
_nextDictionaryId = (ushort)(maxId + 1);
}
@@ -72,93 +148,25 @@ public sealed partial class StorageEngine
// Pre-register common array indices to avoid mapping during high-frequency writes
var indices = new List<string>(101);
for (int i = 0; i <= 100; i++) indices.Add(i.ToString());
for (var i = 0; i <= 100; i++) indices.Add(i.ToString());
RegisterKeys(indices);
}
/// <summary>
/// Gets the key-to-id dictionary cache.
/// </summary>
/// <returns>The key-to-id map.</returns>
public ConcurrentDictionary<string, ushort> GetKeyMap() => _dictionaryCache;
/// <summary>
/// Gets the id-to-key dictionary cache.
/// </summary>
/// <returns>The id-to-key map.</returns>
public ConcurrentDictionary<ushort, string> GetKeyReverseMap() => _dictionaryReverseCache;
/// <summary>
/// Gets the ID for a dictionary key, creating it if it doesn't exist.
/// Thread-safe.
/// </summary>
/// <param name="key">The dictionary key.</param>
/// <returns>The dictionary identifier for the key.</returns>
public ushort GetOrAddDictionaryEntry(string key)
{
key = key.ToLowerInvariant();
if (_dictionaryCache.TryGetValue(key, out var id))
{
return id;
}
lock (_dictionaryLock)
{
// Double checked locking
if (_dictionaryCache.TryGetValue(key, out id))
{
return id;
}
// Try to find in storage (in case cache is incomplete or another process?)
// Note: FindAllGlobal loaded everything, so strict cache miss means it's not in DB.
// BUT if we support concurrent writers (multiple processed), we should re-check DB.
// Current CBDD seems to be single-process exclusive lock (FileShare.None).
// So in-memory cache is authoritative after load.
// Generate New ID
ushort nextId = _nextDictionaryId;
if (nextId == 0) nextId = DictionaryPage.ReservedValuesEnd + 1; // Should be init, but safety
// Insert into Page
// usage of default(ulong) or null transaction?
// Dictionary updates should ideally be transactional or immediate?
// "Immediate" for now to simplify, as dictionary is cross-collection.
// If we use transaction, we need to pass it in. For now, immediate write.
// We need to support "Insert Global" which handles overflow.
// DictionaryPage.Insert only handles single page.
// We need logic here to traverse chain and find space.
if (InsertDictionaryEntryGlobal(key, nextId))
{
_dictionaryCache[key] = nextId;
_dictionaryReverseCache[nextId] = key;
_nextDictionaryId++;
return nextId;
}
else
{
throw new InvalidOperationException("Failed to insert dictionary entry (Storage Full?)");
}
}
}
/// <summary>
/// Gets the dictionary key for an identifier.
/// Gets the dictionary key for an identifier.
/// </summary>
/// <param name="id">The dictionary identifier.</param>
/// <returns>The dictionary key if found; otherwise, <see langword="null"/>.</returns>
/// <returns>The dictionary key if found; otherwise, <see langword="null" />.</returns>
public string? GetDictionaryKey(ushort id)
{
if (_dictionaryReverseCache.TryGetValue(id, out var key))
if (_dictionaryReverseCache.TryGetValue(id, out string? key))
return key;
return null;
}
private bool InsertDictionaryEntryGlobal(string key, ushort value)
{
var pageId = _dictionaryRootPageId;
uint pageId = _dictionaryRootPageId;
var pageBuffer = new byte[PageSize];
while (true)
@@ -182,7 +190,7 @@ public sealed partial class StorageEngine
}
// No Next Page - Allocate New
var newPageId = AllocatePage();
uint newPageId = AllocatePage();
var newPageBuffer = new byte[PageSize];
DictionaryPage.Initialize(newPageBuffer, newPageId);
@@ -203,17 +211,4 @@ public sealed partial class StorageEngine
return true;
}
}
/// <summary>
/// Registers a set of keys in the global dictionary.
/// Ensures all keys are assigned an ID and persisted.
/// </summary>
/// <param name="keys">The keys to register.</param>
public void RegisterKeys(IEnumerable<string> keys)
{
foreach (var key in keys)
{
GetOrAddDictionaryEntry(key.ToLowerInvariant());
}
}
}
src/CBDD.Core/Storage/StorageEngine.Format.cs
+17 -14
View File
@@ -15,31 +15,32 @@ internal readonly struct StorageFormatMetadata
internal const int WireSize = 16;
/// <summary>
/// Gets a value indicating whether format metadata is present.
/// Gets a value indicating whether format metadata is present.
/// </summary>
public bool IsPresent { get; }
/// <summary>
/// Gets the storage format version.
/// Gets the storage format version.
/// </summary>
public byte Version { get; }
/// <summary>
/// Gets enabled storage feature flags.
/// Gets enabled storage feature flags.
/// </summary>
public StorageFeatureFlags FeatureFlags { get; }
/// <summary>
/// Gets the default compression codec.
/// Gets the default compression codec.
/// </summary>
public CompressionCodec DefaultCodec { get; }
/// <summary>
/// Gets a value indicating whether compression capability is enabled.
/// Gets a value indicating whether compression capability is enabled.
/// </summary>
public bool CompressionCapabilityEnabled => (FeatureFlags & StorageFeatureFlags.CompressionCapability) != 0;
private StorageFormatMetadata(bool isPresent, byte version, StorageFeatureFlags featureFlags, CompressionCodec defaultCodec)
private StorageFormatMetadata(bool isPresent, byte version, StorageFeatureFlags featureFlags,
CompressionCodec defaultCodec)
{
IsPresent = isPresent;
Version = version;
@@ -48,18 +49,19 @@ internal readonly struct StorageFormatMetadata
}
/// <summary>
/// Creates metadata representing a modern format-aware file.
/// Creates metadata representing a modern format-aware file.
/// </summary>
/// <param name="version">The storage format version.</param>
/// <param name="featureFlags">Enabled feature flags.</param>
/// <param name="defaultCodec">The default compression codec.</param>
public static StorageFormatMetadata Present(byte version, StorageFeatureFlags featureFlags, CompressionCodec defaultCodec)
public static StorageFormatMetadata Present(byte version, StorageFeatureFlags featureFlags,
CompressionCodec defaultCodec)
{
return new StorageFormatMetadata(true, version, featureFlags, defaultCodec);
}
/// <summary>
/// Creates metadata representing a legacy file without format metadata.
/// Creates metadata representing a legacy file without format metadata.
/// </summary>
/// <param name="defaultCodec">The default compression codec.</param>
public static StorageFormatMetadata Legacy(CompressionCodec defaultCodec)
@@ -88,12 +90,13 @@ public sealed partial class StorageEngine
return metadata;
if (!_pageFile.WasCreated)
return StorageFormatMetadata.Legacy(_compressionOptions.Codec);
return StorageFormatMetadata.Legacy(CompressionOptions.Codec);
var featureFlags = _compressionOptions.EnableCompression
var featureFlags = CompressionOptions.EnableCompression
? StorageFeatureFlags.CompressionCapability
: StorageFeatureFlags.None;
var initialMetadata = StorageFormatMetadata.Present(CurrentStorageFormatVersion, featureFlags, _compressionOptions.Codec);
var initialMetadata =
StorageFormatMetadata.Present(CurrentStorageFormatVersion, featureFlags, CompressionOptions.Codec);
WriteStorageFormatMetadata(initialMetadata);
return initialMetadata;
}
@@ -104,11 +107,11 @@ public sealed partial class StorageEngine
if (source.Length < StorageFormatMetadata.WireSize)
return false;
var magic = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(0, 4));
uint magic = BinaryPrimitives.ReadUInt32LittleEndian(source.Slice(0, 4));
if (magic != StorageFormatMagic)
return false;
var version = source[4];
byte version = source[4];
var featureFlags = (StorageFeatureFlags)source[5];
var codec = (CompressionCodec)source[6];
if (!Enum.IsDefined(codec))
src/CBDD.Core/Storage/StorageEngine.Maintenance.cs
+510 -613
View File
File diff suppressed because it is too large Load Diff
src/CBDD.Core/Storage/StorageEngine.Memory.cs
+2 -4
View File
@@ -1,11 +1,9 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Allocates a new page.
/// Allocates a new page.
/// </summary>
/// <returns>Page ID of the allocated page</returns>
public uint AllocatePage()
@@ -14,7 +12,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Frees a page.
/// Frees a page.
/// </summary>
/// <param name="pageId">Page to free</param>
public void FreePage(uint pageId)
src/CBDD.Core/Storage/StorageEngine.Migration.cs
+62 -60
View File
@@ -5,98 +5,98 @@ using ZB.MOM.WW.CBDD.Core.Compression;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Options controlling compression migration.
/// Options controlling compression migration.
/// </summary>
public sealed class CompressionMigrationOptions
{
/// <summary>
/// Enables dry-run estimation without mutating database contents.
/// Enables dry-run estimation without mutating database contents.
/// </summary>
public bool DryRun { get; init; } = true;
/// <summary>
/// Target codec for migrated payloads.
/// Target codec for migrated payloads.
/// </summary>
public CompressionCodec Codec { get; init; } = CompressionCodec.Brotli;
/// <summary>
/// Target compression level.
/// Target compression level.
/// </summary>
public CompressionLevel Level { get; init; } = CompressionLevel.Fastest;
/// <summary>
/// Minimum logical payload size required before compression is attempted.
/// Minimum logical payload size required before compression is attempted.
/// </summary>
public int MinSizeBytes { get; init; } = 1024;
/// <summary>
/// Minimum savings percent required to keep compressed output.
/// Minimum savings percent required to keep compressed output.
/// </summary>
public int MinSavingsPercent { get; init; } = 10;
/// <summary>
/// Optional include-only collection list (case-insensitive).
/// Optional include-only collection list (case-insensitive).
/// </summary>
public IReadOnlyList<string>? IncludeCollections { get; init; }
/// <summary>
/// Optional exclusion collection list (case-insensitive).
/// Optional exclusion collection list (case-insensitive).
/// </summary>
public IReadOnlyList<string>? ExcludeCollections { get; init; }
}
/// <summary>
/// Result of a compression migration run.
/// Result of a compression migration run.
/// </summary>
public sealed class CompressionMigrationResult
{
/// <summary>
/// Gets a value indicating whether this run was executed in dry-run mode.
/// Gets a value indicating whether this run was executed in dry-run mode.
/// </summary>
public bool DryRun { get; init; }
/// <summary>
/// Gets the target codec used for migration output.
/// Gets the target codec used for migration output.
/// </summary>
public CompressionCodec Codec { get; init; }
/// <summary>
/// Gets the target compression level used for migration output.
/// Gets the target compression level used for migration output.
/// </summary>
public CompressionLevel Level { get; init; }
/// <summary>
/// Gets the number of collections processed.
/// Gets the number of collections processed.
/// </summary>
public int CollectionsProcessed { get; init; }
/// <summary>
/// Gets the number of documents scanned.
/// Gets the number of documents scanned.
/// </summary>
public long DocumentsScanned { get; init; }
/// <summary>
/// Gets the number of documents rewritten.
/// Gets the number of documents rewritten.
/// </summary>
public long DocumentsRewritten { get; init; }
/// <summary>
/// Gets the number of documents skipped.
/// Gets the number of documents skipped.
/// </summary>
public long DocumentsSkipped { get; init; }
/// <summary>
/// Gets the total logical bytes observed before migration decisions.
/// Gets the total logical bytes observed before migration decisions.
/// </summary>
public long BytesBefore { get; init; }
/// <summary>
/// Gets the estimated total stored bytes after migration.
/// Gets the estimated total stored bytes after migration.
/// </summary>
public long BytesEstimatedAfter { get; init; }
/// <summary>
/// Gets the actual total stored bytes after migration when not in dry-run mode.
/// Gets the actual total stored bytes after migration when not in dry-run mode.
/// </summary>
public long BytesActualAfter { get; init; }
}
@@ -104,7 +104,7 @@ public sealed class CompressionMigrationResult
public sealed partial class StorageEngine
{
/// <summary>
/// Estimates or applies a one-time compression migration.
/// Estimates or applies a one-time compression migration.
/// </summary>
/// <param name="options">Optional compression migration options.</param>
public CompressionMigrationResult MigrateCompression(CompressionMigrationOptions? options = null)
@@ -113,11 +113,12 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Estimates or applies a one-time compression migration.
/// Estimates or applies a one-time compression migration.
/// </summary>
/// <param name="options">Optional compression migration options.</param>
/// <param name="ct">A token used to cancel the operation.</param>
public async Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null, CancellationToken ct = default)
public async Task<CompressionMigrationResult> MigrateCompressionAsync(CompressionMigrationOptions? options = null,
CancellationToken ct = default)
{
var normalized = NormalizeMigrationOptions(options);
@@ -147,13 +148,13 @@ public sealed partial class StorageEngine
{
ct.ThrowIfCancellationRequested();
if (!TryReadStoredPayload(location, out var storedPayload, out var isCompressed))
if (!TryReadStoredPayload(location, out byte[] storedPayload, out bool isCompressed))
{
docsSkipped++;
continue;
}
if (!TryGetLogicalPayload(storedPayload, isCompressed, out var logicalPayload))
if (!TryGetLogicalPayload(storedPayload, isCompressed, out byte[] logicalPayload))
{
docsSkipped++;
continue;
@@ -162,15 +163,14 @@ public sealed partial class StorageEngine
docsScanned++;
bytesBefore += logicalPayload.Length;
var targetStored = BuildTargetStoredPayload(logicalPayload, normalized, out var targetCompressed);
byte[] targetStored =
BuildTargetStoredPayload(logicalPayload, normalized, out bool targetCompressed);
bytesEstimatedAfter += targetStored.Length;
if (normalized.DryRun)
{
continue;
}
if (normalized.DryRun) continue;
if (!TryRewriteStoredPayloadAtLocation(location, targetStored, targetCompressed, out var actualStoredBytes))
if (!TryRewriteStoredPayloadAtLocation(location, targetStored, targetCompressed,
out int actualStoredBytes))
{
docsSkipped++;
continue;
@@ -184,9 +184,9 @@ public sealed partial class StorageEngine
if (!normalized.DryRun)
{
var metadata = StorageFormatMetadata.Present(
version: 1,
featureFlags: StorageFeatureFlags.CompressionCapability,
defaultCodec: normalized.Codec);
1,
StorageFeatureFlags.CompressionCapability,
normalized.Codec);
WriteStorageFormatMetadata(metadata);
_pageFile.Flush();
}
@@ -221,7 +221,8 @@ public sealed partial class StorageEngine
var normalized = options ?? new CompressionMigrationOptions();
if (!Enum.IsDefined(normalized.Codec) || normalized.Codec == CompressionCodec.None)
throw new ArgumentOutOfRangeException(nameof(options), "Migration codec must be a supported non-None codec.");
throw new ArgumentOutOfRangeException(nameof(options),
"Migration codec must be a supported non-None codec.");
if (normalized.MinSizeBytes < 0)
throw new ArgumentOutOfRangeException(nameof(options), "MinSizeBytes must be non-negative.");
@@ -250,7 +251,8 @@ public sealed partial class StorageEngine
.ToList();
}
private byte[] BuildTargetStoredPayload(ReadOnlySpan<byte> logicalPayload, CompressionMigrationOptions options, out bool compressed)
private byte[] BuildTargetStoredPayload(ReadOnlySpan<byte> logicalPayload, CompressionMigrationOptions options,
out bool compressed)
{
compressed = false;
@@ -259,10 +261,10 @@ public sealed partial class StorageEngine
try
{
var compressedPayload = _compressionService.Compress(logicalPayload, options.Codec, options.Level);
var storedLength = CompressedPayloadHeader.Size + compressedPayload.Length;
var savings = logicalPayload.Length - storedLength;
var savingsPercent = logicalPayload.Length == 0 ? 0 : (int)((savings * 100L) / logicalPayload.Length);
byte[] compressedPayload = CompressionService.Compress(logicalPayload, options.Codec, options.Level);
int storedLength = CompressedPayloadHeader.Size + compressedPayload.Length;
int savings = logicalPayload.Length - storedLength;
int savingsPercent = logicalPayload.Length == 0 ? 0 : (int)(savings * 100L / logicalPayload.Length);
if (savings <= 0 || savingsPercent < options.MinSavingsPercent)
return logicalPayload.ToArray();
@@ -308,11 +310,11 @@ public sealed partial class StorageEngine
try
{
logicalPayload = _compressionService.Decompress(
logicalPayload = CompressionService.Decompress(
compressedPayload,
header.Codec,
header.OriginalLength,
Math.Max(header.OriginalLength, _compressionOptions.MaxDecompressedSizeBytes));
Math.Max(header.OriginalLength, CompressionOptions.MaxDecompressedSizeBytes));
return true;
}
catch
@@ -336,13 +338,13 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= header.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
isCompressed = (slot.Flags & SlotFlags.Compressed) != 0;
var hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool hasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
if (!hasOverflow)
{
@@ -354,14 +356,14 @@ public sealed partial class StorageEngine
return false;
var primaryPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
var totalStoredLength = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
var nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
int totalStoredLength = BinaryPrimitives.ReadInt32LittleEndian(primaryPayload.Slice(0, 4));
uint nextOverflow = BinaryPrimitives.ReadUInt32LittleEndian(primaryPayload.Slice(4, 4));
if (totalStoredLength < 0)
return false;
var output = new byte[totalStoredLength];
var primaryChunk = primaryPayload.Slice(8);
var copied = Math.Min(primaryChunk.Length, output.Length);
int copied = Math.Min(primaryChunk.Length, output.Length);
primaryChunk.Slice(0, copied).CopyTo(output);
var overflowBuffer = new byte[_pageFile.PageSize];
@@ -372,7 +374,7 @@ public sealed partial class StorageEngine
if (overflowHeader.PageType != PageType.Overflow)
return false;
var chunk = Math.Min(output.Length - copied, _pageFile.PageSize - SlottedPageHeader.Size);
int chunk = Math.Min(output.Length - copied, _pageFile.PageSize - SlottedPageHeader.Size);
overflowBuffer.AsSpan(SlottedPageHeader.Size, chunk).CopyTo(output.AsSpan(copied));
copied += chunk;
nextOverflow = overflowHeader.NextOverflowPage;
@@ -403,12 +405,12 @@ public sealed partial class StorageEngine
if (location.SlotIndex >= pageHeader.SlotCount)
return false;
var slotOffset = SlottedPageHeader.Size + (location.SlotIndex * SlotEntry.Size);
int slotOffset = SlottedPageHeader.Size + location.SlotIndex * SlotEntry.Size;
var slot = SlotEntry.ReadFrom(pageBuffer.AsSpan(slotOffset, SlotEntry.Size));
if ((slot.Flags & SlotFlags.Deleted) != 0)
return false;
var oldHasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
bool oldHasOverflow = (slot.Flags & SlotFlags.HasOverflow) != 0;
uint oldOverflowHead = 0;
if (oldHasOverflow)
{
@@ -442,12 +444,12 @@ public sealed partial class StorageEngine
if (slot.Length < 8)
return false;
var primaryChunkSize = slot.Length - 8;
int primaryChunkSize = slot.Length - 8;
if (primaryChunkSize < 0)
return false;
var remainder = newStoredPayload.Slice(primaryChunkSize);
var newOverflowHead = BuildOverflowChainForMigration(remainder);
uint newOverflowHead = BuildOverflowChainForMigration(remainder);
var slotPayload = pageBuffer.AsSpan(slot.Offset, slot.Length);
slotPayload.Clear();
@@ -475,22 +477,22 @@ public sealed partial class StorageEngine
if (overflowPayload.IsEmpty)
return 0;
var chunkSize = _pageFile.PageSize - SlottedPageHeader.Size;
int chunkSize = _pageFile.PageSize - SlottedPageHeader.Size;
uint nextOverflowPageId = 0;
var tailSize = overflowPayload.Length % chunkSize;
var fullPages = overflowPayload.Length / chunkSize;
int tailSize = overflowPayload.Length % chunkSize;
int fullPages = overflowPayload.Length / chunkSize;
if (tailSize > 0)
{
var tailOffset = fullPages * chunkSize;
int tailOffset = fullPages * chunkSize;
var tailSlice = overflowPayload.Slice(tailOffset, tailSize);
nextOverflowPageId = AllocateOverflowPageForMigration(tailSlice, nextOverflowPageId);
}
for (var i = fullPages - 1; i >= 0; i--)
for (int i = fullPages - 1; i >= 0; i--)
{
var chunkOffset = i * chunkSize;
int chunkOffset = i * chunkSize;
var chunk = overflowPayload.Slice(chunkOffset, chunkSize);
nextOverflowPageId = AllocateOverflowPageForMigration(chunk, nextOverflowPageId);
}
@@ -500,7 +502,7 @@ public sealed partial class StorageEngine
private uint AllocateOverflowPageForMigration(ReadOnlySpan<byte> payloadChunk, uint nextOverflowPageId)
{
var pageId = _pageFile.AllocatePage();
uint pageId = _pageFile.AllocatePage();
var buffer = new byte[_pageFile.PageSize];
var header = new SlottedPageHeader
@@ -524,13 +526,13 @@ public sealed partial class StorageEngine
{
var buffer = new byte[_pageFile.PageSize];
var visited = new HashSet<uint>();
var current = firstOverflowPage;
uint current = firstOverflowPage;
while (current != 0 && current < _pageFile.NextPageId && visited.Add(current))
{
_pageFile.ReadPage(current, buffer);
var header = SlottedPageHeader.ReadFrom(buffer);
var next = header.NextOverflowPage;
uint next = header.NextOverflowPage;
_pageFile.FreePage(current);
current = next;
}
src/CBDD.Core/Storage/StorageEngine.Pages.cs
+18 -18
View File
@@ -1,14 +1,14 @@
using ZB.MOM.WW.CBDD.Core.Transactions;
using System.Collections.Concurrent;
namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Reads a page with transaction isolation.
/// 1. Check WAL cache for uncommitted writes (Read Your Own Writes)
/// 2. Check WAL index for committed writes (lazy replay)
/// 3. Read from PageFile (committed baseline)
/// Reads a page with transaction isolation.
/// 1. Check WAL cache for uncommitted writes (Read Your Own Writes)
/// 2. Check WAL index for committed writes (lazy replay)
/// 3. Read from PageFile (committed baseline)
/// </summary>
/// <param name="pageId">Page to read</param>
/// <param name="transactionId">Optional transaction ID for isolation</param>
@@ -20,17 +20,17 @@ public sealed partial class StorageEngine
if (transactionId.HasValue &&
transactionId.Value != 0 &&
_walCache.TryGetValue(transactionId.Value, out var txnPages) &&
txnPages.TryGetValue(pageId, out var uncommittedData))
txnPages.TryGetValue(pageId, out byte[]? uncommittedData))
{
var length = Math.Min(uncommittedData.Length, destination.Length);
int length = Math.Min(uncommittedData.Length, destination.Length);
uncommittedData.AsSpan(0, length).CopyTo(destination);
return;
}
// 2. Check WAL index (committed but not checkpointed)
if (_walIndex.TryGetValue(pageId, out var committedData))
if (_walIndex.TryGetValue(pageId, out byte[]? committedData))
{
var length = Math.Min(committedData.Length, destination.Length);
int length = Math.Min(committedData.Length, destination.Length);
committedData.AsSpan(0, length).CopyTo(destination);
return;
}
@@ -40,9 +40,9 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Writes a page within a transaction.
/// Data goes to WAL cache immediately and becomes visible to that transaction only.
/// Will be written to WAL on commit.
/// Writes a page within a transaction.
/// Data goes to WAL cache immediately and becomes visible to that transaction only.
/// Will be written to WAL on commit.
/// </summary>
/// <param name="pageId">Page to write</param>
/// <param name="transactionId">Transaction ID owning this write</param>
@@ -54,16 +54,16 @@ public sealed partial class StorageEngine
// Get or create transaction-local cache
var txnPages = _walCache.GetOrAdd(transactionId,
_ => new System.Collections.Concurrent.ConcurrentDictionary<uint, byte[]>());
_ => new ConcurrentDictionary<uint, byte[]>());
// Store defensive copy
var copy = data.ToArray();
byte[] copy = data.ToArray();
txnPages[pageId] = copy;
}
/// <summary>
/// Writes a page immediately to disk (non-transactional).
/// Used for initialization and metadata updates outside of transactions.
/// Writes a page immediately to disk (non-transactional).
/// Used for initialization and metadata updates outside of transactions.
/// </summary>
/// <param name="pageId">Page to write</param>
/// <param name="data">Page data</param>
@@ -73,8 +73,8 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Gets the number of pages currently allocated in the page file.
/// Useful for full database scans.
/// Gets the number of pages currently allocated in the page file.
/// Useful for full database scans.
/// </summary>
public uint PageCount => _pageFile.NextPageId;
}
src/CBDD.Core/Storage/StorageEngine.Recovery.cs
+29 -50
View File
@@ -5,7 +5,7 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Gets the current size of the WAL file.
/// Gets the current size of the WAL file.
/// </summary>
public long GetWalSize()
{
@@ -13,8 +13,8 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Truncates the WAL file.
/// Should only be called after a successful checkpoint.
/// Truncates the WAL file.
/// Should only be called after a successful checkpoint.
/// </summary>
public void TruncateWal()
{
@@ -22,7 +22,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Flushes the WAL to disk.
/// Flushes the WAL to disk.
/// </summary>
public void FlushWal()
{
@@ -30,7 +30,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Performs a truncate checkpoint by default.
/// Performs a truncate checkpoint by default.
/// </summary>
public void Checkpoint()
{
@@ -38,7 +38,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Performs a checkpoint using the requested mode.
/// Performs a checkpoint using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <returns>The checkpoint execution result.</returns>
@@ -50,7 +50,7 @@ public sealed partial class StorageEngine
lockAcquired = _commitLock.Wait(0);
if (!lockAcquired)
{
var walSize = _wal.GetCurrentSize();
long walSize = _wal.GetCurrentSize();
return new CheckpointResult(mode, false, 0, walSize, walSize, false, false);
}
}
@@ -66,19 +66,18 @@ public sealed partial class StorageEngine
}
finally
{
if (lockAcquired)
{
_commitLock.Release();
}
if (lockAcquired) _commitLock.Release();
}
}
private void CheckpointInternal()
=> _ = CheckpointInternal(CheckpointMode.Truncate);
{
_ = CheckpointInternal(CheckpointMode.Truncate);
}
private CheckpointResult CheckpointInternal(CheckpointMode mode)
{
var walBytesBefore = _wal.GetCurrentSize();
long walBytesBefore = _wal.GetCurrentSize();
var appliedPages = 0;
var truncated = false;
var restarted = false;
@@ -91,10 +90,7 @@ public sealed partial class StorageEngine
}
// 2. Flush PageFile to ensure durability.
if (appliedPages > 0)
{
_pageFile.Flush();
}
if (appliedPages > 0) _pageFile.Flush();
// 3. Clear in-memory WAL index (now persisted).
_walIndex.Clear();
@@ -109,6 +105,7 @@ public sealed partial class StorageEngine
_wal.WriteCheckpointRecord();
_wal.Flush();
}
break;
case CheckpointMode.Truncate:
if (walBytesBefore > 0)
@@ -116,6 +113,7 @@ public sealed partial class StorageEngine
_wal.Truncate();
truncated = true;
}
break;
case CheckpointMode.Restart:
_wal.Restart();
@@ -126,12 +124,12 @@ public sealed partial class StorageEngine
throw new ArgumentOutOfRangeException(nameof(mode), mode, "Unsupported checkpoint mode.");
}
var walBytesAfter = _wal.GetCurrentSize();
long walBytesAfter = _wal.GetCurrentSize();
return new CheckpointResult(mode, true, appliedPages, walBytesBefore, walBytesAfter, truncated, restarted);
}
/// <summary>
/// Performs a truncate checkpoint asynchronously by default.
/// Performs a truncate checkpoint asynchronously by default.
/// </summary>
/// <param name="ct">The cancellation token.</param>
public async Task CheckpointAsync(CancellationToken ct = default)
@@ -140,7 +138,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Performs a checkpoint asynchronously using the requested mode.
/// Performs a checkpoint asynchronously using the requested mode.
/// </summary>
/// <param name="mode">Checkpoint mode to execute.</param>
/// <param name="ct">The cancellation token.</param>
@@ -153,7 +151,7 @@ public sealed partial class StorageEngine
lockAcquired = await _commitLock.WaitAsync(0, ct);
if (!lockAcquired)
{
var walSize = _wal.GetCurrentSize();
long walSize = _wal.GetCurrentSize();
return new CheckpointResult(mode, false, 0, walSize, walSize, false, false);
}
}
@@ -170,16 +168,13 @@ public sealed partial class StorageEngine
}
finally
{
if (lockAcquired)
{
_commitLock.Release();
}
if (lockAcquired) _commitLock.Release();
}
}
/// <summary>
/// Recovers from crash by replaying WAL.
/// Applies committed transactions to PageFile in deterministic WAL order, then truncates WAL.
/// Recovers from crash by replaying WAL.
/// Applies committed transactions to PageFile in deterministic WAL order, then truncates WAL.
/// </summary>
public void Recover()
{
@@ -189,35 +184,28 @@ public sealed partial class StorageEngine
// 1. Read WAL and locate the latest checkpoint boundary.
var records = _wal.ReadAll();
var startIndex = 0;
for (var i = records.Count - 1; i >= 0; i--)
{
for (int i = records.Count - 1; i >= 0; i--)
if (records[i].Type == WalRecordType.Checkpoint)
{
startIndex = i + 1;
break;
}
}
// 2. Replay WAL in source order with deterministic commit application.
var pendingWrites = new Dictionary<ulong, List<(uint pageId, byte[] data)>>();
var appliedAny = false;
for (var i = startIndex; i < records.Count; i++)
for (int i = startIndex; i < records.Count; i++)
{
var record = records[i];
switch (record.Type)
{
case WalRecordType.Begin:
if (!pendingWrites.ContainsKey(record.TransactionId))
{
pendingWrites[record.TransactionId] = new List<(uint, byte[])>();
}
break;
case WalRecordType.Write:
if (record.AfterImage == null)
{
break;
}
if (record.AfterImage == null) break;
if (!pendingWrites.TryGetValue(record.TransactionId, out var writes))
{
@@ -228,12 +216,9 @@ public sealed partial class StorageEngine
writes.Add((record.PageId, record.AfterImage));
break;
case WalRecordType.Commit:
if (!pendingWrites.TryGetValue(record.TransactionId, out var committedWrites))
{
break;
}
if (!pendingWrites.TryGetValue(record.TransactionId, out var committedWrites)) break;
foreach (var (pageId, data) in committedWrites)
foreach ((uint pageId, byte[] data) in committedWrites)
{
_pageFile.WritePage(pageId, data);
appliedAny = true;
@@ -251,19 +236,13 @@ public sealed partial class StorageEngine
}
// 3. Flush PageFile to ensure durability.
if (appliedAny)
{
_pageFile.Flush();
}
if (appliedAny) _pageFile.Flush();
// 4. Clear in-memory WAL index (redundant since we just recovered).
_walIndex.Clear();
// 5. Truncate WAL (all changes now in PageFile).
if (_wal.GetCurrentSize() > 0)
{
_wal.Truncate();
}
if (_wal.GetCurrentSize() > 0) _wal.Truncate();
}
finally
{
src/CBDD.Core/Storage/StorageEngine.Schema.cs
+6 -8
View File
@@ -1,5 +1,3 @@
using System;
using System.Collections.Generic;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Bson.Schema;
@@ -8,7 +6,7 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
/// <summary>
/// Reads all schemas from the schema page chain.
/// Reads all schemas from the schema page chain.
/// </summary>
/// <param name="rootPageId">The root page identifier of the schema chain.</param>
/// <returns>The list of schemas in chain order.</returns>
@@ -17,7 +15,7 @@ public sealed partial class StorageEngine
var schemas = new List<BsonSchema>();
if (rootPageId == 0) return schemas;
var pageId = rootPageId;
uint pageId = rootPageId;
var buffer = new byte[PageSize];
while (pageId != 0)
@@ -33,7 +31,7 @@ public sealed partial class StorageEngine
var reader = new BsonSpanReader(buffer.AsSpan(32, used), GetKeyReverseMap());
while (reader.Remaining >= 4)
{
var docSize = reader.PeekInt32();
int docSize = reader.PeekInt32();
if (docSize <= 0 || docSize > reader.Remaining) break;
var schema = BsonSchema.FromBson(ref reader);
@@ -48,7 +46,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Appends a new schema version. Returns the root page ID (which might be new if it was 0 initially).
/// Appends a new schema version. Returns the root page ID (which might be new if it was 0 initially).
/// </summary>
/// <param name="rootPageId">The root page identifier of the schema chain.</param>
/// <param name="schema">The schema to append.</param>
@@ -60,7 +58,7 @@ public sealed partial class StorageEngine
var tempBuffer = new byte[PageSize];
var tempWriter = new BsonSpanWriter(tempBuffer, GetKeyMap());
schema.ToBson(ref tempWriter);
var schemaSize = tempWriter.Position;
int schemaSize = tempWriter.Position;
if (rootPageId == 0)
{
@@ -106,7 +104,7 @@ public sealed partial class StorageEngine
else
{
// Allocate new page
var newPageId = AllocatePage();
uint newPageId = AllocatePage();
lastHeader.NextPageId = newPageId;
lastHeader.WriteTo(buffer);
WritePageImmediate(lastPageId, buffer);
src/CBDD.Core/Storage/StorageEngine.Transactions.cs
+139 -168
View File
@@ -4,127 +4,14 @@ namespace ZB.MOM.WW.CBDD.Core.Storage;
public sealed partial class StorageEngine
{
#region Transaction Management
/// <summary>
/// Begins a new transaction.
/// Gets the number of active transactions (diagnostics).
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <returns>The started transaction.</returns>
public Transaction BeginTransaction(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
{
_commitLock.Wait();
try
{
var txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
public int ActiveTransactionCount => _walCache.Count;
/// <summary>
/// Begins a new transaction asynchronously.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The started transaction.</returns>
public async Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
var txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
public void CommitTransaction(Transaction transaction)
{
_commitLock.Wait();
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
// In a fuller 2PC, this would be separate. Here we do it as part of commit.
if (!PrepareTransaction(transaction.TransactionId))
throw new IOException("Failed to write transaction to WAL");
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
CommitTransactionCore(transaction.TransactionId);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction asynchronously.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
/// <param name="ct">The cancellation token.</param>
public async Task CommitTransactionAsync(Transaction transaction, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
if (!await PrepareTransactionAsync(transaction.TransactionId, ct))
throw new IOException("Failed to write transaction to WAL");
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
await CommitTransactionCoreAsync(transaction.TransactionId, ct);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to roll back.</param>
public void RollbackTransaction(Transaction transaction)
{
RollbackTransaction(transaction.TransactionId);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
// Rollback doesn't usually require async logic unless logging abort record is async,
// but for consistency we might consider it. For now, sync is fine as it's not the happy path bottleneck.
#endregion
/// <summary>
/// Prepares a transaction: writes all changes to WAL but doesn't commit yet.
/// Part of 2-Phase Commit protocol.
/// Prepares a transaction: writes all changes to WAL but doesn't commit yet.
/// Part of 2-Phase Commit protocol.
/// </summary>
/// <param name="transactionId">Transaction ID</param>
/// <param name="writeSet">All writes to record in WAL</param>
@@ -136,9 +23,7 @@ public sealed partial class StorageEngine
_wal.WriteBeginRecord(transactionId);
foreach (var walEntry in _walCache[transactionId])
{
_wal.WriteDataRecord(transactionId, walEntry.Key, walEntry.Value);
}
_wal.Flush(); // Ensure WAL is persisted
return true;
@@ -151,11 +36,11 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Prepares a transaction asynchronously by writing pending changes to the WAL.
/// Prepares a transaction asynchronously by writing pending changes to the WAL.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns><see langword="true"/> if preparation succeeds; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if preparation succeeds; otherwise, <see langword="false" />.</returns>
public async Task<bool> PrepareTransactionAsync(ulong transactionId, CancellationToken ct = default)
{
try
@@ -163,12 +48,8 @@ public sealed partial class StorageEngine
await _wal.WriteBeginRecordAsync(transactionId, ct);
if (_walCache.TryGetValue(transactionId, out var changes))
{
foreach (var walEntry in changes)
{
await _wal.WriteDataRecordAsync(transactionId, walEntry.Key, walEntry.Value, ct);
}
}
await _wal.FlushAsync(ct); // Ensure WAL is persisted
return true;
@@ -180,12 +61,12 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Commits a transaction:
/// 1. Writes all changes to WAL (for durability)
/// 2. Writes commit record
/// 3. Flushes WAL to disk
/// 4. Moves pages from cache to WAL index (for future reads)
/// 5. Clears WAL cache
/// Commits a transaction:
/// 1. Writes all changes to WAL (for durability)
/// 2. Writes commit record
/// 3. Flushes WAL to disk
/// 4. Moves pages from cache to WAL index (for future reads)
/// 5. Clears WAL cache
/// </summary>
/// <param name="transactionId">Transaction to commit</param>
/// <param name="writeSet">All writes performed in this transaction (unused, kept for compatibility)</param>
@@ -216,10 +97,7 @@ public sealed partial class StorageEngine
// 1. Write all changes to WAL (from cache, not writeSet!)
_wal.WriteBeginRecord(transactionId);
foreach (var (pageId, data) in pages)
{
_wal.WriteDataRecord(transactionId, pageId, data);
}
foreach ((uint pageId, byte[] data) in pages) _wal.WriteDataRecord(transactionId, pageId, data);
// 2. Write commit record and flush
_wal.WriteCommitRecord(transactionId);
@@ -227,20 +105,14 @@ public sealed partial class StorageEngine
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
foreach (var kvp in pages) _walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
CheckpointInternal();
}
if (_wal.GetCurrentSize() > MaxWalSize) CheckpointInternal();
}
/// <summary>
/// Commits a prepared transaction asynchronously by identifier.
/// Commits a prepared transaction asynchronously by identifier.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
@@ -271,10 +143,7 @@ public sealed partial class StorageEngine
// 1. Write all changes to WAL (from cache, not writeSet!)
await _wal.WriteBeginRecordAsync(transactionId, ct);
foreach (var (pageId, data) in pages)
{
await _wal.WriteDataRecordAsync(transactionId, pageId, data, ct);
}
foreach ((uint pageId, byte[] data) in pages) await _wal.WriteDataRecordAsync(transactionId, pageId, data, ct);
// 2. Write commit record and flush
await _wal.WriteCommitRecordAsync(transactionId, ct);
@@ -282,23 +151,18 @@ public sealed partial class StorageEngine
// 3. Move pages from cache to WAL index (for reads)
_walCache.TryRemove(transactionId, out _);
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
foreach (var kvp in pages) _walIndex[kvp.Key] = kvp.Value;
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
// Checkpoint might be sync or async. For now sync inside the lock is "safe" but blocking.
// Ideally this should be async too.
CheckpointInternal();
}
}
/// <summary>
/// Marks a transaction as committed after WAL writes.
/// Used for 2PC: after Prepare() writes to WAL, this finalizes the commit.
/// Marks a transaction as committed after WAL writes.
/// Used for 2PC: after Prepare() writes to WAL, this finalizes the commit.
/// </summary>
/// <param name="transactionId">Transaction to mark committed</param>
public void MarkTransactionCommitted(ulong transactionId)
@@ -311,18 +175,11 @@ public sealed partial class StorageEngine
// Move from cache to WAL index
if (_walCache.TryRemove(transactionId, out var pages))
{
foreach (var kvp in pages)
{
_walIndex[kvp.Key] = kvp.Value;
}
}
// Auto-checkpoint if WAL grows too large
if (_wal.GetCurrentSize() > MaxWalSize)
{
CheckpointInternal();
}
if (_wal.GetCurrentSize() > MaxWalSize) CheckpointInternal();
}
finally
{
@@ -331,7 +188,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Rolls back a transaction: discards all uncommitted changes.
/// Rolls back a transaction: discards all uncommitted changes.
/// </summary>
/// <param name="transactionId">Transaction to rollback</param>
public void RollbackTransaction(ulong transactionId)
@@ -341,7 +198,7 @@ public sealed partial class StorageEngine
}
/// <summary>
/// Writes an abort record for the specified transaction.
/// Writes an abort record for the specified transaction.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
internal void WriteAbortRecord(ulong transactionId)
@@ -349,8 +206,122 @@ public sealed partial class StorageEngine
_wal.WriteAbortRecord(transactionId);
}
#region Transaction Management
/// <summary>
/// Gets the number of active transactions (diagnostics).
/// Begins a new transaction.
/// </summary>
public int ActiveTransactionCount => _walCache.Count;
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <returns>The started transaction.</returns>
public Transaction BeginTransaction(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
{
_commitLock.Wait();
try
{
ulong txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Begins a new transaction asynchronously.
/// </summary>
/// <param name="isolationLevel">The transaction isolation level.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The started transaction.</returns>
public async Task<Transaction> BeginTransactionAsync(IsolationLevel isolationLevel = IsolationLevel.ReadCommitted,
CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
ulong txnId = _nextTransactionId++;
var transaction = new Transaction(txnId, this, isolationLevel);
_activeTransactions[txnId] = transaction;
return transaction;
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
public void CommitTransaction(Transaction transaction)
{
_commitLock.Wait();
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
// In a fuller 2PC, this would be separate. Here we do it as part of commit.
if (!PrepareTransaction(transaction.TransactionId))
throw new IOException("Failed to write transaction to WAL");
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
CommitTransactionCore(transaction.TransactionId);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Commits the specified transaction asynchronously.
/// </summary>
/// <param name="transaction">The transaction to commit.</param>
/// <param name="ct">The cancellation token.</param>
public async Task CommitTransactionAsync(Transaction transaction, CancellationToken ct = default)
{
await _commitLock.WaitAsync(ct);
try
{
if (!_activeTransactions.ContainsKey(transaction.TransactionId))
throw new InvalidOperationException($"Transaction {transaction.TransactionId} is not active.");
// 1. Prepare (Write to WAL)
if (!await PrepareTransactionAsync(transaction.TransactionId, ct))
throw new IOException("Failed to write transaction to WAL");
// 2. Commit (Write commit record, flush, move to cache)
// Use core commit path to avoid re-entering _commitLock.
await CommitTransactionCoreAsync(transaction.TransactionId, ct);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
finally
{
_commitLock.Release();
}
}
/// <summary>
/// Rolls back the specified transaction.
/// </summary>
/// <param name="transaction">The transaction to roll back.</param>
public void RollbackTransaction(Transaction transaction)
{
RollbackTransaction(transaction.TransactionId);
_activeTransactions.TryRemove(transaction.TransactionId, out _);
}
// Rollback doesn't usually require async logic unless logging abort record is async,
// but for consistency we might consider it. For now, sync is fine as it's not the happy path bottleneck.
#endregion
}
src/CBDD.Core/Storage/StorageEngine.cs
+89 -90
View File
@@ -1,29 +1,30 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Compression;
using ZB.MOM.WW.CBDD.Core.Transactions;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Central storage engine managing page-based storage with WAL for durability.
///
/// Architecture (WAL-based like SQLite/PostgreSQL):
/// - PageFile: Committed baseline (persistent on disk)
/// - WAL Cache: Uncommitted transaction writes (in-memory)
/// - Read: PageFile + WAL cache overlay (for Read Your Own Writes)
/// - Commit: Flush to WAL, clear cache
/// - Checkpoint: Merge WAL ? PageFile periodically
/// Central storage engine managing page-based storage with WAL for durability.
/// Architecture (WAL-based like SQLite/PostgreSQL):
/// - PageFile: Committed baseline (persistent on disk)
/// - WAL Cache: Uncommitted transaction writes (in-memory)
/// - Read: PageFile + WAL cache overlay (for Read Your Own Writes)
/// - Commit: Flush to WAL, clear cache
/// - Checkpoint: Merge WAL ? PageFile periodically
/// </summary>
public sealed partial class StorageEngine : IStorageEngine, IDisposable
{
private const long MaxWalSize = 4 * 1024 * 1024; // 4MB
// Transaction Management
private readonly ConcurrentDictionary<ulong, Transaction> _activeTransactions;
// Global lock for commit/checkpoint synchronization
private readonly SemaphoreSlim _commitLock = new(1, 1);
private readonly PageFile _pageFile;
private readonly WriteAheadLog _wal;
private readonly CompressionOptions _compressionOptions;
private readonly CompressionService _compressionService;
private readonly CompressionTelemetry _compressionTelemetry;
private readonly StorageFormatMetadata _storageFormatMetadata;
private readonly MaintenanceOptions _maintenanceOptions;
private CDC.ChangeStreamDispatcher? _cdc;
// WAL cache: TransactionId → (PageId → PageData)
// Stores uncommitted writes for "Read Your Own Writes" isolation
@@ -32,18 +33,10 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
// WAL index cache: PageId → PageData (from latest committed transaction)
// Lazily populated on first read after commit
private readonly ConcurrentDictionary<uint, byte[]> _walIndex;
// Global lock for commit/checkpoint synchronization
private readonly SemaphoreSlim _commitLock = new(1, 1);
// Transaction Management
private readonly ConcurrentDictionary<ulong, Transaction> _activeTransactions;
private ulong _nextTransactionId;
private const long MaxWalSize = 4 * 1024 * 1024; // 4MB
/// <summary>
/// Initializes a new instance of the <see cref="StorageEngine"/> class.
/// Initializes a new instance of the <see cref="StorageEngine" /> class.
/// </summary>
/// <param name="databasePath">The database file path.</param>
/// <param name="config">The page file configuration.</param>
@@ -55,13 +48,13 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
CompressionOptions? compressionOptions = null,
MaintenanceOptions? maintenanceOptions = null)
{
_compressionOptions = CompressionOptions.Normalize(compressionOptions);
_compressionService = new CompressionService();
_compressionTelemetry = new CompressionTelemetry();
_maintenanceOptions = maintenanceOptions ?? new MaintenanceOptions();
CompressionOptions = CompressionOptions.Normalize(compressionOptions);
CompressionService = new CompressionService();
CompressionTelemetry = new CompressionTelemetry();
MaintenanceOptions = maintenanceOptions ?? new MaintenanceOptions();
// Auto-derive WAL path
var walPath = Path.ChangeExtension(databasePath, ".wal");
string walPath = Path.ChangeExtension(databasePath, ".wal");
// Initialize storage infrastructure
_pageFile = new PageFile(databasePath, config);
@@ -72,14 +65,11 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
_walIndex = new ConcurrentDictionary<uint, byte[]>();
_activeTransactions = new ConcurrentDictionary<ulong, Transaction>();
_nextTransactionId = 1;
_storageFormatMetadata = InitializeStorageFormatMetadata();
StorageFormatMetadata = InitializeStorageFormatMetadata();
// Recover from WAL if exists (crash recovery or resume after close)
// This replays any committed transactions not yet checkpointed
if (_wal.GetCurrentSize() > 0)
{
Recover();
}
if (_wal.GetCurrentSize() > 0) Recover();
_ = ResumeCompactionIfNeeded();
@@ -92,58 +82,59 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
}
/// <summary>
/// Page size for this storage engine
/// Compression options for this engine instance.
/// </summary>
public CompressionOptions CompressionOptions { get; }
/// <summary>
/// Compression codec service for payload roundtrip operations.
/// </summary>
public CompressionService CompressionService { get; }
/// <summary>
/// Compression telemetry counters for this engine instance.
/// </summary>
public CompressionTelemetry CompressionTelemetry { get; }
/// <summary>
/// Gets storage format metadata associated with the current database.
/// </summary>
internal StorageFormatMetadata StorageFormatMetadata { get; }
/// <summary>
/// Gets the registered change stream dispatcher, if available.
/// </summary>
internal ChangeStreamDispatcher? Cdc { get; private set; }
/// <summary>
/// Page size for this storage engine
/// </summary>
public int PageSize => _pageFile.PageSize;
/// <summary>
/// Compression options for this engine instance.
/// </summary>
public CompressionOptions CompressionOptions => _compressionOptions;
/// <summary>
/// Compression codec service for payload roundtrip operations.
/// </summary>
public CompressionService CompressionService => _compressionService;
/// <summary>
/// Compression telemetry counters for this engine instance.
/// </summary>
public CompressionTelemetry CompressionTelemetry => _compressionTelemetry;
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats() => _compressionTelemetry.GetSnapshot();
/// <summary>
/// Gets storage format metadata associated with the current database.
/// </summary>
internal StorageFormatMetadata StorageFormatMetadata => _storageFormatMetadata;
/// <summary>
/// Checks if a page is currently being modified by another active transaction.
/// This is used to implement pessimistic locking for page allocation/selection.
/// Checks if a page is currently being modified by another active transaction.
/// This is used to implement pessimistic locking for page allocation/selection.
/// </summary>
/// <param name="pageId">The page identifier to check.</param>
/// <param name="excludingTxId">The transaction identifier to exclude from the check.</param>
/// <returns><see langword="true"/> if another transaction holds the page; otherwise, <see langword="false"/>.</returns>
/// <returns><see langword="true" /> if another transaction holds the page; otherwise, <see langword="false" />.</returns>
public bool IsPageLocked(uint pageId, ulong excludingTxId)
{
foreach (var kvp in _walCache)
{
var txId = kvp.Key;
ulong txId = kvp.Key;
if (txId == excludingTxId) continue;
var txnPages = kvp.Value;
if (txnPages.ContainsKey(pageId))
return true;
}
return false;
}
/// <summary>
/// Disposes the storage engine and closes WAL.
/// Disposes the storage engine and closes WAL.
/// </summary>
public void Dispose()
{
@@ -151,13 +142,15 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
if (_activeTransactions != null)
{
foreach (var txn in _activeTransactions.Values)
{
try
{
RollbackTransaction(txn.TransactionId);
}
catch { /* Ignore errors during dispose */ }
}
catch
{
/* Ignore errors during dispose */
}
_activeTransactions.Clear();
}
@@ -168,32 +161,38 @@ public sealed partial class StorageEngine : IStorageEngine, IDisposable
_commitLock?.Dispose();
}
/// <summary>
/// Registers the change stream dispatcher used for CDC notifications.
/// </summary>
/// <param name="cdc">The change stream dispatcher instance.</param>
internal void RegisterCdc(CDC.ChangeStreamDispatcher cdc)
/// <inheritdoc />
void IStorageEngine.RegisterCdc(ChangeStreamDispatcher cdc)
{
_cdc = cdc;
RegisterCdc(cdc);
}
/// <inheritdoc />
ChangeStreamDispatcher? IStorageEngine.Cdc => Cdc;
/// <inheritdoc />
CompressionOptions IStorageEngine.CompressionOptions => CompressionOptions;
/// <inheritdoc />
CompressionService IStorageEngine.CompressionService => CompressionService;
/// <inheritdoc />
CompressionTelemetry IStorageEngine.CompressionTelemetry => CompressionTelemetry;
/// <summary>
/// Returns a point-in-time snapshot of compression telemetry counters.
/// </summary>
public CompressionStats GetCompressionStats()
{
return CompressionTelemetry.GetSnapshot();
}
/// <summary>
/// Gets the registered change stream dispatcher, if available.
/// Registers the change stream dispatcher used for CDC notifications.
/// </summary>
internal CDC.ChangeStreamDispatcher? Cdc => _cdc;
/// <inheritdoc />
void IStorageEngine.RegisterCdc(CDC.ChangeStreamDispatcher cdc) => RegisterCdc(cdc);
/// <inheritdoc />
CDC.ChangeStreamDispatcher? IStorageEngine.Cdc => _cdc;
/// <inheritdoc />
CompressionOptions IStorageEngine.CompressionOptions => _compressionOptions;
/// <inheritdoc />
CompressionService IStorageEngine.CompressionService => _compressionService;
/// <inheritdoc />
CompressionTelemetry IStorageEngine.CompressionTelemetry => _compressionTelemetry;
/// <param name="cdc">The change stream dispatcher instance.</param>
internal void RegisterCdc(ChangeStreamDispatcher cdc)
{
Cdc = cdc;
}
}
src/CBDD.Core/Storage/VectorPage.cs
+39 -35
View File
@@ -1,11 +1,11 @@
using System.Buffers.Binary;
using System.Runtime.InteropServices;
using ZB.MOM.WW.CBDD.Core.Indexing;
namespace ZB.MOM.WW.CBDD.Core.Storage;
/// <summary>
/// Page for storing HNSW Vector Index nodes.
/// Each page stores a fixed number of nodes based on vector dimensions and M.
/// Page for storing HNSW Vector Index nodes.
/// Each page stores a fixed number of nodes based on vector dimensions and M.
/// </summary>
public struct VectorPage
{
@@ -24,17 +24,17 @@ public struct VectorPage
private const int DataOffset = 48;
/// <summary>
/// Increments the node count stored in the vector page header.
/// Increments the node count stored in the vector page header.
/// </summary>
/// <param name="page">The page buffer.</param>
public static void IncrementNodeCount(Span<byte> page)
{
int count = GetNodeCount(page);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), count + 1);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), count + 1);
}
/// <summary>
/// Initializes a vector page with header metadata and sizing information.
/// Initializes a vector page with header metadata and sizing information.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="pageId">The page identifier.</param>
@@ -52,45 +52,51 @@ public struct VectorPage
};
header.WriteTo(page);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(DimensionsOffset), dimensions);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(MaxMOffset), maxM);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(DimensionsOffset), dimensions);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(MaxMOffset), maxM);
// Node Size Calculation:
// Location (6) + MaxLevel (1) + Vector (dim * 4) + Links (maxM * 10 * 6) -- estimating 10 levels for simplicity
// Better: Node size is variable? No, let's keep it fixed per index configuration to avoid fragmentation.
// HNSW standard: level 0 has 2*M links, levels > 0 have M links.
// Max level is typically < 16. Let's reserve space for 16 levels.
int nodeSize = 6 + 1 + (dimensions * 4) + (maxM * (2 + 15) * 6);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeSizeOffset), nodeSize);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), 0);
int nodeSize = 6 + 1 + dimensions * 4 + maxM * (2 + 15) * 6;
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeSizeOffset), nodeSize);
BinaryPrimitives.WriteInt32LittleEndian(page.Slice(NodeCountOffset), 0);
}
/// <summary>
/// Gets the number of nodes currently stored in the page.
/// Gets the number of nodes currently stored in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The node count.</returns>
public static int GetNodeCount(ReadOnlySpan<byte> page) =>
System.Buffers.Binary.BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeCountOffset));
public static int GetNodeCount(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeCountOffset));
}
/// <summary>
/// Gets the configured node size for the page.
/// Gets the configured node size for the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The node size in bytes.</returns>
public static int GetNodeSize(ReadOnlySpan<byte> page) =>
System.Buffers.Binary.BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeSizeOffset));
public static int GetNodeSize(ReadOnlySpan<byte> page)
{
return BinaryPrimitives.ReadInt32LittleEndian(page.Slice(NodeSizeOffset));
}
/// <summary>
/// Gets the maximum number of nodes that can fit in the page.
/// Gets the maximum number of nodes that can fit in the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <returns>The maximum node count.</returns>
public static int GetMaxNodes(ReadOnlySpan<byte> page) =>
(page.Length - DataOffset) / GetNodeSize(page);
public static int GetMaxNodes(ReadOnlySpan<byte> page)
{
return (page.Length - DataOffset) / GetNodeSize(page);
}
/// <summary>
/// Writes a node to the page at the specified index.
/// Writes a node to the page at the specified index.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="nodeIndex">The zero-based node index.</param>
@@ -98,10 +104,11 @@ public struct VectorPage
/// <param name="maxLevel">The maximum graph level for the node.</param>
/// <param name="vector">The vector values to store.</param>
/// <param name="dimensions">The vector dimensionality.</param>
public static void WriteNode(Span<byte> page, int nodeIndex, DocumentLocation loc, int maxLevel, ReadOnlySpan<float> vector, int dimensions)
public static void WriteNode(Span<byte> page, int nodeIndex, DocumentLocation loc, int maxLevel,
ReadOnlySpan<float> vector, int dimensions)
{
int nodeSize = GetNodeSize(page);
int offset = DataOffset + (nodeIndex * nodeSize);
int offset = DataOffset + nodeIndex * nodeSize;
var nodeSpan = page.Slice(offset, nodeSize);
// 1. Document Location
@@ -120,17 +127,18 @@ public struct VectorPage
}
/// <summary>
/// Reads node metadata and vector data from the page.
/// Reads node metadata and vector data from the page.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="nodeIndex">The zero-based node index.</param>
/// <param name="loc">When this method returns, contains the node document location.</param>
/// <param name="maxLevel">When this method returns, contains the node max level.</param>
/// <param name="vector">The destination span for vector values.</param>
public static void ReadNodeData(ReadOnlySpan<byte> page, int nodeIndex, out DocumentLocation loc, out int maxLevel, Span<float> vector)
public static void ReadNodeData(ReadOnlySpan<byte> page, int nodeIndex, out DocumentLocation loc, out int maxLevel,
Span<float> vector)
{
int nodeSize = GetNodeSize(page);
int offset = DataOffset + (nodeIndex * nodeSize);
int offset = DataOffset + nodeIndex * nodeSize;
var nodeSpan = page.Slice(offset, nodeSize);
loc = DocumentLocation.ReadFrom(nodeSpan.Slice(0, 6));
@@ -141,7 +149,7 @@ public struct VectorPage
}
/// <summary>
/// Gets the span that stores links for a node at a specific level.
/// Gets the span that stores links for a node at a specific level.
/// </summary>
/// <param name="page">The page buffer.</param>
/// <param name="nodeIndex">The zero-based node index.</param>
@@ -152,23 +160,19 @@ public struct VectorPage
public static Span<byte> GetLinksSpan(Span<byte> page, int nodeIndex, int level, int dimensions, int maxM)
{
int nodeSize = GetNodeSize(page);
int nodeOffset = DataOffset + (nodeIndex * nodeSize);
int nodeOffset = DataOffset + nodeIndex * nodeSize;
// Link offset: Location(6) + MaxLevel(1) + Vector(dim*4)
int linkBaseOffset = nodeOffset + 7 + (dimensions * 4);
int linkBaseOffset = nodeOffset + 7 + dimensions * 4;
int levelOffset;
if (level == 0)
{
levelOffset = 0;
}
else
{
// Level 0 has 2*M links
levelOffset = (2 * maxM * 6) + ((level - 1) * maxM * 6);
}
levelOffset = 2 * maxM * 6 + (level - 1) * maxM * 6;
int count = (level == 0) ? (2 * maxM) : maxM;
int count = level == 0 ? 2 * maxM : maxM;
return page.Slice(linkBaseOffset + levelOffset, count * 6);
}
}
src/CBDD.Core/Transactions/CheckpointMode.cs
+12 -12
View File
@@ -1,39 +1,39 @@
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Defines checkpoint modes for WAL (Write-Ahead Log) checkpointing.
/// Similar to SQLite's checkpoint strategies.
/// Defines checkpoint modes for WAL (Write-Ahead Log) checkpointing.
/// Similar to SQLite's checkpoint strategies.
/// </summary>
public enum CheckpointMode
{
/// <summary>
/// Passive checkpoint: non-blocking, best-effort transfer from WAL to database.
/// If the checkpoint lock is busy, the operation is skipped.
/// WAL content is preserved and a checkpoint marker is appended when work is applied.
/// Passive checkpoint: non-blocking, best-effort transfer from WAL to database.
/// If the checkpoint lock is busy, the operation is skipped.
/// WAL content is preserved and a checkpoint marker is appended when work is applied.
/// </summary>
Passive = 0,
/// <summary>
/// Full checkpoint: waits for the checkpoint lock, transfers committed pages to
/// the page file, and preserves WAL content by appending a checkpoint marker.
/// Full checkpoint: waits for the checkpoint lock, transfers committed pages to
/// the page file, and preserves WAL content by appending a checkpoint marker.
/// </summary>
Full = 1,
/// <summary>
/// Truncate checkpoint: same as <see cref="Full"/> but truncates WAL after
/// successfully applying committed pages. Use this to reclaim disk space.
/// Truncate checkpoint: same as <see cref="Full" /> but truncates WAL after
/// successfully applying committed pages. Use this to reclaim disk space.
/// </summary>
Truncate = 2,
/// <summary>
/// Restart checkpoint: same as <see cref="Truncate"/> and then reinitializes
/// the WAL stream for a fresh writer session.
/// Restart checkpoint: same as <see cref="Truncate" /> and then reinitializes
/// the WAL stream for a fresh writer session.
/// </summary>
Restart = 3
}
/// <summary>
/// Result of a checkpoint execution.
/// Result of a checkpoint execution.
/// </summary>
/// <param name="Mode">Requested checkpoint mode.</param>
/// <param name="Executed">True when checkpoint logic ran; false when skipped (for passive mode contention).</param>
src/CBDD.Core/Transactions/ITransaction.cs
+16 -20
View File
@@ -1,66 +1,62 @@
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Public interface for database transactions.
/// Allows user-controlled transaction boundaries for batch operations.
/// Public interface for database transactions.
/// Allows user-controlled transaction boundaries for batch operations.
/// </summary>
/// <example>
/// using (var txn = collection.BeginTransaction())
/// {
/// using (var txn = collection.BeginTransaction())
/// {
/// collection.Insert(entity1, txn);
/// collection.Insert(entity2, txn);
/// txn.Commit();
/// }
/// }
/// </example>
public interface ITransaction : IDisposable
{
/// <summary>
/// Unique transaction identifier
/// Unique transaction identifier
/// </summary>
ulong TransactionId { get; }
/// <summary>
/// Current state of the transaction
/// Current state of the transaction
/// </summary>
TransactionState State { get; }
/// <summary>
/// Commits the transaction, making all changes permanent.
/// Must be called before Dispose() to persist changes.
/// Commits the transaction, making all changes permanent.
/// Must be called before Dispose() to persist changes.
/// </summary>
void Commit();
/// <summary>
/// Asynchronously commits the transaction, making all changes permanent.
/// Asynchronously commits the transaction, making all changes permanent.
/// </summary>
/// <param name="ct">The cancellation token.</param>
Task CommitAsync(CancellationToken ct = default);
/// <summary>
/// Rolls back the transaction, discarding all changes.
/// Called automatically on Dispose() if Commit() was not called.
/// Rolls back the transaction, discarding all changes.
/// Called automatically on Dispose() if Commit() was not called.
/// </summary>
void Rollback();
/// <summary>
/// Adds a write operation to the current batch or transaction.
/// Adds a write operation to the current batch or transaction.
/// </summary>
/// <param name="operation">The write operation to add. Cannot be null.</param>
void AddWrite(WriteOperation operation);
/// <summary>
/// Prepares the object for use by performing any necessary initialization or setup.
/// Prepares the object for use by performing any necessary initialization or setup.
/// </summary>
/// <returns>true if the preparation was successful; otherwise, false.</returns>
bool Prepare();
/// <summary>
/// Event triggered when the transaction acts rollback.
/// Useful for restoring in-memory state (like ID maps).
/// Event triggered when the transaction acts rollback.
/// Useful for restoring in-memory state (like ID maps).
/// </summary>
event Action? OnRollback;
}
src/CBDD.Core/Transactions/ITransactionHolder.cs
+18 -12
View File
@@ -1,26 +1,32 @@
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Defines a contract for managing and providing access to the current transaction context.
/// Defines a contract for managing and providing access to the current transaction context.
/// </summary>
/// <remarks>Implementations of this interface are responsible for tracking the current transaction and starting a
/// new one if none exists. This is typically used in scenarios where transactional consistency is required across
/// multiple operations.</remarks>
/// <remarks>
/// Implementations of this interface are responsible for tracking the current transaction and starting a
/// new one if none exists. This is typically used in scenarios where transactional consistency is required across
/// multiple operations.
/// </remarks>
public interface ITransactionHolder
{
/// <summary>
/// Gets the current transaction if one exists; otherwise, starts a new transaction.
/// Gets the current transaction if one exists; otherwise, starts a new transaction.
/// </summary>
/// <remarks>Use this method to ensure that a transaction context is available for the current operation.
/// If a transaction is already in progress, it is returned; otherwise, a new transaction is started and returned.
/// The caller is responsible for managing the transaction's lifetime as appropriate.</remarks>
/// <returns>An <see cref="ITransaction"/> representing the current transaction, or a new transaction if none is active.</returns>
/// <remarks>
/// Use this method to ensure that a transaction context is available for the current operation.
/// If a transaction is already in progress, it is returned; otherwise, a new transaction is started and returned.
/// The caller is responsible for managing the transaction's lifetime as appropriate.
/// </remarks>
/// <returns>An <see cref="ITransaction" /> representing the current transaction, or a new transaction if none is active.</returns>
ITransaction GetCurrentTransactionOrStart();
/// <summary>
/// Gets the current transaction if one exists; otherwise, starts a new transaction asynchronously.
/// Gets the current transaction if one exists; otherwise, starts a new transaction asynchronously.
/// </summary>
/// <returns>A task that represents the asynchronous operation. The task result contains an <see cref="ITransaction"/>
/// representing the current or newly started transaction.</returns>
/// <returns>
/// A task that represents the asynchronous operation. The task result contains an <see cref="ITransaction" />
/// representing the current or newly started transaction.
/// </returns>
Task<ITransaction> GetCurrentTransactionOrStartAsync();
}
src/CBDD.Core/Transactions/Transaction.cs
+81 -97
View File
@@ -1,233 +1,217 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.CDC;
using ZB.MOM.WW.CBDD.Core.Storage;
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Represents a transaction with ACID properties.
/// Uses MVCC (Multi-Version Concurrency Control) for isolation.
/// Represents a transaction with ACID properties.
/// Uses MVCC (Multi-Version Concurrency Control) for isolation.
/// </summary>
public sealed class Transaction : ITransaction
{
private readonly ulong _transactionId;
private readonly IsolationLevel _isolationLevel;
private readonly DateTime _startTime;
private readonly List<InternalChangeEvent> _pendingChanges = new();
private readonly StorageEngine _storage;
private readonly List<CDC.InternalChangeEvent> _pendingChanges = new();
private TransactionState _state;
private bool _disposed;
/// <summary>
/// Initializes a new transaction.
/// Initializes a new transaction.
/// </summary>
/// <param name="transactionId">The unique transaction identifier.</param>
/// <param name="storage">The storage engine used by this transaction.</param>
/// <param name="isolationLevel">The transaction isolation level.</param>
public Transaction(ulong transactionId,
StorageEngine storage,
IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
StorageEngine storage,
IsolationLevel isolationLevel = IsolationLevel.ReadCommitted)
{
_transactionId = transactionId;
TransactionId = transactionId;
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_isolationLevel = isolationLevel;
_startTime = DateTime.UtcNow;
_state = TransactionState.Active;
IsolationLevel = isolationLevel;
StartTime = DateTime.UtcNow;
State = TransactionState.Active;
}
/// <summary>
/// Adds a pending CDC change to be published after commit.
/// Gets the configured transaction isolation level.
/// </summary>
/// <param name="change">The change event to buffer.</param>
internal void AddChange(CDC.InternalChangeEvent change)
{
_pendingChanges.Add(change);
}
public IsolationLevel IsolationLevel { get; }
/// <summary>
/// Gets the unique transaction identifier.
/// Gets the UTC start time of the transaction.
/// </summary>
public ulong TransactionId => _transactionId;
public DateTime StartTime { get; }
/// <summary>
/// Gets the current transaction state.
/// Gets the unique transaction identifier.
/// </summary>
public TransactionState State => _state;
public ulong TransactionId { get; }
/// <summary>
/// Gets the configured transaction isolation level.
/// Gets the current transaction state.
/// </summary>
public IsolationLevel IsolationLevel => _isolationLevel;
public TransactionState State { get; private set; }
/// <summary>
/// Gets the UTC start time of the transaction.
/// </summary>
public DateTime StartTime => _startTime;
/// <summary>
/// Adds a write operation to the transaction's write set.
/// NOTE: Makes a defensive copy of the data to ensure memory safety.
/// This allocation is necessary because the caller may return the buffer to a pool.
/// Adds a write operation to the transaction's write set.
/// NOTE: Makes a defensive copy of the data to ensure memory safety.
/// This allocation is necessary because the caller may return the buffer to a pool.
/// </summary>
/// <param name="operation">The write operation to add.</param>
public void AddWrite(WriteOperation operation)
{
if (_state != TransactionState.Active)
throw new InvalidOperationException($"Cannot add writes to transaction in state {_state}");
if (State != TransactionState.Active)
throw new InvalidOperationException($"Cannot add writes to transaction in state {State}");
// Defensive copy: necessary to prevent use-after-return if caller uses pooled buffers
byte[] ownedCopy = operation.NewValue.ToArray();
// StorageEngine gestisce tutte le scritture transazionali
_storage.WritePage(operation.PageId, _transactionId, ownedCopy);
_storage.WritePage(operation.PageId, TransactionId, ownedCopy);
}
/// <summary>
/// Prepares the transaction for commit (2PC first phase)
/// Prepares the transaction for commit (2PC first phase)
/// </summary>
public bool Prepare()
{
if (_state != TransactionState.Active)
if (State != TransactionState.Active)
return false;
_state = TransactionState.Preparing;
State = TransactionState.Preparing;
// StorageEngine handles WAL writes
return _storage.PrepareTransaction(_transactionId);
return _storage.PrepareTransaction(TransactionId);
}
/// <summary>
/// Commits the transaction.
/// Writes to WAL for durability and moves data to committed buffer.
/// Pages remain in memory until CheckpointManager writes them to disk.
/// Commits the transaction.
/// Writes to WAL for durability and moves data to committed buffer.
/// Pages remain in memory until CheckpointManager writes them to disk.
/// </summary>
public void Commit()
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine handles WAL writes and buffer management
_storage.CommitTransaction(_transactionId);
_storage.CommitTransaction(TransactionId);
_state = TransactionState.Committed;
State = TransactionState.Committed;
// Publish CDC events after successful commit
if (_pendingChanges.Count > 0 && _storage.Cdc != null)
{
foreach (var change in _pendingChanges)
{
_storage.Cdc.Publish(change);
}
}
}
/// <summary>
/// Asynchronously commits the transaction.
/// Asynchronously commits the transaction.
/// </summary>
/// <param name="ct">A cancellation token.</param>
public async Task CommitAsync(CancellationToken ct = default)
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine handles WAL writes and buffer management
await _storage.CommitTransactionAsync(_transactionId, ct);
await _storage.CommitTransactionAsync(TransactionId, ct);
_state = TransactionState.Committed;
State = TransactionState.Committed;
// Publish CDC events after successful commit
if (_pendingChanges.Count > 0 && _storage.Cdc != null)
{
foreach (var change in _pendingChanges)
{
_storage.Cdc.Publish(change);
}
}
}
/// <summary>
/// Marks the transaction as committed without writing to PageFile.
/// Used by TransactionManager with lazy checkpointing.
/// </summary>
internal void MarkCommitted()
{
if (_state != TransactionState.Preparing && _state != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {_state}");
// StorageEngine marks transaction as committed and moves to committed buffer
_storage.MarkTransactionCommitted(_transactionId);
_state = TransactionState.Committed;
}
/// <summary>
/// Rolls back the transaction (discards all writes)
/// Rolls back the transaction (discards all writes)
/// </summary>
public event Action? OnRollback;
/// <summary>
/// Rolls back the transaction and discards pending writes.
/// Rolls back the transaction and discards pending writes.
/// </summary>
public void Rollback()
{
if (_state == TransactionState.Committed)
if (State == TransactionState.Committed)
throw new InvalidOperationException("Cannot rollback committed transaction");
_pendingChanges.Clear();
_storage.RollbackTransaction(_transactionId);
_state = TransactionState.Aborted;
_storage.RollbackTransaction(TransactionId);
State = TransactionState.Aborted;
OnRollback?.Invoke();
}
/// <summary>
/// Releases transaction resources and rolls back if still active.
/// Releases transaction resources and rolls back if still active.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
if (_state == TransactionState.Active || _state == TransactionState.Preparing)
{
if (State == TransactionState.Active || State == TransactionState.Preparing)
// Auto-rollback if not committed
Rollback();
}
_disposed = true;
GC.SuppressFinalize(this);
}
/// <summary>
/// Adds a pending CDC change to be published after commit.
/// </summary>
/// <param name="change">The change event to buffer.</param>
internal void AddChange(InternalChangeEvent change)
{
_pendingChanges.Add(change);
}
/// <summary>
/// Marks the transaction as committed without writing to PageFile.
/// Used by TransactionManager with lazy checkpointing.
/// </summary>
internal void MarkCommitted()
{
if (State != TransactionState.Preparing && State != TransactionState.Active)
throw new InvalidOperationException($"Cannot commit transaction in state {State}");
// StorageEngine marks transaction as committed and moves to committed buffer
_storage.MarkTransactionCommitted(TransactionId);
State = TransactionState.Committed;
}
}
/// <summary>
/// Represents a write operation in a transaction.
/// Optimized to avoid allocations by using ReadOnlyMemory instead of byte[].
/// Represents a write operation in a transaction.
/// Optimized to avoid allocations by using ReadOnlyMemory instead of byte[].
/// </summary>
public struct WriteOperation
{
/// <summary>
/// Gets or sets the identifier of the affected document.
/// Gets or sets the identifier of the affected document.
/// </summary>
public ObjectId DocumentId { get; set; }
/// <summary>
/// Gets or sets the new serialized value.
/// Gets or sets the new serialized value.
/// </summary>
public ReadOnlyMemory<byte> NewValue { get; set; }
/// <summary>
/// Gets or sets the target page identifier.
/// Gets or sets the target page identifier.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Gets or sets the operation type.
/// Gets or sets the operation type.
/// </summary>
public OperationType Type { get; set; }
/// <summary>
/// Initializes a new write operation.
/// Initializes a new write operation.
/// </summary>
/// <param name="documentId">The identifier of the affected document.</param>
/// <param name="newValue">The new serialized value.</param>
@@ -243,7 +227,7 @@ public struct WriteOperation
// Backward compatibility constructor
/// <summary>
/// Initializes a new write operation from a byte array payload.
/// Initializes a new write operation from a byte array payload.
/// </summary>
/// <param name="documentId">The identifier of the affected document.</param>
/// <param name="newValue">The new serialized value.</param>
@@ -259,7 +243,7 @@ public struct WriteOperation
}
/// <summary>
/// Type of write operation
/// Type of write operation
/// </summary>
public enum OperationType : byte
{
src/CBDD.Core/Transactions/TransactionState.cs
+2 -2
View File
@@ -1,7 +1,7 @@
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// Transaction states
/// Transaction states
/// </summary>
public enum TransactionState : byte
{
@@ -19,7 +19,7 @@ public enum TransactionState : byte
}
/// <summary>
/// Transaction isolation levels
/// Transaction isolation levels
/// </summary>
public enum IsolationLevel : byte
{
src/CBDD.Core/Transactions/WriteAheadLog.cs
+97 -107
View File
@@ -1,7 +1,9 @@
using System.Buffers;
namespace ZB.MOM.WW.CBDD.Core.Transactions;
/// <summary>
/// WAL record types
/// WAL record types
/// </summary>
public enum WalRecordType : byte
{
@@ -13,18 +15,18 @@ public enum WalRecordType : byte
}
/// <summary>
/// Write-Ahead Log (WAL) for durability and recovery.
/// All changes are logged before being applied.
/// Write-Ahead Log (WAL) for durability and recovery.
/// All changes are logged before being applied.
/// </summary>
public sealed class WriteAheadLog : IDisposable
{
private readonly string _walPath;
private FileStream? _walStream;
private readonly SemaphoreSlim _lock = new(1, 1);
private readonly string _walPath;
private bool _disposed;
private FileStream? _walStream;
/// <summary>
/// Initializes a new instance of the <see cref="WriteAheadLog"/> class.
/// Initializes a new instance of the <see cref="WriteAheadLog" /> class.
/// </summary>
/// <param name="walPath">The file path of the write-ahead log.</param>
public WriteAheadLog(string walPath)
@@ -35,14 +37,37 @@ public sealed class WriteAheadLog : IDisposable
_walPath,
FileMode.OpenOrCreate,
FileAccess.ReadWrite,
FileShare.None, // Exclusive access like PageFile
bufferSize: 64 * 1024); // 64KB buffer for better sequential write performance
FileShare.None, // Exclusive access like PageFile
64 * 1024); // 64KB buffer for better sequential write performance
// REMOVED FileOptions.WriteThrough for SQLite-style lazy checkpointing
// Durability is ensured by explicit Flush() calls
}
/// <summary>
/// Writes a begin transaction record
/// Releases resources used by the write-ahead log.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
_lock.Wait();
try
{
_walStream?.Dispose();
_disposed = true;
}
finally
{
_lock.Release();
_lock.Dispose();
}
GC.SuppressFinalize(this);
}
/// <summary>
/// Writes a begin transaction record
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
public void WriteBeginRecord(ulong transactionId)
@@ -59,7 +84,7 @@ public sealed class WriteAheadLog : IDisposable
}
/// <summary>
/// Writes a begin transaction record asynchronously.
/// Writes a begin transaction record asynchronously.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
@@ -70,7 +95,7 @@ public sealed class WriteAheadLog : IDisposable
try
{
// Use ArrayPool for async I/O compatibility (cannot use stackalloc with async)
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Begin;
@@ -81,7 +106,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -92,7 +117,7 @@ public sealed class WriteAheadLog : IDisposable
private void WriteBeginRecordInternal(ulong transactionId)
{
Span<byte> buffer = stackalloc byte[17]; // type(1) + txnId(8) + timestamp(8)
Span<byte> buffer = stackalloc byte[17]; // type(1) + txnId(8) + timestamp(8)
buffer[0] = (byte)WalRecordType.Begin;
BitConverter.TryWriteBytes(buffer[1..9], transactionId);
BitConverter.TryWriteBytes(buffer[9..17], DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());
@@ -102,10 +127,10 @@ public sealed class WriteAheadLog : IDisposable
/// <summary>
/// Writes a commit record
/// Writes a commit record
/// </summary>
/// <summary>
/// Writes a commit record
/// Writes a commit record
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
public void WriteCommitRecord(ulong transactionId)
@@ -122,7 +147,7 @@ public sealed class WriteAheadLog : IDisposable
}
/// <summary>
/// Writes a commit record asynchronously.
/// Writes a commit record asynchronously.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
@@ -132,7 +157,7 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Commit;
@@ -143,7 +168,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -154,7 +179,7 @@ public sealed class WriteAheadLog : IDisposable
private void WriteCommitRecordInternal(ulong transactionId)
{
Span<byte> buffer = stackalloc byte[17]; // type(1) + txnId(8) + timestamp(8)
Span<byte> buffer = stackalloc byte[17]; // type(1) + txnId(8) + timestamp(8)
buffer[0] = (byte)WalRecordType.Commit;
BitConverter.TryWriteBytes(buffer[1..9], transactionId);
BitConverter.TryWriteBytes(buffer[9..17], DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());
@@ -164,10 +189,10 @@ public sealed class WriteAheadLog : IDisposable
/// <summary>
/// Writes an abort record
/// Writes an abort record
/// </summary>
/// <summary>
/// Writes an abort record
/// Writes an abort record
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
public void WriteAbortRecord(ulong transactionId)
@@ -184,7 +209,7 @@ public sealed class WriteAheadLog : IDisposable
}
/// <summary>
/// Writes an abort record asynchronously.
/// Writes an abort record asynchronously.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="ct">The cancellation token.</param>
@@ -194,7 +219,7 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Abort;
@@ -205,7 +230,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -216,7 +241,7 @@ public sealed class WriteAheadLog : IDisposable
private void WriteAbortRecordInternal(ulong transactionId)
{
Span<byte> buffer = stackalloc byte[17]; // type(1) + txnId(8) + timestamp(8)
Span<byte> buffer = stackalloc byte[17]; // type(1) + txnId(8) + timestamp(8)
buffer[0] = (byte)WalRecordType.Abort;
BitConverter.TryWriteBytes(buffer[1..9], transactionId);
BitConverter.TryWriteBytes(buffer[9..17], DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());
@@ -226,7 +251,7 @@ public sealed class WriteAheadLog : IDisposable
/// <summary>
/// Writes a checkpoint marker record.
/// Writes a checkpoint marker record.
/// </summary>
public void WriteCheckpointRecord()
{
@@ -242,7 +267,7 @@ public sealed class WriteAheadLog : IDisposable
}
/// <summary>
/// Writes a checkpoint marker record asynchronously.
/// Writes a checkpoint marker record asynchronously.
/// </summary>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task that represents the asynchronous write operation.</returns>
@@ -251,7 +276,7 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(17);
byte[] buffer = ArrayPool<byte>.Shared.Rent(17);
try
{
buffer[0] = (byte)WalRecordType.Checkpoint;
@@ -261,7 +286,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -272,7 +297,7 @@ public sealed class WriteAheadLog : IDisposable
private void WriteCheckpointRecordInternal()
{
Span<byte> buffer = stackalloc byte[17]; // type(1) + reserved(8) + timestamp(8)
Span<byte> buffer = stackalloc byte[17]; // type(1) + reserved(8) + timestamp(8)
buffer[0] = (byte)WalRecordType.Checkpoint;
BitConverter.TryWriteBytes(buffer[1..9], 0UL);
BitConverter.TryWriteBytes(buffer[9..17], DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());
@@ -281,10 +306,10 @@ public sealed class WriteAheadLog : IDisposable
/// <summary>
/// Writes a data modification record
/// Writes a data modification record
/// </summary>
/// <summary>
/// Writes a data modification record
/// Writes a data modification record
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="pageId">The page identifier of the modified page.</param>
@@ -303,22 +328,23 @@ public sealed class WriteAheadLog : IDisposable
}
/// <summary>
/// Writes a data modification record asynchronously.
/// Writes a data modification record asynchronously.
/// </summary>
/// <param name="transactionId">The transaction identifier.</param>
/// <param name="pageId">The page identifier of the modified page.</param>
/// <param name="afterImage">The page contents after modification.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task that represents the asynchronous write operation.</returns>
public async ValueTask WriteDataRecordAsync(ulong transactionId, uint pageId, ReadOnlyMemory<byte> afterImage, CancellationToken ct = default)
public async ValueTask WriteDataRecordAsync(ulong transactionId, uint pageId, ReadOnlyMemory<byte> afterImage,
CancellationToken ct = default)
{
await _lock.WaitAsync(ct);
try
{
var headerSize = 17;
var totalSize = headerSize + afterImage.Length;
int totalSize = headerSize + afterImage.Length;
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(totalSize);
byte[] buffer = ArrayPool<byte>.Shared.Rent(totalSize);
try
{
buffer[0] = (byte)WalRecordType.Write;
@@ -332,7 +358,7 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
finally
@@ -345,9 +371,9 @@ public sealed class WriteAheadLog : IDisposable
{
// Header: type(1) + txnId(8) + pageId(4) + afterSize(4) = 17 bytes
var headerSize = 17;
var totalSize = headerSize + afterImage.Length;
int totalSize = headerSize + afterImage.Length;
var buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(totalSize);
byte[] buffer = ArrayPool<byte>.Shared.Rent(totalSize);
try
{
buffer[0] = (byte)WalRecordType.Write;
@@ -361,23 +387,23 @@ public sealed class WriteAheadLog : IDisposable
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
ArrayPool<byte>.Shared.Return(buffer);
}
}
/// <summary>
/// Flushes all buffered writes to disk
/// Flushes all buffered writes to disk
/// </summary>
/// <summary>
/// Flushes all buffered writes to disk
/// Flushes all buffered writes to disk
/// </summary>
public void Flush()
{
_lock.Wait();
try
{
_walStream?.Flush(flushToDisk: true);
_walStream?.Flush(true);
}
finally
{
@@ -386,7 +412,7 @@ public sealed class WriteAheadLog : IDisposable
}
/// <summary>
/// Flushes all buffered writes to disk asynchronously.
/// Flushes all buffered writes to disk asynchronously.
/// </summary>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task that represents the asynchronous flush operation.</returns>
@@ -395,18 +421,15 @@ public sealed class WriteAheadLog : IDisposable
await _lock.WaitAsync(ct);
try
{
if (_walStream != null)
{
await _walStream.FlushAsync(ct);
// FlushAsync doesn't guarantee flushToDisk on all platforms/implementations in the same way as Flush(true)
// but FileStream in .NET 6+ handles this reasonable well.
// For strict durability, we might still want to invoke a sync flush or check platform specifics,
// but typically FlushAsync(ct) is sufficient for "Async" pattern.
// However, FileStream.FlushAsync() acts like flushToDisk=false by default in older .NET?
// Actually, FileStream.Flush() has flushToDisk arg, FlushAsync does not but implementation usually does buffer flush.
// To be safe for WAL, we might care about fsync.
// For now, just FlushAsync();
}
if (_walStream != null) await _walStream.FlushAsync(ct);
// FlushAsync doesn't guarantee flushToDisk on all platforms/implementations in the same way as Flush(true)
// but FileStream in .NET 6+ handles this reasonable well.
// For strict durability, we might still want to invoke a sync flush or check platform specifics,
// but typically FlushAsync(ct) is sufficient for "Async" pattern.
// However, FileStream.FlushAsync() acts like flushToDisk=false by default in older .NET?
// Actually, FileStream.Flush() has flushToDisk arg, FlushAsync does not but implementation usually does buffer flush.
// To be safe for WAL, we might care about fsync.
// For now, just FlushAsync();
}
finally
{
@@ -416,7 +439,7 @@ public sealed class WriteAheadLog : IDisposable
/// <summary>
/// Gets the current size of the WAL file in bytes
/// Gets the current size of the WAL file in bytes
/// </summary>
public long GetCurrentSize()
{
@@ -433,8 +456,8 @@ public sealed class WriteAheadLog : IDisposable
/// <summary>
/// Truncates the WAL file (removes all content).
/// Should only be called after successful checkpoint.
/// Truncates the WAL file (removes all content).
/// Should only be called after successful checkpoint.
/// </summary>
public void Truncate()
{
@@ -445,7 +468,7 @@ public sealed class WriteAheadLog : IDisposable
{
_walStream.SetLength(0);
_walStream.Position = 0;
_walStream.Flush(flushToDisk: true);
_walStream.Flush(true);
}
}
finally
@@ -455,7 +478,7 @@ public sealed class WriteAheadLog : IDisposable
}
/// <summary>
/// Truncates and reopens the WAL stream to start a fresh writer session.
/// Truncates and reopens the WAL stream to start a fresh writer session.
/// </summary>
public void Restart()
{
@@ -468,7 +491,7 @@ public sealed class WriteAheadLog : IDisposable
FileMode.Create,
FileAccess.ReadWrite,
FileShare.None,
bufferSize: 64 * 1024);
64 * 1024);
}
finally
{
@@ -478,7 +501,7 @@ public sealed class WriteAheadLog : IDisposable
/// <summary>
/// Reads all WAL records (for recovery)
/// Reads all WAL records (for recovery)
/// </summary>
public List<WalRecord> ReadAll()
{
@@ -498,17 +521,15 @@ public sealed class WriteAheadLog : IDisposable
while (_walStream.Position < _walStream.Length)
{
var typeByte = _walStream.ReadByte();
int typeByte = _walStream.ReadByte();
if (typeByte == -1) break;
var type = (WalRecordType)typeByte;
// Check for invalid record type (file padding or corruption)
if (typeByte == 0 || !Enum.IsDefined(typeof(WalRecordType), type))
{
// Reached end of valid records (file may have padding)
break;
}
WalRecord record;
@@ -519,14 +540,12 @@ public sealed class WriteAheadLog : IDisposable
case WalRecordType.Abort:
case WalRecordType.Checkpoint:
// Read common fields (txnId + timestamp = 16 bytes)
var bytesRead = _walStream.Read(headerBuf);
int bytesRead = _walStream.Read(headerBuf);
if (bytesRead < 16)
{
// Incomplete record, stop reading
return records;
}
var txnId = BitConverter.ToUInt64(headerBuf[0..8]);
var txnId = BitConverter.ToUInt64(headerBuf[..8]);
var timestamp = BitConverter.ToInt64(headerBuf[8..16]);
record = new WalRecord
@@ -542,30 +561,24 @@ public sealed class WriteAheadLog : IDisposable
// Read txnId + pageId + afterSize = 16 bytes
bytesRead = _walStream.Read(headerBuf);
if (bytesRead < 16)
{
// Incomplete write record header, stop reading
return records;
}
txnId = BitConverter.ToUInt64(headerBuf[0..8]);
txnId = BitConverter.ToUInt64(headerBuf[..8]);
var pageId = BitConverter.ToUInt32(headerBuf[8..12]);
var afterSize = BitConverter.ToInt32(headerBuf[12..16]);
// Validate afterSize to prevent overflow or corruption
if (afterSize < 0 || afterSize > 100 * 1024 * 1024) // Max 100MB per record
{
// Corrupted size, stop reading
return records;
}
var afterImage = new byte[afterSize];
// Read afterImage
if (_walStream.Read(afterImage) < afterSize)
{
// Incomplete after image, stop reading
return records;
}
record = new WalRecord
{
@@ -592,59 +605,36 @@ public sealed class WriteAheadLog : IDisposable
_lock.Release();
}
}
/// <summary>
/// Releases resources used by the write-ahead log.
/// </summary>
public void Dispose()
{
if (_disposed)
return;
_lock.Wait();
try
{
_walStream?.Dispose();
_disposed = true;
}
finally
{
_lock.Release();
_lock.Dispose();
}
GC.SuppressFinalize(this);
}
}
/// <summary>
/// Represents a WAL record.
/// Implemented as struct for memory efficiency.
/// Represents a WAL record.
/// Implemented as struct for memory efficiency.
/// </summary>
public struct WalRecord
{
/// <summary>
/// Gets or sets the WAL record type.
/// Gets or sets the WAL record type.
/// </summary>
public WalRecordType Type { get; set; }
/// <summary>
/// Gets or sets the transaction identifier.
/// Gets or sets the transaction identifier.
/// </summary>
public ulong TransactionId { get; set; }
/// <summary>
/// Gets or sets the record timestamp in Unix milliseconds.
/// Gets or sets the record timestamp in Unix milliseconds.
/// </summary>
public long Timestamp { get; set; }
/// <summary>
/// Gets or sets the page identifier for write records.
/// Gets or sets the page identifier for write records.
/// </summary>
public uint PageId { get; set; }
/// <summary>
/// Gets or sets the after-image payload for write records.
/// Gets or sets the after-image payload for write records.
/// </summary>
public byte[]? AfterImage { get; set; }
}
src/CBDD.Core/ZB.MOM.WW.CBDD.Core.csproj
+30 -30
View File
@@ -1,38 +1,38 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Core</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Core</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.Core</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.Core</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PackageId>ZB.MOM.WW.CBDD.Core</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine Core Library for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<PackageId>ZB.MOM.WW.CBDD.Core</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine Core Library for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj"/>
</ItemGroup>
<ItemGroup>
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
<_Parameter1>ZB.MOM.WW.CBDD.Tests</_Parameter1>
</AssemblyAttribute>
</ItemGroup>
<ItemGroup>
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
<_Parameter1>ZB.MOM.WW.CBDD.Tests</_Parameter1>
</AssemblyAttribute>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>
src/CBDD.SourceGenerators/Analysis/EntityAnalyzer.cs
+226 -230
View File
@@ -1,261 +1,257 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
namespace ZB.MOM.WW.CBDD.SourceGenerators
namespace ZB.MOM.WW.CBDD.SourceGenerators;
public static class EntityAnalyzer
{
public static class EntityAnalyzer
/// <summary>
/// Analyzes an entity symbol and builds source-generation metadata.
/// </summary>
/// <param name="entityType">The entity type symbol to analyze.</param>
/// <param name="semanticModel">The semantic model for symbol and syntax analysis.</param>
/// <returns>The analyzed entity metadata.</returns>
public static EntityInfo Analyze(INamedTypeSymbol entityType, SemanticModel semanticModel)
{
/// <summary>
/// Analyzes an entity symbol and builds source-generation metadata.
/// </summary>
/// <param name="entityType">The entity type symbol to analyze.</param>
/// <param name="semanticModel">The semantic model for symbol and syntax analysis.</param>
/// <returns>The analyzed entity metadata.</returns>
public static EntityInfo Analyze(INamedTypeSymbol entityType, SemanticModel semanticModel)
var entityInfo = new EntityInfo
{
var entityInfo = new EntityInfo
{
Name = entityType.Name,
Namespace = entityType.ContainingNamespace.ToDisplayString(),
FullTypeName = SyntaxHelper.GetFullName(entityType),
CollectionName = entityType.Name.ToLowerInvariant() + "s"
};
Name = entityType.Name,
Namespace = entityType.ContainingNamespace.ToDisplayString(),
FullTypeName = SyntaxHelper.GetFullName(entityType),
CollectionName = entityType.Name.ToLowerInvariant() + "s"
};
var tableAttr = AttributeHelper.GetAttribute(entityType, "Table");
if (tableAttr != null)
{
var tableName = tableAttr.ConstructorArguments.Length > 0 ? tableAttr.ConstructorArguments[0].Value?.ToString() : null;
var schema = AttributeHelper.GetNamedArgumentValue(tableAttr, "Schema");
var tableAttr = AttributeHelper.GetAttribute(entityType, "Table");
if (tableAttr != null)
{
string? tableName = tableAttr.ConstructorArguments.Length > 0
? tableAttr.ConstructorArguments[0].Value?.ToString()
: null;
string? schema = AttributeHelper.GetNamedArgumentValue(tableAttr, "Schema");
var collectionName = !string.IsNullOrEmpty(tableName) ? tableName! : entityInfo.Name;
if (!string.IsNullOrEmpty(schema))
{
collectionName = $"{schema}.{collectionName}";
}
entityInfo.CollectionName = collectionName;
}
// Analyze properties of the root entity
AnalyzeProperties(entityType, entityInfo.Properties);
// Check if entity needs reflection-based deserialization
// Include properties with private setters or init-only setters (which can't be set outside initializers)
entityInfo.HasPrivateSetters = entityInfo.Properties.Any(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter);
// Check if entity has public parameterless constructor
var hasPublicParameterlessConstructor = entityType.Constructors
.Any(c => c.DeclaredAccessibility == Accessibility.Public && c.Parameters.Length == 0);
entityInfo.HasPrivateOrNoConstructor = !hasPublicParameterlessConstructor;
// Analyze nested types recursively
// We use a dictionary for nested types to ensure uniqueness by name
var analyzedTypes = new HashSet<string>();
AnalyzeNestedTypesRecursive(entityInfo.Properties, entityInfo.NestedTypes, semanticModel, analyzedTypes, 1, 3);
// Determine ID property
// entityInfo.IdProperty is computed from Properties.FirstOrDefault(p => p.IsKey)
if (entityInfo.IdProperty == null)
{
// Fallback to convention: property named "Id"
var idProp = entityInfo.Properties.FirstOrDefault(p => p.Name == "Id");
if (idProp != null)
{
idProp.IsKey = true;
}
}
// Check for AutoId (int/long keys)
if (entityInfo.IdProperty != null)
{
var idType = entityInfo.IdProperty.TypeName.TrimEnd('?');
if (idType == "int" || idType == "Int32" || idType == "long" || idType == "Int64")
{
entityInfo.AutoId = true;
}
}
return entityInfo;
string collectionName = !string.IsNullOrEmpty(tableName) ? tableName! : entityInfo.Name;
if (!string.IsNullOrEmpty(schema)) collectionName = $"{schema}.{collectionName}";
entityInfo.CollectionName = collectionName;
}
private static void AnalyzeProperties(INamedTypeSymbol typeSymbol, List<PropertyInfo> properties)
// Analyze properties of the root entity
AnalyzeProperties(entityType, entityInfo.Properties);
// Check if entity needs reflection-based deserialization
// Include properties with private setters or init-only setters (which can't be set outside initializers)
entityInfo.HasPrivateSetters =
entityInfo.Properties.Any(p => (!p.HasPublicSetter && p.HasAnySetter) || p.HasInitOnlySetter);
// Check if entity has public parameterless constructor
bool hasPublicParameterlessConstructor = entityType.Constructors
.Any(c => c.DeclaredAccessibility == Accessibility.Public && c.Parameters.Length == 0);
entityInfo.HasPrivateOrNoConstructor = !hasPublicParameterlessConstructor;
// Analyze nested types recursively
// We use a dictionary for nested types to ensure uniqueness by name
var analyzedTypes = new HashSet<string>();
AnalyzeNestedTypesRecursive(entityInfo.Properties, entityInfo.NestedTypes, semanticModel, analyzedTypes, 1, 3);
// Determine ID property
// entityInfo.IdProperty is computed from Properties.FirstOrDefault(p => p.IsKey)
if (entityInfo.IdProperty == null)
{
// Collect properties from the entire inheritance hierarchy
var seenProperties = new HashSet<string>();
var currentType = typeSymbol;
// Fallback to convention: property named "Id"
var idProp = entityInfo.Properties.FirstOrDefault(p => p.Name == "Id");
if (idProp != null) idProp.IsKey = true;
}
while (currentType != null && currentType.SpecialType != SpecialType.System_Object)
// Check for AutoId (int/long keys)
if (entityInfo.IdProperty != null)
{
string idType = entityInfo.IdProperty.TypeName.TrimEnd('?');
if (idType == "int" || idType == "Int32" || idType == "long" || idType == "Int64") entityInfo.AutoId = true;
}
return entityInfo;
}
private static void AnalyzeProperties(INamedTypeSymbol typeSymbol, List<PropertyInfo> properties)
{
// Collect properties from the entire inheritance hierarchy
var seenProperties = new HashSet<string>();
var currentType = typeSymbol;
while (currentType != null && currentType.SpecialType != SpecialType.System_Object)
{
var sourceProps = currentType.GetMembers()
.OfType<IPropertySymbol>()
.Where(p => p.DeclaredAccessibility == Accessibility.Public && !p.IsStatic);
foreach (var prop in sourceProps)
{
var sourceProps = currentType.GetMembers()
.OfType<IPropertySymbol>()
.Where(p => p.DeclaredAccessibility == Accessibility.Public && !p.IsStatic);
// Skip if already seen (overridden property in derived class takes precedence)
if (!seenProperties.Add(prop.Name))
continue;
foreach (var prop in sourceProps)
{
// Skip if already seen (overridden property in derived class takes precedence)
if (!seenProperties.Add(prop.Name))
continue;
if (AttributeHelper.ShouldIgnore(prop))
continue;
if (AttributeHelper.ShouldIgnore(prop))
continue;
// Skip computed getter-only properties (no setter, no backing field)
bool isReadOnlyGetter = prop.SetMethod == null && !SyntaxHelper.HasBackingField(prop);
if (isReadOnlyGetter)
continue;
// Skip computed getter-only properties (no setter, no backing field)
bool isReadOnlyGetter = prop.SetMethod == null && !SyntaxHelper.HasBackingField(prop);
if (isReadOnlyGetter)
continue;
var columnAttr = AttributeHelper.GetAttribute(prop, "Column");
var bsonFieldName = AttributeHelper.GetAttributeStringValue(prop, "BsonProperty") ??
var columnAttr = AttributeHelper.GetAttribute(prop, "Column");
string? bsonFieldName = AttributeHelper.GetAttributeStringValue(prop, "BsonProperty") ??
AttributeHelper.GetAttributeStringValue(prop, "JsonPropertyName");
if (bsonFieldName == null && columnAttr != null)
{
bsonFieldName = columnAttr.ConstructorArguments.Length > 0 ? columnAttr.ConstructorArguments[0].Value?.ToString() : null;
}
if (bsonFieldName == null && columnAttr != null)
bsonFieldName = columnAttr.ConstructorArguments.Length > 0
? columnAttr.ConstructorArguments[0].Value?.ToString()
: null;
var propInfo = new PropertyInfo
{
Name = prop.Name,
TypeName = SyntaxHelper.GetTypeName(prop.Type),
BsonFieldName = bsonFieldName ?? prop.Name.ToLowerInvariant(),
ColumnTypeName = columnAttr != null ? AttributeHelper.GetNamedArgumentValue(columnAttr, "TypeName") : null,
IsNullable = SyntaxHelper.IsNullableType(prop.Type),
IsKey = AttributeHelper.IsKey(prop),
IsRequired = AttributeHelper.HasAttribute(prop, "Required"),
HasPublicSetter = prop.SetMethod?.DeclaredAccessibility == Accessibility.Public,
HasInitOnlySetter = prop.SetMethod?.IsInitOnly == true,
HasAnySetter = prop.SetMethod != null,
IsReadOnlyGetter = isReadOnlyGetter,
BackingFieldName = (prop.SetMethod?.DeclaredAccessibility != Accessibility.Public)
? $"<{prop.Name}>k__BackingField"
: null
};
// MaxLength / MinLength
propInfo.MaxLength = AttributeHelper.GetAttributeIntValue(prop, "MaxLength");
propInfo.MinLength = AttributeHelper.GetAttributeIntValue(prop, "MinLength");
var stringLengthAttr = AttributeHelper.GetAttribute(prop, "StringLength");
if (stringLengthAttr != null)
{
if (stringLengthAttr.ConstructorArguments.Length > 0 && stringLengthAttr.ConstructorArguments[0].Value is int max)
propInfo.MaxLength = max;
var minLenStr = AttributeHelper.GetNamedArgumentValue(stringLengthAttr, "MinimumLength");
if (int.TryParse(minLenStr, out var min))
propInfo.MinLength = min;
}
// Range
var rangeAttr = AttributeHelper.GetAttribute(prop, "Range");
if (rangeAttr != null && rangeAttr.ConstructorArguments.Length >= 2)
{
if (rangeAttr.ConstructorArguments[0].Value is double dmin) propInfo.RangeMin = dmin;
else if (rangeAttr.ConstructorArguments[0].Value is int imin) propInfo.RangeMin = (double)imin;
if (rangeAttr.ConstructorArguments[1].Value is double dmax) propInfo.RangeMax = dmax;
else if (rangeAttr.ConstructorArguments[1].Value is int imax) propInfo.RangeMax = (double)imax;
}
if (SyntaxHelper.IsCollectionType(prop.Type, out var itemType))
{
propInfo.IsCollection = true;
propInfo.IsArray = prop.Type is IArrayTypeSymbol;
// Determine concrete collection type name
propInfo.CollectionConcreteTypeName = SyntaxHelper.GetTypeName(prop.Type);
if (itemType != null)
{
propInfo.CollectionItemType = SyntaxHelper.GetTypeName(itemType);
// Check if collection item is nested object
if (SyntaxHelper.IsNestedObjectType(itemType))
{
propInfo.IsCollectionItemNested = true;
propInfo.NestedTypeName = itemType.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)itemType);
}
}
}
// Check for Nested Object
else if (SyntaxHelper.IsNestedObjectType(prop.Type))
{
propInfo.IsNestedObject = true;
propInfo.NestedTypeName = prop.Type.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)prop.Type);
}
properties.Add(propInfo);
}
currentType = currentType.BaseType;
}
}
private static void AnalyzeNestedTypesRecursive(
List<PropertyInfo> properties,
Dictionary<string, NestedTypeInfo> targetNestedTypes,
SemanticModel semanticModel,
HashSet<string> analyzedTypes,
int currentDepth,
int maxDepth)
{
if (currentDepth > maxDepth) return;
// Identify properties that reference nested types (either directly or via collection)
var nestedProps = properties
.Where(p => (p.IsNestedObject || p.IsCollectionItemNested) && !string.IsNullOrEmpty(p.NestedTypeFullName))
.ToList();
foreach (var prop in nestedProps)
{
var fullTypeName = prop.NestedTypeFullName!;
var simpleName = prop.NestedTypeName!;
// Avoid cycles
if (analyzedTypes.Contains(fullTypeName)) continue;
// If already in target list, skip
if (targetNestedTypes.ContainsKey(fullTypeName)) continue;
// Try to find the symbol
INamedTypeSymbol? nestedTypeSymbol = null;
// Try by full name
nestedTypeSymbol = semanticModel.Compilation.GetTypeByMetadataName(fullTypeName);
// If not found, try to resolve via semantic model (might be in the same compilation)
if (nestedTypeSymbol == null)
var propInfo = new PropertyInfo
{
// This is more complex, but usually fullTypeName from ToDisplayString() is traceable.
// For now, let's assume GetTypeByMetadataName works for fully qualified names.
}
Name = prop.Name,
TypeName = SyntaxHelper.GetTypeName(prop.Type),
BsonFieldName = bsonFieldName ?? prop.Name.ToLowerInvariant(),
ColumnTypeName = columnAttr != null
? AttributeHelper.GetNamedArgumentValue(columnAttr, "TypeName")
: null,
IsNullable = SyntaxHelper.IsNullableType(prop.Type),
IsKey = AttributeHelper.IsKey(prop),
IsRequired = AttributeHelper.HasAttribute(prop, "Required"),
if (nestedTypeSymbol == null) continue;
analyzedTypes.Add(fullTypeName);
var nestedInfo = new NestedTypeInfo
{
Name = simpleName,
Namespace = nestedTypeSymbol.ContainingNamespace.ToDisplayString(),
FullTypeName = fullTypeName,
Depth = currentDepth
HasPublicSetter = prop.SetMethod?.DeclaredAccessibility == Accessibility.Public,
HasInitOnlySetter = prop.SetMethod?.IsInitOnly == true,
HasAnySetter = prop.SetMethod != null,
IsReadOnlyGetter = isReadOnlyGetter,
BackingFieldName = prop.SetMethod?.DeclaredAccessibility != Accessibility.Public
? $"<{prop.Name}>k__BackingField"
: null
};
// Analyze properties of this nested type
AnalyzeProperties(nestedTypeSymbol, nestedInfo.Properties);
targetNestedTypes[fullTypeName] = nestedInfo;
// MaxLength / MinLength
propInfo.MaxLength = AttributeHelper.GetAttributeIntValue(prop, "MaxLength");
propInfo.MinLength = AttributeHelper.GetAttributeIntValue(prop, "MinLength");
// Recurse
AnalyzeNestedTypesRecursive(nestedInfo.Properties, nestedInfo.NestedTypes, semanticModel, analyzedTypes, currentDepth + 1, maxDepth);
var stringLengthAttr = AttributeHelper.GetAttribute(prop, "StringLength");
if (stringLengthAttr != null)
{
if (stringLengthAttr.ConstructorArguments.Length > 0 &&
stringLengthAttr.ConstructorArguments[0].Value is int max)
propInfo.MaxLength = max;
string? minLenStr = AttributeHelper.GetNamedArgumentValue(stringLengthAttr, "MinimumLength");
if (int.TryParse(minLenStr, out int min))
propInfo.MinLength = min;
}
// Range
var rangeAttr = AttributeHelper.GetAttribute(prop, "Range");
if (rangeAttr != null && rangeAttr.ConstructorArguments.Length >= 2)
{
if (rangeAttr.ConstructorArguments[0].Value is double dmin) propInfo.RangeMin = dmin;
else if (rangeAttr.ConstructorArguments[0].Value is int imin) propInfo.RangeMin = (double)imin;
if (rangeAttr.ConstructorArguments[1].Value is double dmax) propInfo.RangeMax = dmax;
else if (rangeAttr.ConstructorArguments[1].Value is int imax) propInfo.RangeMax = (double)imax;
}
if (SyntaxHelper.IsCollectionType(prop.Type, out var itemType))
{
propInfo.IsCollection = true;
propInfo.IsArray = prop.Type is IArrayTypeSymbol;
// Determine concrete collection type name
propInfo.CollectionConcreteTypeName = SyntaxHelper.GetTypeName(prop.Type);
if (itemType != null)
{
propInfo.CollectionItemType = SyntaxHelper.GetTypeName(itemType);
// Check if collection item is nested object
if (SyntaxHelper.IsNestedObjectType(itemType))
{
propInfo.IsCollectionItemNested = true;
propInfo.NestedTypeName = itemType.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)itemType);
}
}
}
// Check for Nested Object
else if (SyntaxHelper.IsNestedObjectType(prop.Type))
{
propInfo.IsNestedObject = true;
propInfo.NestedTypeName = prop.Type.Name;
propInfo.NestedTypeFullName = SyntaxHelper.GetFullName((INamedTypeSymbol)prop.Type);
}
properties.Add(propInfo);
}
currentType = currentType.BaseType;
}
}
private static void AnalyzeNestedTypesRecursive(
List<PropertyInfo> properties,
Dictionary<string, NestedTypeInfo> targetNestedTypes,
SemanticModel semanticModel,
HashSet<string> analyzedTypes,
int currentDepth,
int maxDepth)
{
if (currentDepth > maxDepth) return;
// Identify properties that reference nested types (either directly or via collection)
var nestedProps = properties
.Where(p => (p.IsNestedObject || p.IsCollectionItemNested) && !string.IsNullOrEmpty(p.NestedTypeFullName))
.ToList();
foreach (var prop in nestedProps)
{
string fullTypeName = prop.NestedTypeFullName!;
string simpleName = prop.NestedTypeName!;
// Avoid cycles
if (analyzedTypes.Contains(fullTypeName)) continue;
// If already in target list, skip
if (targetNestedTypes.ContainsKey(fullTypeName)) continue;
// Try to find the symbol
INamedTypeSymbol? nestedTypeSymbol = null;
// Try by full name
nestedTypeSymbol = semanticModel.Compilation.GetTypeByMetadataName(fullTypeName);
// If not found, try to resolve via semantic model (might be in the same compilation)
if (nestedTypeSymbol == null)
{
// This is more complex, but usually fullTypeName from ToDisplayString() is traceable.
// For now, let's assume GetTypeByMetadataName works for fully qualified names.
}
if (nestedTypeSymbol == null) continue;
analyzedTypes.Add(fullTypeName);
var nestedInfo = new NestedTypeInfo
{
Name = simpleName,
Namespace = nestedTypeSymbol.ContainingNamespace.ToDisplayString(),
FullTypeName = fullTypeName,
Depth = currentDepth
};
// Analyze properties of this nested type
AnalyzeProperties(nestedTypeSymbol, nestedInfo.Properties);
targetNestedTypes[fullTypeName] = nestedInfo;
// Recurse
AnalyzeNestedTypesRecursive(nestedInfo.Properties, nestedInfo.NestedTypes, semanticModel, analyzedTypes,
currentDepth + 1, maxDepth);
}
}
}
src/CBDD.SourceGenerators/Generators/CodeGenerator.cs
+834 -831
View File
File diff suppressed because it is too large Load Diff
src/CBDD.SourceGenerators/Generators/MapperGenerator.cs
+376 -381
View File
@@ -1,403 +1,401 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
using ZB.MOM.WW.CBDD.SourceGenerators.Models;
namespace ZB.MOM.WW.CBDD.SourceGenerators
namespace ZB.MOM.WW.CBDD.SourceGenerators;
public class DbContextInfo
{
public class DbContextInfo
/// <summary>
/// Gets or sets the simple class name of the DbContext.
/// </summary>
public string ClassName { get; set; } = "";
/// <summary>
/// Gets the fully qualified class name of the DbContext.
/// </summary>
public string FullClassName => string.IsNullOrEmpty(Namespace) ? ClassName : $"{Namespace}.{ClassName}";
/// <summary>
/// Gets or sets the namespace that contains the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the source file path where the DbContext was found.
/// </summary>
public string FilePath { get; set; } = "";
/// <summary>
/// Gets or sets a value indicating whether the DbContext is nested.
/// </summary>
public bool IsNested { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext is partial.
/// </summary>
public bool IsPartial { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext inherits from another DbContext.
/// </summary>
public bool
HasBaseDbContext { get; set; } // True if inherits from another DbContext (not DocumentDbContext directly)
/// <summary>
/// Gets or sets the entities discovered for this DbContext.
/// </summary>
public List<EntityInfo> Entities { get; set; } = new();
/// <summary>
/// Gets or sets the collected nested types keyed by full type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; set; } = new();
}
[Generator]
public class MapperGenerator : IIncrementalGenerator
{
/// <summary>
/// Initializes the mapper source generator pipeline.
/// </summary>
/// <param name="context">The incremental generator initialization context.</param>
public void Initialize(IncrementalGeneratorInitializationContext context)
{
/// <summary>
/// Gets or sets the simple class name of the DbContext.
/// </summary>
public string ClassName { get; set; } = "";
// Find all classes that inherit from DocumentDbContext
var dbContextClasses = context.SyntaxProvider
.CreateSyntaxProvider(
static (node, _) => IsPotentialDbContext(node),
static (ctx, _) => GetDbContextInfo(ctx))
.Where(static context => context is not null)
.Collect()
.SelectMany(static (contexts, _) => contexts.GroupBy(c => c!.FullClassName).Select(g => g.First())!);
/// <summary>
/// Gets the fully qualified class name of the DbContext.
/// </summary>
public string FullClassName => string.IsNullOrEmpty(Namespace) ? ClassName : $"{Namespace}.{ClassName}";
/// <summary>
/// Gets or sets the namespace that contains the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the source file path where the DbContext was found.
/// </summary>
public string FilePath { get; set; } = "";
/// <summary>
/// Gets or sets a value indicating whether the DbContext is nested.
/// </summary>
public bool IsNested { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext is partial.
/// </summary>
public bool IsPartial { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the DbContext inherits from another DbContext.
/// </summary>
public bool HasBaseDbContext { get; set; } // True if inherits from another DbContext (not DocumentDbContext directly)
/// <summary>
/// Gets or sets the entities discovered for this DbContext.
/// </summary>
public List<EntityInfo> Entities { get; set; } = new List<EntityInfo>();
/// <summary>
/// Gets or sets the collected nested types keyed by full type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; set; } = new Dictionary<string, NestedTypeInfo>();
}
[Generator]
public class MapperGenerator : IIncrementalGenerator
{
/// <summary>
/// Initializes the mapper source generator pipeline.
/// </summary>
/// <param name="context">The incremental generator initialization context.</param>
public void Initialize(IncrementalGeneratorInitializationContext context)
// Generate code for each DbContext
context.RegisterSourceOutput(dbContextClasses, static (spc, dbContext) =>
{
// Find all classes that inherit from DocumentDbContext
var dbContextClasses = context.SyntaxProvider
.CreateSyntaxProvider(
predicate: static (node, _) => IsPotentialDbContext(node),
transform: static (ctx, _) => GetDbContextInfo(ctx))
.Where(static context => context is not null)
.Collect()
.SelectMany(static (contexts, _) => contexts.GroupBy(c => c!.FullClassName).Select(g => g.First())!);
if (dbContext == null) return;
// Generate code for each DbContext
context.RegisterSourceOutput(dbContextClasses, static (spc, dbContext) =>
var sb = new StringBuilder();
sb.AppendLine($"// Found DbContext: {dbContext.ClassName}");
sb.AppendLine(
$"// BaseType: {(dbContext.HasBaseDbContext ? "inherits from another DbContext" : "inherits from DocumentDbContext directly")}");
foreach (var entity in dbContext.Entities)
// Aggregate nested types recursively
CollectNestedTypes(entity.NestedTypes, dbContext.GlobalNestedTypes);
// Collect namespaces
var namespaces = new HashSet<string>
{
if (dbContext == null) return;
var sb = new StringBuilder();
sb.AppendLine($"// Found DbContext: {dbContext.ClassName}");
sb.AppendLine($"// BaseType: {(dbContext.HasBaseDbContext ? "inherits from another DbContext" : "inherits from DocumentDbContext directly")}");
foreach (var entity in dbContext.Entities)
{
// Aggregate nested types recursively
CollectNestedTypes(entity.NestedTypes, dbContext.GlobalNestedTypes);
}
// Collect namespaces
var namespaces = new HashSet<string>
{
"System",
"System.Collections.Generic",
"ZB.MOM.WW.CBDD.Bson",
"ZB.MOM.WW.CBDD.Core.Collections"
};
// Add Entity namespaces
foreach (var entity in dbContext.Entities)
{
if (!string.IsNullOrEmpty(entity.Namespace))
namespaces.Add(entity.Namespace);
}
foreach (var nested in dbContext.GlobalNestedTypes.Values)
{
if (!string.IsNullOrEmpty(nested.Namespace))
namespaces.Add(nested.Namespace);
}
// Sanitize file path for name uniqueness
var safeName = dbContext.ClassName;
if (!string.IsNullOrEmpty(dbContext.FilePath))
{
var fileName = System.IO.Path.GetFileNameWithoutExtension(dbContext.FilePath);
safeName += $"_{fileName}";
}
sb.AppendLine("// <auto-generated/>");
sb.AppendLine("#nullable enable");
foreach (var ns in namespaces.OrderBy(n => n))
{
sb.AppendLine($"using {ns};");
}
sb.AppendLine();
// Use safeName (Context + Filename) to avoid collisions
var mapperNamespace = $"{dbContext.Namespace}.{safeName}_Mappers";
sb.AppendLine($"namespace {mapperNamespace}");
sb.AppendLine($"{{");
var generatedMappers = new HashSet<string>();
// Generate Entity Mappers
foreach (var entity in dbContext.Entities)
{
if (generatedMappers.Add(entity.FullTypeName))
{
sb.AppendLine(CodeGenerator.GenerateMapperClass(entity, mapperNamespace));
}
}
// Generate Nested Mappers
foreach (var nested in dbContext.GlobalNestedTypes.Values)
{
if (generatedMappers.Add(nested.FullTypeName))
{
var nestedEntity = new EntityInfo
{
Name = nested.Name,
Namespace = nested.Namespace,
FullTypeName = nested.FullTypeName, // Ensure FullTypeName is copied
// Helper to copy properties
};
nestedEntity.Properties.AddRange(nested.Properties);
sb.AppendLine(CodeGenerator.GenerateMapperClass(nestedEntity, mapperNamespace));
}
}
sb.AppendLine($"}}");
sb.AppendLine();
// Partial DbContext for InitializeCollections (Only for top-level partial classes)
if (!dbContext.IsNested && dbContext.IsPartial)
{
sb.AppendLine($"namespace {dbContext.Namespace}");
sb.AppendLine($"{{");
sb.AppendLine($" public partial class {dbContext.ClassName}");
sb.AppendLine($" {{");
sb.AppendLine($" protected override void InitializeCollections()");
sb.AppendLine($" {{");
// Call base.InitializeCollections() if this context inherits from another DbContext
if (dbContext.HasBaseDbContext)
{
sb.AppendLine($" base.InitializeCollections();");
}
foreach (var entity in dbContext.Entities)
{
if (!string.IsNullOrEmpty(entity.CollectionPropertyName))
{
var mapperName = $"global::{mapperNamespace}.{CodeGenerator.GetMapperName(entity.FullTypeName)}";
sb.AppendLine($" this.{entity.CollectionPropertyName} = CreateCollection(new {mapperName}());");
}
}
sb.AppendLine($" }}");
sb.AppendLine();
// Generate Set<TId, T>() override
var collectionsWithProperties = dbContext.Entities
.Where(e => !string.IsNullOrEmpty(e.CollectionPropertyName) && !string.IsNullOrEmpty(e.CollectionIdTypeFullName))
.ToList();
if (collectionsWithProperties.Any())
{
sb.AppendLine($" public override global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T> Set<TId, T>()");
sb.AppendLine($" {{");
foreach (var entity in collectionsWithProperties)
{
var entityTypeStr = $"global::{entity.FullTypeName}";
var idTypeStr = entity.CollectionIdTypeFullName;
sb.AppendLine($" if (typeof(TId) == typeof({idTypeStr}) && typeof(T) == typeof({entityTypeStr}))");
sb.AppendLine($" return (global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T>)(object)this.{entity.CollectionPropertyName};");
}
if (dbContext.HasBaseDbContext)
{
sb.AppendLine($" return base.Set<TId, T>();");
}
else
{
sb.AppendLine($" throw new global::System.InvalidOperationException($\"No collection registered for entity type '{{typeof(T).Name}}' with key type '{{typeof(TId).Name}}'.\");");
}
sb.AppendLine($" }}");
}
sb.AppendLine($" }}");
sb.AppendLine($"}}");
}
spc.AddSource($"{dbContext.Namespace}.{safeName}.Mappers.g.cs", sb.ToString());
});
}
private static void CollectNestedTypes(Dictionary<string, NestedTypeInfo> source, Dictionary<string, NestedTypeInfo> target)
{
foreach (var kvp in source)
{
if (!target.ContainsKey(kvp.Value.FullTypeName))
{
target[kvp.Value.FullTypeName] = kvp.Value;
CollectNestedTypes(kvp.Value.NestedTypes, target);
}
}
}
private static void PrintNestedTypes(StringBuilder sb, Dictionary<string, NestedTypeInfo> nestedTypes, string indent)
{
foreach (var nt in nestedTypes.Values)
{
sb.AppendLine($"//{indent}- {nt.Name} (Depth: {nt.Depth})");
if (nt.Properties.Count > 0)
{
// Print properties for nested type to be sure
foreach (var p in nt.Properties)
{
var flags = new List<string>();
if (p.IsCollection) flags.Add($"Collection<{p.CollectionItemType}>");
if (p.IsNestedObject) flags.Add($"Nested<{p.NestedTypeName}>");
var flagStr = flags.Any() ? $" [{string.Join(", ", flags)}]" : "";
sb.AppendLine($"//{indent} - {p.Name}: {p.TypeName}{flagStr}");
}
}
if (nt.NestedTypes.Any())
{
PrintNestedTypes(sb, nt.NestedTypes, indent + " ");
}
}
}
private static bool IsPotentialDbContext(SyntaxNode node)
{
if (node.SyntaxTree.FilePath.EndsWith(".g.cs")) return false;
return node is ClassDeclarationSyntax classDecl &&
classDecl.BaseList != null &&
classDecl.Identifier.Text.EndsWith("Context");
}
private static DbContextInfo? GetDbContextInfo(GeneratorSyntaxContext context)
{
var classDecl = (ClassDeclarationSyntax)context.Node;
var semanticModel = context.SemanticModel;
var classSymbol = semanticModel.GetDeclaredSymbol(classDecl) as INamedTypeSymbol;
if (classSymbol == null) return null;
if (!SyntaxHelper.InheritsFrom(classSymbol, "DocumentDbContext"))
return null;
// Check if this context inherits from another DbContext (not DocumentDbContext directly)
var baseType = classSymbol.BaseType;
bool hasBaseDbContext = baseType != null &&
baseType.Name != "DocumentDbContext" &&
SyntaxHelper.InheritsFrom(baseType, "DocumentDbContext");
var info = new DbContextInfo
{
ClassName = classSymbol.Name,
Namespace = classSymbol.ContainingNamespace.ToDisplayString(),
FilePath = classDecl.SyntaxTree.FilePath,
IsNested = classSymbol.ContainingType != null,
IsPartial = classDecl.Modifiers.Any(m => m.IsKind(Microsoft.CodeAnalysis.CSharp.SyntaxKind.PartialKeyword)),
HasBaseDbContext = hasBaseDbContext
"System",
"System.Collections.Generic",
"ZB.MOM.WW.CBDD.Bson",
"ZB.MOM.WW.CBDD.Core.Collections"
};
// Analyze OnModelCreating to find entities
var onModelCreating = classDecl.Members
.OfType<MethodDeclarationSyntax>()
.FirstOrDefault(m => m.Identifier.Text == "OnModelCreating");
// Add Entity namespaces
foreach (var entity in dbContext.Entities)
if (!string.IsNullOrEmpty(entity.Namespace))
namespaces.Add(entity.Namespace);
foreach (var nested in dbContext.GlobalNestedTypes.Values)
if (!string.IsNullOrEmpty(nested.Namespace))
namespaces.Add(nested.Namespace);
if (onModelCreating != null)
// Sanitize file path for name uniqueness
string safeName = dbContext.ClassName;
if (!string.IsNullOrEmpty(dbContext.FilePath))
{
var entityCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "Entity");
foreach (var call in entityCalls)
string fileName = Path.GetFileNameWithoutExtension(dbContext.FilePath);
safeName += $"_{fileName}";
}
sb.AppendLine("// <auto-generated/>");
sb.AppendLine("#nullable enable");
foreach (string ns in namespaces.OrderBy(n => n)) sb.AppendLine($"using {ns};");
sb.AppendLine();
// Use safeName (Context + Filename) to avoid collisions
var mapperNamespace = $"{dbContext.Namespace}.{safeName}_Mappers";
sb.AppendLine($"namespace {mapperNamespace}");
sb.AppendLine("{");
var generatedMappers = new HashSet<string>();
// Generate Entity Mappers
foreach (var entity in dbContext.Entities)
if (generatedMappers.Add(entity.FullTypeName))
sb.AppendLine(CodeGenerator.GenerateMapperClass(entity, mapperNamespace));
// Generate Nested Mappers
foreach (var nested in dbContext.GlobalNestedTypes.Values)
if (generatedMappers.Add(nested.FullTypeName))
{
var typeName = SyntaxHelper.GetGenericTypeArgument(call);
if (typeName != null)
var nestedEntity = new EntityInfo
{
// Try to find the symbol
INamedTypeSymbol? entityType = null;
Name = nested.Name,
Namespace = nested.Namespace,
FullTypeName = nested.FullTypeName // Ensure FullTypeName is copied
// Helper to copy properties
};
nestedEntity.Properties.AddRange(nested.Properties);
// 1. Try by name in current compilation (simple name)
var symbols = semanticModel.Compilation.GetSymbolsWithName(typeName);
entityType = symbols.OfType<INamedTypeSymbol>().FirstOrDefault();
sb.AppendLine(CodeGenerator.GenerateMapperClass(nestedEntity, mapperNamespace));
}
// 2. Try by metadata name (if fully qualified)
if (entityType == null)
sb.AppendLine("}");
sb.AppendLine();
// Partial DbContext for InitializeCollections (Only for top-level partial classes)
if (!dbContext.IsNested && dbContext.IsPartial)
{
sb.AppendLine($"namespace {dbContext.Namespace}");
sb.AppendLine("{");
sb.AppendLine($" public partial class {dbContext.ClassName}");
sb.AppendLine(" {");
sb.AppendLine(" protected override void InitializeCollections()");
sb.AppendLine(" {");
// Call base.InitializeCollections() if this context inherits from another DbContext
if (dbContext.HasBaseDbContext) sb.AppendLine(" base.InitializeCollections();");
foreach (var entity in dbContext.Entities)
if (!string.IsNullOrEmpty(entity.CollectionPropertyName))
{
var mapperName =
$"global::{mapperNamespace}.{CodeGenerator.GetMapperName(entity.FullTypeName)}";
sb.AppendLine(
$" this.{entity.CollectionPropertyName} = CreateCollection(new {mapperName}());");
}
sb.AppendLine(" }");
sb.AppendLine();
// Generate Set<TId, T>() override
var collectionsWithProperties = dbContext.Entities
.Where(e => !string.IsNullOrEmpty(e.CollectionPropertyName) &&
!string.IsNullOrEmpty(e.CollectionIdTypeFullName))
.ToList();
if (collectionsWithProperties.Any())
{
sb.AppendLine(
" public override global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T> Set<TId, T>()");
sb.AppendLine(" {");
foreach (var entity in collectionsWithProperties)
{
var entityTypeStr = $"global::{entity.FullTypeName}";
string? idTypeStr = entity.CollectionIdTypeFullName;
sb.AppendLine(
$" if (typeof(TId) == typeof({idTypeStr}) && typeof(T) == typeof({entityTypeStr}))");
sb.AppendLine(
$" return (global::ZB.MOM.WW.CBDD.Core.Collections.DocumentCollection<TId, T>)(object)this.{entity.CollectionPropertyName};");
}
if (dbContext.HasBaseDbContext)
sb.AppendLine(" return base.Set<TId, T>();");
else
sb.AppendLine(
" throw new global::System.InvalidOperationException($\"No collection registered for entity type '{typeof(T).Name}' with key type '{typeof(TId).Name}'.\");");
sb.AppendLine(" }");
}
sb.AppendLine(" }");
sb.AppendLine("}");
}
spc.AddSource($"{dbContext.Namespace}.{safeName}.Mappers.g.cs", sb.ToString());
});
}
private static void CollectNestedTypes(Dictionary<string, NestedTypeInfo> source,
Dictionary<string, NestedTypeInfo> target)
{
foreach (var kvp in source)
if (!target.ContainsKey(kvp.Value.FullTypeName))
{
target[kvp.Value.FullTypeName] = kvp.Value;
CollectNestedTypes(kvp.Value.NestedTypes, target);
}
}
private static void PrintNestedTypes(StringBuilder sb, Dictionary<string, NestedTypeInfo> nestedTypes,
string indent)
{
foreach (var nt in nestedTypes.Values)
{
sb.AppendLine($"//{indent}- {nt.Name} (Depth: {nt.Depth})");
if (nt.Properties.Count > 0)
// Print properties for nested type to be sure
foreach (var p in nt.Properties)
{
var flags = new List<string>();
if (p.IsCollection) flags.Add($"Collection<{p.CollectionItemType}>");
if (p.IsNestedObject) flags.Add($"Nested<{p.NestedTypeName}>");
string flagStr = flags.Any() ? $" [{string.Join(", ", flags)}]" : "";
sb.AppendLine($"//{indent} - {p.Name}: {p.TypeName}{flagStr}");
}
if (nt.NestedTypes.Any()) PrintNestedTypes(sb, nt.NestedTypes, indent + " ");
}
}
private static bool IsPotentialDbContext(SyntaxNode node)
{
if (node.SyntaxTree.FilePath.EndsWith(".g.cs")) return false;
return node is ClassDeclarationSyntax classDecl &&
classDecl.BaseList != null &&
classDecl.Identifier.Text.EndsWith("Context");
}
private static DbContextInfo? GetDbContextInfo(GeneratorSyntaxContext context)
{
var classDecl = (ClassDeclarationSyntax)context.Node;
var semanticModel = context.SemanticModel;
var classSymbol = ModelExtensions.GetDeclaredSymbol(semanticModel, classDecl) as INamedTypeSymbol;
if (classSymbol == null) return null;
if (!SyntaxHelper.InheritsFrom(classSymbol, "DocumentDbContext"))
return null;
// Check if this context inherits from another DbContext (not DocumentDbContext directly)
var baseType = classSymbol.BaseType;
bool hasBaseDbContext = baseType != null &&
baseType.Name != "DocumentDbContext" &&
SyntaxHelper.InheritsFrom(baseType, "DocumentDbContext");
var info = new DbContextInfo
{
ClassName = classSymbol.Name,
Namespace = classSymbol.ContainingNamespace.ToDisplayString(),
FilePath = classDecl.SyntaxTree.FilePath,
IsNested = classSymbol.ContainingType != null,
IsPartial = classDecl.Modifiers.Any(m => m.IsKind(SyntaxKind.PartialKeyword)),
HasBaseDbContext = hasBaseDbContext
};
// Analyze OnModelCreating to find entities
var onModelCreating = classDecl.Members
.OfType<MethodDeclarationSyntax>()
.FirstOrDefault(m => m.Identifier.Text == "OnModelCreating");
if (onModelCreating != null)
{
var entityCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "Entity");
foreach (var call in entityCalls)
{
string? typeName = SyntaxHelper.GetGenericTypeArgument(call);
if (typeName != null)
{
// Try to find the symbol
INamedTypeSymbol? entityType = null;
// 1. Try by name in current compilation (simple name)
var symbols = semanticModel.Compilation.GetSymbolsWithName(typeName);
entityType = symbols.OfType<INamedTypeSymbol>().FirstOrDefault();
// 2. Try by metadata name (if fully qualified)
if (entityType == null) entityType = semanticModel.Compilation.GetTypeByMetadataName(typeName);
if (entityType != null)
{
// Check for duplicates
string fullTypeName = SyntaxHelper.GetFullName(entityType);
if (!info.Entities.Any(e => e.FullTypeName == fullTypeName))
{
entityType = semanticModel.Compilation.GetTypeByMetadataName(typeName);
}
if (entityType != null)
{
// Check for duplicates
var fullTypeName = SyntaxHelper.GetFullName(entityType);
if (!info.Entities.Any(e => e.FullTypeName == fullTypeName))
{
var entityInfo = EntityAnalyzer.Analyze(entityType, semanticModel);
info.Entities.Add(entityInfo);
}
var entityInfo = EntityAnalyzer.Analyze(entityType, semanticModel);
info.Entities.Add(entityInfo);
}
}
}
}
}
// Analyze OnModelCreating for HasConversion
if (onModelCreating != null)
// Analyze OnModelCreating for HasConversion
if (onModelCreating != null)
{
var conversionCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "HasConversion");
foreach (var call in conversionCalls)
{
var conversionCalls = SyntaxHelper.FindMethodInvocations(onModelCreating, "HasConversion");
foreach (var call in conversionCalls)
{
var converterName = SyntaxHelper.GetGenericTypeArgument(call);
if (converterName == null) continue;
string? converterName = SyntaxHelper.GetGenericTypeArgument(call);
if (converterName == null) continue;
// Trace back: .Property(x => x.Id).HasConversion<T>() or .HasKey(x => x.Id).HasConversion<T>()
if (call.Expression is MemberAccessExpressionSyntax { Expression: InvocationExpressionSyntax propertyCall } &&
propertyCall.Expression is MemberAccessExpressionSyntax { Name: IdentifierNameSyntax { Identifier: { Text: var propertyMethod } } } &&
(propertyMethod == "Property" || propertyMethod == "HasKey"))
// Trace back: .Property(x => x.Id).HasConversion<T>() or .HasKey(x => x.Id).HasConversion<T>()
if (call.Expression is MemberAccessExpressionSyntax
{
var propertyName = SyntaxHelper.GetPropertyName(propertyCall.ArgumentList.Arguments.FirstOrDefault()?.Expression);
if (propertyName == null) continue;
Expression: InvocationExpressionSyntax propertyCall
} &&
propertyCall.Expression is MemberAccessExpressionSyntax
{
Name: IdentifierNameSyntax { Identifier: { Text: var propertyMethod } }
} &&
(propertyMethod == "Property" || propertyMethod == "HasKey"))
{
string? propertyName =
SyntaxHelper.GetPropertyName(propertyCall.ArgumentList.Arguments.FirstOrDefault()?.Expression);
if (propertyName == null) continue;
// Trace further back: Entity<T>().Property(...)
if (propertyCall.Expression is MemberAccessExpressionSyntax { Expression: InvocationExpressionSyntax entityCall } &&
entityCall.Expression is MemberAccessExpressionSyntax { Name: GenericNameSyntax { Identifier: { Text: "Entity" } } })
// Trace further back: Entity<T>().Property(...)
if (propertyCall.Expression is MemberAccessExpressionSyntax
{
var entityTypeName = SyntaxHelper.GetGenericTypeArgument(entityCall);
if (entityTypeName != null)
Expression: InvocationExpressionSyntax entityCall
} &&
entityCall.Expression is MemberAccessExpressionSyntax
{
Name: GenericNameSyntax { Identifier: { Text: "Entity" } }
})
{
string? entityTypeName = SyntaxHelper.GetGenericTypeArgument(entityCall);
if (entityTypeName != null)
{
var entity = info.Entities.FirstOrDefault(e =>
e.Name == entityTypeName || e.FullTypeName.EndsWith("." + entityTypeName));
if (entity != null)
{
var entity = info.Entities.FirstOrDefault(e => e.Name == entityTypeName || e.FullTypeName.EndsWith("." + entityTypeName));
if (entity != null)
var prop = entity.Properties.FirstOrDefault(p => p.Name == propertyName);
if (prop != null)
{
var prop = entity.Properties.FirstOrDefault(p => p.Name == propertyName);
if (prop != null)
// Resolve TProvider from ValueConverter<TModel, TProvider>
var converterType =
semanticModel.Compilation.GetTypeByMetadataName(converterName) ??
semanticModel.Compilation.GetSymbolsWithName(converterName)
.OfType<INamedTypeSymbol>().FirstOrDefault();
prop.ConverterTypeName = converterType != null
? SyntaxHelper.GetFullName(converterType)
: converterName;
if (converterType != null && converterType.BaseType != null &&
converterType.BaseType.Name == "ValueConverter" &&
converterType.BaseType.TypeArguments.Length == 2)
{
// Resolve TProvider from ValueConverter<TModel, TProvider>
var converterType = semanticModel.Compilation.GetTypeByMetadataName(converterName) ??
semanticModel.Compilation.GetSymbolsWithName(converterName).OfType<INamedTypeSymbol>().FirstOrDefault();
prop.ConverterTypeName = converterType != null ? SyntaxHelper.GetFullName(converterType) : converterName;
if (converterType != null && converterType.BaseType != null &&
converterType.BaseType.Name == "ValueConverter" &&
converterType.BaseType.TypeArguments.Length == 2)
prop.ProviderTypeName = converterType.BaseType.TypeArguments[1].Name;
}
else if (converterType != null)
{
// Fallback: search deeper in base types
var converterBaseType = converterType.BaseType;
while (converterBaseType != null)
{
prop.ProviderTypeName = converterType.BaseType.TypeArguments[1].Name;
}
else if (converterType != null)
{
// Fallback: search deeper in base types
var converterBaseType = converterType.BaseType;
while (converterBaseType != null)
if (converterBaseType.Name == "ValueConverter" &&
converterBaseType.TypeArguments.Length == 2)
{
if (converterBaseType.Name == "ValueConverter" && converterBaseType.TypeArguments.Length == 2)
{
prop.ProviderTypeName = converterBaseType.TypeArguments[1].Name;
break;
}
converterBaseType = converterBaseType.BaseType;
prop.ProviderTypeName = converterBaseType.TypeArguments[1].Name;
break;
}
converterBaseType = converterBaseType.BaseType;
}
}
}
@@ -406,31 +404,28 @@ namespace ZB.MOM.WW.CBDD.SourceGenerators
}
}
}
}
// Analyze properties to find DocumentCollection<TId, TEntity>
var properties = classSymbol.GetMembers().OfType<IPropertySymbol>();
foreach (var prop in properties)
{
if (prop.Type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.Name == "DocumentCollection")
// Analyze properties to find DocumentCollection<TId, TEntity>
var properties = classSymbol.GetMembers().OfType<IPropertySymbol>();
foreach (var prop in properties)
if (prop.Type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.Name == "DocumentCollection")
// Expecting 2 type arguments: TId, TEntity
if (namedType.TypeArguments.Length == 2)
{
// Expecting 2 type arguments: TId, TEntity
if (namedType.TypeArguments.Length == 2)
{
var entityType = namedType.TypeArguments[1];
var entityInfo = info.Entities.FirstOrDefault(e => e.FullTypeName == entityType.ToDisplayString());
var entityType = namedType.TypeArguments[1];
var entityInfo = info.Entities.FirstOrDefault(e => e.FullTypeName == entityType.ToDisplayString());
// If found, update
if (entityInfo != null)
{
entityInfo.CollectionPropertyName = prop.Name;
entityInfo.CollectionIdTypeFullName = namedType.TypeArguments[0].ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
}
// If found, update
if (entityInfo != null)
{
entityInfo.CollectionPropertyName = prop.Name;
entityInfo.CollectionIdTypeFullName = namedType.TypeArguments[0]
.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
}
}
}
return info;
}
return info;
}
}
src/CBDD.SourceGenerators/Helpers/AttributeHelper.cs
+102 -108
View File
@@ -1,121 +1,115 @@
using System;
using System.Linq;
using Microsoft.CodeAnalysis;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
public static class AttributeHelper
{
public static class AttributeHelper
/// <summary>
/// Determines whether a property should be ignored during mapping.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true" /> when the property has an ignore attribute; otherwise, <see langword="false" />.</returns>
public static bool ShouldIgnore(IPropertySymbol property)
{
/// <summary>
/// Determines whether a property should be ignored during mapping.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true"/> when the property has an ignore attribute; otherwise, <see langword="false"/>.</returns>
public static bool ShouldIgnore(IPropertySymbol property)
return HasAttribute(property, "BsonIgnore") ||
HasAttribute(property, "JsonIgnore") ||
HasAttribute(property, "NotMapped");
}
/// <summary>
/// Determines whether a property is marked as a key.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true" /> when the property has a key attribute; otherwise, <see langword="false" />.</returns>
public static bool IsKey(IPropertySymbol property)
{
return HasAttribute(property, "Key") ||
HasAttribute(property, "BsonId");
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a string.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static string? GetAttributeStringValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0) return attr.ConstructorArguments[0].Value?.ToString();
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as an integer.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static int? GetAttributeIntValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
if (attr.ConstructorArguments[0].Value is int val)
return val;
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a double.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null" />.</returns>
public static double? GetAttributeDoubleValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
return HasAttribute(property, "BsonIgnore") ||
HasAttribute(property, "JsonIgnore") ||
HasAttribute(property, "NotMapped");
if (attr.ConstructorArguments[0].Value is double val) return val;
if (attr.ConstructorArguments[0].Value is float fval) return (double)fval;
if (attr.ConstructorArguments[0].Value is int ival) return (double)ival;
}
/// <summary>
/// Determines whether a property is marked as a key.
/// </summary>
/// <param name="property">The property symbol to inspect.</param>
/// <returns><see langword="true"/> when the property has a key attribute; otherwise, <see langword="false"/>.</returns>
public static bool IsKey(IPropertySymbol property)
{
return HasAttribute(property, "Key") ||
HasAttribute(property, "BsonId");
}
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a string.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
public static string? GetAttributeStringValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
return attr.ConstructorArguments[0].Value?.ToString();
}
/// <summary>
/// Gets a named argument value from an attribute.
/// </summary>
/// <param name="attr">The attribute data.</param>
/// <param name="name">The named argument key.</param>
/// <returns>The named argument value if present; otherwise, <see langword="null" />.</returns>
public static string? GetNamedArgumentValue(AttributeData attr, string name)
{
return attr.NamedArguments.FirstOrDefault(a => a.Key == name).Value.Value?.ToString();
}
return null;
}
/// <summary>
/// Gets the first attribute that matches the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The matching attribute data if found; otherwise, <see langword="null" />.</returns>
public static AttributeData? GetAttribute(ISymbol symbol, string attributeName)
{
return symbol.GetAttributes().FirstOrDefault(a =>
a.AttributeClass != null &&
(a.AttributeClass.Name == attributeName ||
a.AttributeClass.Name == attributeName + "Attribute"));
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as an integer.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
public static int? GetAttributeIntValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
if (attr.ConstructorArguments[0].Value is int val) return val;
}
return null;
}
/// <summary>
/// Gets the first constructor argument value for the specified attribute as a double.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The attribute value if found; otherwise, <see langword="null"/>.</returns>
public static double? GetAttributeDoubleValue(ISymbol symbol, string attributeName)
{
var attr = GetAttribute(symbol, attributeName);
if (attr != null && attr.ConstructorArguments.Length > 0)
{
if (attr.ConstructorArguments[0].Value is double val) return val;
if (attr.ConstructorArguments[0].Value is float fval) return (double)fval;
if (attr.ConstructorArguments[0].Value is int ival) return (double)ival;
}
return null;
}
/// <summary>
/// Gets a named argument value from an attribute.
/// </summary>
/// <param name="attr">The attribute data.</param>
/// <param name="name">The named argument key.</param>
/// <returns>The named argument value if present; otherwise, <see langword="null"/>.</returns>
public static string? GetNamedArgumentValue(AttributeData attr, string name)
{
return attr.NamedArguments.FirstOrDefault(a => a.Key == name).Value.Value?.ToString();
}
/// <summary>
/// Gets the first attribute that matches the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns>The matching attribute data if found; otherwise, <see langword="null"/>.</returns>
public static AttributeData? GetAttribute(ISymbol symbol, string attributeName)
{
return symbol.GetAttributes().FirstOrDefault(a =>
a.AttributeClass != null &&
(a.AttributeClass.Name == attributeName ||
a.AttributeClass.Name == attributeName + "Attribute"));
}
/// <summary>
/// Determines whether a symbol has the specified attribute.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns><see langword="true"/> when a matching attribute exists; otherwise, <see langword="false"/>.</returns>
public static bool HasAttribute(ISymbol symbol, string attributeName)
{
return GetAttribute(symbol, attributeName) != null;
}
/// <summary>
/// Determines whether a symbol has the specified attribute.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="attributeName">The attribute name to match.</param>
/// <returns><see langword="true" /> when a matching attribute exists; otherwise, <see langword="false" />.</returns>
public static bool HasAttribute(ISymbol symbol, string attributeName)
{
return GetAttribute(symbol, attributeName) != null;
}
}
src/CBDD.SourceGenerators/Helpers/SyntaxHelper.cs
+198 -222
View File
@@ -1,253 +1,229 @@
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers
namespace ZB.MOM.WW.CBDD.SourceGenerators.Helpers;
public static class SyntaxHelper
{
public static class SyntaxHelper
/// <summary>
/// Determines whether a symbol inherits from a base type with the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="baseTypeName">The base type name to match.</param>
/// <returns><see langword="true" /> if the symbol inherits from the base type; otherwise, <see langword="false" />.</returns>
public static bool InheritsFrom(INamedTypeSymbol symbol, string baseTypeName)
{
/// <summary>
/// Determines whether a symbol inherits from a base type with the specified name.
/// </summary>
/// <param name="symbol">The symbol to inspect.</param>
/// <param name="baseTypeName">The base type name to match.</param>
/// <returns><see langword="true"/> if the symbol inherits from the base type; otherwise, <see langword="false"/>.</returns>
public static bool InheritsFrom(INamedTypeSymbol symbol, string baseTypeName)
var current = symbol.BaseType;
while (current != null)
{
var current = symbol.BaseType;
while (current != null)
{
if (current.Name == baseTypeName)
return true;
current = current.BaseType;
}
return false;
}
/// <summary>
/// Finds method invocations with a matching method name under the provided syntax node.
/// </summary>
/// <param name="node">The root syntax node to search.</param>
/// <param name="methodName">The method name to match.</param>
/// <returns>A list of matching invocation expressions.</returns>
public static List<InvocationExpressionSyntax> FindMethodInvocations(SyntaxNode node, string methodName)
{
return node.DescendantNodes()
.OfType<InvocationExpressionSyntax>()
.Where(invocation =>
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess)
{
return memberAccess.Name.Identifier.Text == methodName;
}
return false;
})
.ToList();
}
/// <summary>
/// Gets the first generic type argument from an invocation, if present.
/// </summary>
/// <param name="invocation">The invocation to inspect.</param>
/// <returns>The generic type argument text, or <see langword="null"/> when not available.</returns>
public static string? GetGenericTypeArgument(InvocationExpressionSyntax invocation)
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess &&
memberAccess.Name is GenericNameSyntax genericName &&
genericName.TypeArgumentList.Arguments.Count > 0)
{
return genericName.TypeArgumentList.Arguments[0].ToString();
}
return null;
}
/// <summary>
/// Extracts a property name from an expression.
/// </summary>
/// <param name="expression">The expression to analyze.</param>
/// <returns>The property name when resolved; otherwise, <see langword="null"/>.</returns>
public static string? GetPropertyName(ExpressionSyntax? expression)
{
if (expression == null) return null;
if (expression is LambdaExpressionSyntax lambda)
{
return GetPropertyName(lambda.Body as ExpressionSyntax);
}
if (expression is MemberAccessExpressionSyntax memberAccess)
{
return memberAccess.Name.Identifier.Text;
}
if (expression is PrefixUnaryExpressionSyntax prefixUnary && prefixUnary.Operand is MemberAccessExpressionSyntax prefixMember)
{
return prefixMember.Name.Identifier.Text;
}
if (expression is PostfixUnaryExpressionSyntax postfixUnary && postfixUnary.Operand is MemberAccessExpressionSyntax postfixMember)
{
return postfixMember.Name.Identifier.Text;
}
return null;
}
/// <summary>
/// Gets the fully-qualified type name without the global prefix.
/// </summary>
/// <param name="symbol">The symbol to format.</param>
/// <returns>The formatted full type name.</returns>
public static string GetFullName(INamedTypeSymbol symbol)
{
return symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat)
.Replace("global::", "");
}
/// <summary>
/// Gets a display name for a type symbol.
/// </summary>
/// <param name="type">The type symbol to format.</param>
/// <returns>The display name.</returns>
public static string GetTypeName(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
var underlyingType = namedType.TypeArguments[0];
return GetTypeName(underlyingType) + "?";
}
if (type is IArrayTypeSymbol arrayType)
{
return GetTypeName(arrayType.ElementType) + "[]";
}
if (type is INamedTypeSymbol nt && nt.IsTupleType)
{
return type.ToDisplayString();
}
return type.ToDisplayString();
}
/// <summary>
/// Determines whether a type is nullable.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is nullable; otherwise, <see langword="false"/>.</returns>
public static bool IsNullableType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
if (current.Name == baseTypeName)
return true;
}
return type.NullableAnnotation == NullableAnnotation.Annotated;
current = current.BaseType;
}
/// <summary>
/// Determines whether a type is a collection and returns its item type when available.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <param name="itemType">When this method returns, contains the collection item type if the type is a collection.</param>
/// <returns><see langword="true"/> if the type is a collection; otherwise, <see langword="false"/>.</returns>
public static bool IsCollectionType(ITypeSymbol type, out ITypeSymbol? itemType)
{
itemType = null;
return false;
}
// Exclude string (it's IEnumerable<char> but not a collection for our purposes)
if (type.SpecialType == SpecialType.System_String)
/// <summary>
/// Finds method invocations with a matching method name under the provided syntax node.
/// </summary>
/// <param name="node">The root syntax node to search.</param>
/// <param name="methodName">The method name to match.</param>
/// <returns>A list of matching invocation expressions.</returns>
public static List<InvocationExpressionSyntax> FindMethodInvocations(SyntaxNode node, string methodName)
{
return node.DescendantNodes()
.OfType<InvocationExpressionSyntax>()
.Where(invocation =>
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess)
return memberAccess.Name.Identifier.Text == methodName;
return false;
})
.ToList();
}
// Handle arrays
if (type is IArrayTypeSymbol arrayType)
{
itemType = arrayType.ElementType;
return true;
}
/// <summary>
/// Gets the first generic type argument from an invocation, if present.
/// </summary>
/// <param name="invocation">The invocation to inspect.</param>
/// <returns>The generic type argument text, or <see langword="null" /> when not available.</returns>
public static string? GetGenericTypeArgument(InvocationExpressionSyntax invocation)
{
if (invocation.Expression is MemberAccessExpressionSyntax memberAccess &&
memberAccess.Name is GenericNameSyntax genericName &&
genericName.TypeArgumentList.Arguments.Count > 0)
return genericName.TypeArgumentList.Arguments[0].ToString();
return null;
}
// Check if the type itself is IEnumerable<T>
if (type is INamedTypeSymbol namedType && namedType.IsGenericType)
{
var typeDefName = namedType.OriginalDefinition.ToDisplayString();
if (typeDefName == "System.Collections.Generic.IEnumerable<T>" && namedType.TypeArguments.Length == 1)
{
itemType = namedType.TypeArguments[0];
return true;
}
}
/// <summary>
/// Extracts a property name from an expression.
/// </summary>
/// <param name="expression">The expression to analyze.</param>
/// <returns>The property name when resolved; otherwise, <see langword="null" />.</returns>
public static string? GetPropertyName(ExpressionSyntax? expression)
{
if (expression == null) return null;
if (expression is LambdaExpressionSyntax lambda) return GetPropertyName(lambda.Body as ExpressionSyntax);
if (expression is MemberAccessExpressionSyntax memberAccess) return memberAccess.Name.Identifier.Text;
if (expression is PrefixUnaryExpressionSyntax prefixUnary &&
prefixUnary.Operand is MemberAccessExpressionSyntax prefixMember) return prefixMember.Name.Identifier.Text;
if (expression is PostfixUnaryExpressionSyntax postfixUnary &&
postfixUnary.Operand is MemberAccessExpressionSyntax postfixMember)
return postfixMember.Name.Identifier.Text;
return null;
}
// Check if the type implements IEnumerable<T> by walking all interfaces
var enumerableInterface = type.AllInterfaces
.FirstOrDefault(i => i.IsGenericType &&
i.OriginalDefinition.ToDisplayString() == "System.Collections.Generic.IEnumerable<T>");
/// <summary>
/// Gets the fully-qualified type name without the global prefix.
/// </summary>
/// <param name="symbol">The symbol to format.</param>
/// <returns>The formatted full type name.</returns>
public static string GetFullName(INamedTypeSymbol symbol)
{
return symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat)
.Replace("global::", "");
}
if (enumerableInterface != null && enumerableInterface.TypeArguments.Length == 1)
{
itemType = enumerableInterface.TypeArguments[0];
return true;
}
/// <summary>
/// Gets a display name for a type symbol.
/// </summary>
/// <param name="type">The type symbol to format.</param>
/// <returns>The display name.</returns>
public static string GetTypeName(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
{
var underlyingType = namedType.TypeArguments[0];
return GetTypeName(underlyingType) + "?";
}
if (type is IArrayTypeSymbol arrayType) return GetTypeName(arrayType.ElementType) + "[]";
if (type is INamedTypeSymbol nt && nt.IsTupleType) return type.ToDisplayString();
return type.ToDisplayString();
}
/// <summary>
/// Determines whether a type is nullable.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true" /> if the type is nullable; otherwise, <see langword="false" />.</returns>
public static bool IsNullableType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
return true;
return type.NullableAnnotation == NullableAnnotation.Annotated;
}
/// <summary>
/// Determines whether a type is a collection and returns its item type when available.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <param name="itemType">When this method returns, contains the collection item type if the type is a collection.</param>
/// <returns><see langword="true" /> if the type is a collection; otherwise, <see langword="false" />.</returns>
public static bool IsCollectionType(ITypeSymbol type, out ITypeSymbol? itemType)
{
itemType = null;
// Exclude string (it's IEnumerable<char> but not a collection for our purposes)
if (type.SpecialType == SpecialType.System_String)
return false;
// Handle arrays
if (type is IArrayTypeSymbol arrayType)
{
itemType = arrayType.ElementType;
return true;
}
/// <summary>
/// Determines whether a type should be treated as a primitive value.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is primitive-like; otherwise, <see langword="false"/>.</returns>
public static bool IsPrimitiveType(ITypeSymbol type)
// Check if the type itself is IEnumerable<T>
if (type is INamedTypeSymbol namedType && namedType.IsGenericType)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
string typeDefName = namedType.OriginalDefinition.ToDisplayString();
if (typeDefName == "System.Collections.Generic.IEnumerable<T>" && namedType.TypeArguments.Length == 1)
{
type = namedType.TypeArguments[0];
itemType = namedType.TypeArguments[0];
return true;
}
if (type.SpecialType != SpecialType.None && type.SpecialType != SpecialType.System_Object)
return true;
var typeName = type.Name;
if (typeName == "Guid" || typeName == "DateTime" || typeName == "DateTimeOffset" ||
typeName == "TimeSpan" || typeName == "DateOnly" || typeName == "TimeOnly" ||
typeName == "Decimal" || typeName == "ObjectId")
return true;
if (type.TypeKind == TypeKind.Enum)
return true;
if (type is INamedTypeSymbol nt && nt.IsTupleType)
return true;
return false;
}
/// <summary>
/// Determines whether a type should be treated as a nested object.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true"/> if the type is a nested object; otherwise, <see langword="false"/>.</returns>
public static bool IsNestedObjectType(ITypeSymbol type)
// Check if the type implements IEnumerable<T> by walking all interfaces
var enumerableInterface = type.AllInterfaces
.FirstOrDefault(i => i.IsGenericType &&
i.OriginalDefinition.ToDisplayString() == "System.Collections.Generic.IEnumerable<T>");
if (enumerableInterface != null && enumerableInterface.TypeArguments.Length == 1)
{
if (IsPrimitiveType(type)) return false;
if (type.SpecialType == SpecialType.System_String) return false;
if (IsCollectionType(type, out _)) return false;
if (type.SpecialType == SpecialType.System_Object) return false;
if (type is INamedTypeSymbol nt && nt.IsTupleType) return false;
return type.TypeKind == TypeKind.Class || type.TypeKind == TypeKind.Struct;
itemType = enumerableInterface.TypeArguments[0];
return true;
}
/// <summary>
/// Determines whether a property has an associated backing field.
/// </summary>
/// <param name="property">The property to inspect.</param>
/// <returns><see langword="true"/> if a backing field is found; otherwise, <see langword="false"/>.</returns>
public static bool HasBackingField(IPropertySymbol property)
{
// Auto-properties have compiler-generated backing fields
// Check if there's a field with the pattern <PropertyName>k__BackingField
return property.ContainingType.GetMembers()
.OfType<IFieldSymbol>()
.Any(f => f.AssociatedSymbol?.Equals(property, SymbolEqualityComparer.Default) == true);
}
return false;
}
/// <summary>
/// Determines whether a type should be treated as a primitive value.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true" /> if the type is primitive-like; otherwise, <see langword="false" />.</returns>
public static bool IsPrimitiveType(ITypeSymbol type)
{
if (type is INamedTypeSymbol namedType &&
namedType.OriginalDefinition.SpecialType == SpecialType.System_Nullable_T)
type = namedType.TypeArguments[0];
if (type.SpecialType != SpecialType.None && type.SpecialType != SpecialType.System_Object)
return true;
string typeName = type.Name;
if (typeName == "Guid" || typeName == "DateTime" || typeName == "DateTimeOffset" ||
typeName == "TimeSpan" || typeName == "DateOnly" || typeName == "TimeOnly" ||
typeName == "Decimal" || typeName == "ObjectId")
return true;
if (type.TypeKind == TypeKind.Enum)
return true;
if (type is INamedTypeSymbol nt && nt.IsTupleType)
return true;
return false;
}
/// <summary>
/// Determines whether a type should be treated as a nested object.
/// </summary>
/// <param name="type">The type to evaluate.</param>
/// <returns><see langword="true" /> if the type is a nested object; otherwise, <see langword="false" />.</returns>
public static bool IsNestedObjectType(ITypeSymbol type)
{
if (IsPrimitiveType(type)) return false;
if (type.SpecialType == SpecialType.System_String) return false;
if (IsCollectionType(type, out _)) return false;
if (type.SpecialType == SpecialType.System_Object) return false;
if (type is INamedTypeSymbol nt && nt.IsTupleType) return false;
return type.TypeKind == TypeKind.Class || type.TypeKind == TypeKind.Struct;
}
/// <summary>
/// Determines whether a property has an associated backing field.
/// </summary>
/// <param name="property">The property to inspect.</param>
/// <returns><see langword="true" /> if a backing field is found; otherwise, <see langword="false" />.</returns>
public static bool HasBackingField(IPropertySymbol property)
{
// Auto-properties have compiler-generated backing fields
// Check if there's a field with the pattern <PropertyName>k__BackingField
return property.ContainingType.GetMembers()
.OfType<IFieldSymbol>()
.Any(f => f.AssociatedSymbol?.Equals(property, SymbolEqualityComparer.Default) == true);
}
}
src/CBDD.SourceGenerators/Models/DbContextInfo.cs
+23 -24
View File
@@ -1,32 +1,31 @@
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models;
public class DbContextInfo
{
public class DbContextInfo
{
/// <summary>
/// Gets or sets the DbContext class name.
/// </summary>
public string ClassName { get; set; } = "";
/// <summary>
/// Gets or sets the DbContext class name.
/// </summary>
public string ClassName { get; set; } = "";
/// <summary>
/// Gets or sets the namespace containing the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the namespace containing the DbContext.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the source file path for the DbContext.
/// </summary>
public string FilePath { get; set; } = "";
/// <summary>
/// Gets or sets the source file path for the DbContext.
/// </summary>
public string FilePath { get; set; } = "";
/// <summary>
/// Gets the entity types discovered for the DbContext.
/// </summary>
public List<EntityInfo> Entities { get; } = new List<EntityInfo>();
/// <summary>
/// Gets the entity types discovered for the DbContext.
/// </summary>
public List<EntityInfo> Entities { get; } = new();
/// <summary>
/// Gets global nested types keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
}
/// <summary>
/// Gets global nested types keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> GlobalNestedTypes { get; } = new();
}
src/CBDD.SourceGenerators/Models/EntityInfo.cs
+233 -199
View File
@@ -1,213 +1,247 @@
using System.Collections.Generic;
using System.Linq;
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models
namespace ZB.MOM.WW.CBDD.SourceGenerators.Models;
/// <summary>
/// Contains metadata describing an entity discovered by source generation.
/// </summary>
public class EntityInfo
{
/// <summary>
/// Contains metadata describing an entity discovered by source generation.
/// Gets or sets the entity name.
/// </summary>
public class EntityInfo
{
/// <summary>
/// Gets or sets the entity name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the entity namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified entity type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the collection name for the entity.
/// </summary>
public string CollectionName { get; set; } = "";
/// <summary>
/// Gets or sets the collection property name.
/// </summary>
public string? CollectionPropertyName { get; set; }
/// <summary>
/// Gets or sets the fully qualified collection identifier type name.
/// </summary>
public string? CollectionIdTypeFullName { get; set; }
/// <summary>
/// Gets the key property for the entity if one exists.
/// </summary>
public PropertyInfo? IdProperty => Properties.FirstOrDefault(p => p.IsKey);
/// <summary>
/// Gets or sets a value indicating whether IDs are automatically generated.
/// </summary>
public bool AutoId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity uses private setters.
/// </summary>
public bool HasPrivateSetters { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity has a private or missing constructor.
/// </summary>
public bool HasPrivateOrNoConstructor { get; set; }
/// <summary>
/// Gets the entity properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new List<PropertyInfo>();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
/// <summary>
/// Gets property names that should be ignored by mapping.
/// </summary>
public HashSet<string> IgnoredProperties { get; } = new HashSet<string>();
}
public string Name { get; set; } = "";
/// <summary>
/// Contains metadata describing a mapped property.
/// Gets or sets the entity namespace.
/// </summary>
public class PropertyInfo
{
/// <summary>
/// Gets or sets the property name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the property type name.
/// </summary>
public string TypeName { get; set; } = "";
/// <summary>
/// Gets or sets the BSON field name.
/// </summary>
public string BsonFieldName { get; set; } = "";
/// <summary>
/// Gets or sets the database column type name.
/// </summary>
public string? ColumnTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is nullable.
/// </summary>
public bool IsNullable { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has a public setter.
/// </summary>
public bool HasPublicSetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property uses an init-only setter.
/// </summary>
public bool HasInitOnlySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has any setter.
/// </summary>
public bool HasAnySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the getter is read-only.
/// </summary>
public bool IsReadOnlyGetter { get; set; }
/// <summary>
/// Gets or sets the backing field name if available.
/// </summary>
public string? BackingFieldName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is the key.
/// </summary>
public bool IsKey { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is required.
/// </summary>
public bool IsRequired { get; set; }
/// <summary>
/// Gets or sets the maximum allowed length.
/// </summary>
public int? MaxLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed length.
/// </summary>
public int? MinLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed range value.
/// </summary>
public double? RangeMin { get; set; }
/// <summary>
/// Gets or sets the maximum allowed range value.
/// </summary>
public double? RangeMax { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a collection.
/// </summary>
public bool IsCollection { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is an array.
/// </summary>
public bool IsArray { get; set; }
/// <summary>
/// Gets or sets the collection item type name.
/// </summary>
public string? CollectionItemType { get; set; }
/// <summary>
/// Gets or sets the concrete collection type name.
/// </summary>
public string? CollectionConcreteTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a nested object.
/// </summary>
public bool IsNestedObject { get; set; }
/// <summary>
/// Gets or sets a value indicating whether collection items are nested objects.
/// </summary>
public bool IsCollectionItemNested { get; set; }
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string? NestedTypeName { get; set; }
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string? NestedTypeFullName { get; set; }
/// <summary>
/// Gets or sets the converter type name.
/// </summary>
public string? ConverterTypeName { get; set; }
/// <summary>
/// Gets or sets the provider type name used by the converter.
/// </summary>
public string? ProviderTypeName { get; set; }
}
public string Namespace { get; set; } = "";
/// <summary>
/// Contains metadata describing a nested type.
/// Gets or sets the fully qualified entity type name.
/// </summary>
public class NestedTypeInfo
{
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the nested type namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the depth of the nested type.
/// </summary>
public int Depth { get; set; }
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets the nested type properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new List<PropertyInfo>();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new Dictionary<string, NestedTypeInfo>();
}
/// <summary>
/// Gets or sets the collection name for the entity.
/// </summary>
public string CollectionName { get; set; } = "";
/// <summary>
/// Gets or sets the collection property name.
/// </summary>
public string? CollectionPropertyName { get; set; }
/// <summary>
/// Gets or sets the fully qualified collection identifier type name.
/// </summary>
public string? CollectionIdTypeFullName { get; set; }
/// <summary>
/// Gets the key property for the entity if one exists.
/// </summary>
public PropertyInfo? IdProperty => Properties.FirstOrDefault(p => p.IsKey);
/// <summary>
/// Gets or sets a value indicating whether IDs are automatically generated.
/// </summary>
public bool AutoId { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity uses private setters.
/// </summary>
public bool HasPrivateSetters { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the entity has a private or missing constructor.
/// </summary>
public bool HasPrivateOrNoConstructor { get; set; }
/// <summary>
/// Gets the entity properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new();
/// <summary>
/// Gets property names that should be ignored by mapping.
/// </summary>
public HashSet<string> IgnoredProperties { get; } = new();
}
/// <summary>
/// Contains metadata describing a mapped property.
/// </summary>
public class PropertyInfo
{
/// <summary>
/// Gets or sets the property name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the property type name.
/// </summary>
public string TypeName { get; set; } = "";
/// <summary>
/// Gets or sets the BSON field name.
/// </summary>
public string BsonFieldName { get; set; } = "";
/// <summary>
/// Gets or sets the database column type name.
/// </summary>
public string? ColumnTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is nullable.
/// </summary>
public bool IsNullable { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has a public setter.
/// </summary>
public bool HasPublicSetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property uses an init-only setter.
/// </summary>
public bool HasInitOnlySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property has any setter.
/// </summary>
public bool HasAnySetter { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the getter is read-only.
/// </summary>
public bool IsReadOnlyGetter { get; set; }
/// <summary>
/// Gets or sets the backing field name if available.
/// </summary>
public string? BackingFieldName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is the key.
/// </summary>
public bool IsKey { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is required.
/// </summary>
public bool IsRequired { get; set; }
/// <summary>
/// Gets or sets the maximum allowed length.
/// </summary>
public int? MaxLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed length.
/// </summary>
public int? MinLength { get; set; }
/// <summary>
/// Gets or sets the minimum allowed range value.
/// </summary>
public double? RangeMin { get; set; }
/// <summary>
/// Gets or sets the maximum allowed range value.
/// </summary>
public double? RangeMax { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a collection.
/// </summary>
public bool IsCollection { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is an array.
/// </summary>
public bool IsArray { get; set; }
/// <summary>
/// Gets or sets the collection item type name.
/// </summary>
public string? CollectionItemType { get; set; }
/// <summary>
/// Gets or sets the concrete collection type name.
/// </summary>
public string? CollectionConcreteTypeName { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the property is a nested object.
/// </summary>
public bool IsNestedObject { get; set; }
/// <summary>
/// Gets or sets a value indicating whether collection items are nested objects.
/// </summary>
public bool IsCollectionItemNested { get; set; }
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string? NestedTypeName { get; set; }
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string? NestedTypeFullName { get; set; }
/// <summary>
/// Gets or sets the converter type name.
/// </summary>
public string? ConverterTypeName { get; set; }
/// <summary>
/// Gets or sets the provider type name used by the converter.
/// </summary>
public string? ProviderTypeName { get; set; }
}
/// <summary>
/// Contains metadata describing a nested type.
/// </summary>
public class NestedTypeInfo
{
/// <summary>
/// Gets or sets the nested type name.
/// </summary>
public string Name { get; set; } = "";
/// <summary>
/// Gets or sets the nested type namespace.
/// </summary>
public string Namespace { get; set; } = "";
/// <summary>
/// Gets or sets the fully qualified nested type name.
/// </summary>
public string FullTypeName { get; set; } = "";
/// <summary>
/// Gets or sets the depth of the nested type.
/// </summary>
public int Depth { get; set; }
/// <summary>
/// Gets the nested type properties.
/// </summary>
public List<PropertyInfo> Properties { get; } = new();
/// <summary>
/// Gets nested type metadata keyed by type name.
/// </summary>
public Dictionary<string, NestedTypeInfo> NestedTypes { get; } = new();
}
src/CBDD.SourceGenerators/ZB.MOM.WW.CBDD.SourceGenerators.csproj
+31 -31
View File
@@ -1,38 +1,38 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.SourceGenerators</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.SourceGenerators</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
<IsRoslynComponent>true</IsRoslynComponent>
<PackageId>ZB.MOM.WW.CBDD.SourceGenerators</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>Source Generators for CBDD High-Performance BSON Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation;source-generator</PackageTags>
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
<IncludeBuildOutput>false</IncludeBuildOutput>
<DevelopmentDependency>true</DevelopmentDependency>
<NoPackageAnalysis>true</NoPackageAnalysis>
</PropertyGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD.SourceGenerators</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD.SourceGenerators</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
<IsRoslynComponent>true</IsRoslynComponent>
<PackageId>ZB.MOM.WW.CBDD.SourceGenerators</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>Source Generators for CBDD High-Performance BSON Database Engine</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation;source-generator</PackageTags>
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
<IncludeBuildOutput>false</IncludeBuildOutput>
<DevelopmentDependency>true</DevelopmentDependency>
<NoPackageAnalysis>true</NoPackageAnalysis>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.8.0" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" PrivateAssets="all" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.8.0" PrivateAssets="all"/>
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" PrivateAssets="all"/>
</ItemGroup>
<ItemGroup>
<None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
</ItemGroup>
<ItemGroup>
<None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false"/>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>
src/CBDD/ZB.MOM.WW.CBDD.csproj
+27 -27
View File
@@ -1,34 +1,34 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<AssemblyName>ZB.MOM.WW.CBDD</AssemblyName>
<RootNamespace>ZB.MOM.WW.CBDD</RootNamespace>
<LangVersion>latest</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<PackageId>ZB.MOM.WW.CBDD</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<PackageId>ZB.MOM.WW.CBDD</PackageId>
<Version>1.3.1</Version>
<Authors>CBDD Team</Authors>
<Description>High-Performance BSON Database Engine for .NET 10</Description>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<RepositoryUrl>https://github.com/EntglDb/CBDD</RepositoryUrl>
<PackageTags>database;embedded;bson;nosql;net10;zero-allocation</PackageTags>
<GeneratePackageOnBuild>True</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Core\ZB.MOM.WW.CBDD.Core.csproj" />
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj" />
<ProjectReference Include="..\CBDD.SourceGenerators\ZB.MOM.WW.CBDD.SourceGenerators.csproj" PrivateAssets="none" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\CBDD.Core\ZB.MOM.WW.CBDD.Core.csproj"/>
<ProjectReference Include="..\CBDD.Bson\ZB.MOM.WW.CBDD.Bson.csproj"/>
<ProjectReference Include="..\CBDD.SourceGenerators\ZB.MOM.WW.CBDD.SourceGenerators.csproj" PrivateAssets="none"/>
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup>
<None Include="..\..\README.md" Pack="true" PackagePath="\"/>
</ItemGroup>
</Project>
tests/CBDD.Tests.Benchmark/Compaction/CompactionBenchmarks.cs
+16 -18
View File
@@ -1,6 +1,6 @@
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Storage;
@@ -15,21 +15,22 @@ namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
[JsonExporterAttribute.Full]
public class CompactionBenchmarks
{
private readonly List<ObjectId> _insertedIds = [];
private DocumentCollection<Person> _collection = null!;
private string _dbPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private string _walPath = string.Empty;
/// <summary>
/// Gets or sets the number of documents used per benchmark iteration.
/// Gets or sets the number of documents used per benchmark iteration.
/// </summary>
[Params(2_000)]
public int DocumentCount { get; set; }
private string _dbPath = string.Empty;
private string _walPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private DocumentCollection<Person> _collection = null!;
private List<ObjectId> _insertedIds = [];
/// <summary>
/// Prepares benchmark state and seed data for each iteration.
/// Prepares benchmark state and seed data for each iteration.
/// </summary>
[IterationSetup]
public void Setup()
@@ -53,17 +54,14 @@ public class CompactionBenchmarks
_transactionHolder.CommitAndReset();
_storage.Checkpoint();
for (var i = _insertedIds.Count - 1; i >= _insertedIds.Count / 3; i--)
{
_collection.Delete(_insertedIds[i]);
}
for (int i = _insertedIds.Count - 1; i >= _insertedIds.Count / 3; i--) _collection.Delete(_insertedIds[i]);
_transactionHolder.CommitAndReset();
_storage.Checkpoint();
}
/// <summary>
/// Cleans up benchmark resources and temporary files after each iteration.
/// Cleans up benchmark resources and temporary files after each iteration.
/// </summary>
[IterationCleanup]
public void Cleanup()
@@ -76,7 +74,7 @@ public class CompactionBenchmarks
}
/// <summary>
/// Benchmarks reclaimed file bytes reported by offline compaction.
/// Benchmarks reclaimed file bytes reported by offline compaction.
/// </summary>
/// <returns>The reclaimed file byte count.</returns>
[Benchmark(Baseline = true)]
@@ -95,7 +93,7 @@ public class CompactionBenchmarks
}
/// <summary>
/// Benchmarks tail bytes truncated by offline compaction.
/// Benchmarks tail bytes truncated by offline compaction.
/// </summary>
/// <returns>The truncated tail byte count.</returns>
[Benchmark]
@@ -135,7 +133,7 @@ public class CompactionBenchmarks
private static string BuildPayload(int seed)
{
var builder = new System.Text.StringBuilder(2500);
var builder = new StringBuilder(2500);
for (var i = 0; i < 80; i++)
{
builder.Append("compact-");
tests/CBDD.Tests.Benchmark/Compression/CompressionBenchmarks.cs
+24 -27
View File
@@ -1,7 +1,7 @@
using System.IO.Compression;
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using System.IO.Compression;
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using ZB.MOM.WW.CBDD.Core.Compression;
@@ -19,36 +19,36 @@ public class CompressionBenchmarks
{
private const int SeedCount = 300;
private const int WorkloadCount = 100;
private DocumentCollection<Person> _collection = null!;
private string _dbPath = string.Empty;
private Person[] _insertBatch = Array.Empty<Person>();
private ObjectId[] _seedIds = Array.Empty<ObjectId>();
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private string _walPath = string.Empty;
/// <summary>
/// Gets or sets whether compression is enabled for the benchmark run.
/// Gets or sets whether compression is enabled for the benchmark run.
/// </summary>
[Params(false, true)]
public bool EnableCompression { get; set; }
/// <summary>
/// Gets or sets the compression codec for the benchmark run.
/// Gets or sets the compression codec for the benchmark run.
/// </summary>
[Params(CompressionCodec.Brotli, CompressionCodec.Deflate)]
public CompressionCodec Codec { get; set; }
/// <summary>
/// Gets or sets the compression level for the benchmark run.
/// Gets or sets the compression level for the benchmark run.
/// </summary>
[Params(CompressionLevel.Fastest, CompressionLevel.Optimal)]
public CompressionLevel Level { get; set; }
private string _dbPath = string.Empty;
private string _walPath = string.Empty;
private StorageEngine _storage = null!;
private BenchmarkTransactionHolder _transactionHolder = null!;
private DocumentCollection<Person> _collection = null!;
private Person[] _insertBatch = Array.Empty<Person>();
private ObjectId[] _seedIds = Array.Empty<ObjectId>();
/// <summary>
/// Prepares benchmark storage and seed data for each iteration.
/// Prepares benchmark storage and seed data for each iteration.
/// </summary>
[IterationSetup]
public void Setup()
@@ -73,19 +73,19 @@ public class CompressionBenchmarks
_seedIds = new ObjectId[SeedCount];
for (var i = 0; i < SeedCount; i++)
{
var doc = CreatePerson(i, includeLargeBio: true);
var doc = CreatePerson(i, true);
_seedIds[i] = _collection.Insert(doc);
}
_transactionHolder.CommitAndReset();
_insertBatch = Enumerable.Range(SeedCount, WorkloadCount)
.Select(i => CreatePerson(i, includeLargeBio: true))
.Select(i => CreatePerson(i, true))
.ToArray();
}
/// <summary>
/// Cleans up benchmark resources for each iteration.
/// Cleans up benchmark resources for each iteration.
/// </summary>
[IterationCleanup]
public void Cleanup()
@@ -98,7 +98,7 @@ public class CompressionBenchmarks
}
/// <summary>
/// Benchmarks insert workload performance.
/// Benchmarks insert workload performance.
/// </summary>
[Benchmark(Baseline = true)]
[BenchmarkCategory("Compression_InsertUpdateRead")]
@@ -109,7 +109,7 @@ public class CompressionBenchmarks
}
/// <summary>
/// Benchmarks update workload performance.
/// Benchmarks update workload performance.
/// </summary>
[Benchmark]
[BenchmarkCategory("Compression_InsertUpdateRead")]
@@ -131,7 +131,7 @@ public class CompressionBenchmarks
}
/// <summary>
/// Benchmarks read workload performance.
/// Benchmarks read workload performance.
/// </summary>
[Benchmark]
[BenchmarkCategory("Compression_InsertUpdateRead")]
@@ -141,10 +141,7 @@ public class CompressionBenchmarks
for (var i = 0; i < WorkloadCount; i++)
{
var person = _collection.FindById(_seedIds[i]);
if (person != null)
{
checksum += person.Age;
}
if (person != null) checksum += person.Age;
}
_transactionHolder.CommitAndReset();
@@ -158,7 +155,7 @@ public class CompressionBenchmarks
Id = ObjectId.NewObjectId(),
FirstName = $"First_{i}",
LastName = $"Last_{i}",
Age = 20 + (i % 50),
Age = 20 + i % 50,
Bio = includeLargeBio ? BuildBio(i) : $"bio-{i}",
CreatedAt = DateTime.UnixEpoch.AddMinutes(i),
Balance = 100 + i,
@@ -183,7 +180,7 @@ public class CompressionBenchmarks
private static string BuildBio(int seed)
{
var builder = new System.Text.StringBuilder(4500);
var builder = new StringBuilder(4500);
for (var i = 0; i < 150; i++)
{
builder.Append("bio-");
tests/CBDD.Tests.Benchmark/Fixtures/Person.cs
+28 -18
View File
@@ -1,21 +1,21 @@
using ZB.MOM.WW.CBDD.Bson;
using System;
using System.Collections.Generic;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
public class Address
{
/// <summary>
/// Gets or sets the Street.
/// Gets or sets the Street.
/// </summary>
public string Street { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the City.
/// Gets or sets the City.
/// </summary>
public string City { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the ZipCode.
/// Gets or sets the ZipCode.
/// </summary>
public string ZipCode { get; set; } = string.Empty;
}
@@ -23,19 +23,22 @@ public class Address
public class WorkHistory
{
/// <summary>
/// Gets or sets the CompanyName.
/// Gets or sets the CompanyName.
/// </summary>
public string CompanyName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the Title.
/// Gets or sets the Title.
/// </summary>
public string Title { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the DurationYears.
/// Gets or sets the DurationYears.
/// </summary>
public int DurationYears { get; set; }
/// <summary>
/// Gets or sets the Tags.
/// Gets or sets the Tags.
/// </summary>
public List<string> Tags { get; set; } = new();
}
@@ -43,41 +46,48 @@ public class WorkHistory
public class Person
{
/// <summary>
/// Gets or sets the Id.
/// Gets or sets the Id.
/// </summary>
public ObjectId Id { get; set; }
/// <summary>
/// Gets or sets the FirstName.
/// Gets or sets the FirstName.
/// </summary>
public string FirstName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the LastName.
/// Gets or sets the LastName.
/// </summary>
public string LastName { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the Age.
/// Gets or sets the Age.
/// </summary>
public int Age { get; set; }
/// <summary>
/// Gets or sets the Bio.
/// Gets or sets the Bio.
/// </summary>
public string? Bio { get; set; } = string.Empty;
/// <summary>
/// Gets or sets the CreatedAt.
/// Gets or sets the CreatedAt.
/// </summary>
public DateTime CreatedAt { get; set; }
// Complex fields
/// <summary>
/// Gets or sets the Balance.
/// Gets or sets the Balance.
/// </summary>
public decimal Balance { get; set; }
/// <summary>
/// Gets or sets the HomeAddress.
/// Gets or sets the HomeAddress.
/// </summary>
public Address HomeAddress { get; set; } = new();
/// <summary>
/// Gets or sets the EmploymentHistory.
/// Gets or sets the EmploymentHistory.
/// </summary>
public List<WorkHistory> EmploymentHistory { get; set; } = new();
}
tests/CBDD.Tests.Benchmark/Fixtures/PersonMapper.cs
+38 -21
View File
@@ -1,7 +1,5 @@
using ZB.MOM.WW.CBDD.Bson;
using ZB.MOM.WW.CBDD.Core.Collections;
using System.Buffers;
using System.Runtime.InteropServices;
namespace ZB.MOM.WW.CBDD.Tests.Benchmark;
@@ -11,15 +9,21 @@ public class PersonMapper : ObjectIdMapperBase<Person>
public override string CollectionName => "people";
/// <inheritdoc />
public override ObjectId GetId(Person entity) => entity.Id;
public override ObjectId GetId(Person entity)
{
return entity.Id;
}
/// <inheritdoc />
public override void SetId(Person entity, ObjectId id) => entity.Id = id;
public override void SetId(Person entity, ObjectId id)
{
entity.Id = id;
}
/// <inheritdoc />
public override int Serialize(Person entity, BsonSpanWriter writer)
{
var sizePos = writer.BeginDocument();
int sizePos = writer.BeginDocument();
writer.WriteObjectId("_id", entity.Id);
writer.WriteString("firstname", entity.FirstName);
@@ -36,34 +40,32 @@ public class PersonMapper : ObjectIdMapperBase<Person>
writer.WriteDouble("balance", (double)entity.Balance);
// Nested Object: Address
var addrPos = writer.BeginDocument("homeaddress");
int addrPos = writer.BeginDocument("homeaddress");
writer.WriteString("street", entity.HomeAddress.Street);
writer.WriteString("city", entity.HomeAddress.City);
writer.WriteString("zipcode", entity.HomeAddress.ZipCode);
writer.EndDocument(addrPos);
// Collection: EmploymentHistory
var histPos = writer.BeginArray("employmenthistory");
for (int i = 0; i < entity.EmploymentHistory.Count; i++)
int histPos = writer.BeginArray("employmenthistory");
for (var i = 0; i < entity.EmploymentHistory.Count; i++)
{
var item = entity.EmploymentHistory[i];
// Array elements are keys "0", "1", "2"...
var itemPos = writer.BeginDocument(i.ToString());
int itemPos = writer.BeginDocument(i.ToString());
writer.WriteString("companyname", item.CompanyName);
writer.WriteString("title", item.Title);
writer.WriteInt32("durationyears", item.DurationYears);
// Nested Collection: Tags
var tagsPos = writer.BeginArray("tags");
for (int j = 0; j < item.Tags.Count; j++)
{
writer.WriteString(j.ToString(), item.Tags[j]);
}
int tagsPos = writer.BeginArray("tags");
for (var j = 0; j < item.Tags.Count; j++) writer.WriteString(j.ToString(), item.Tags[j]);
writer.EndArray(tagsPos);
writer.EndDocument(itemPos);
}
writer.EndArray(histPos);
writer.EndDocument(sizePos);
@@ -84,7 +86,7 @@ public class PersonMapper : ObjectIdMapperBase<Person>
if (type == BsonType.EndOfDocument)
break;
var name = reader.ReadElementHeader();
string name = reader.ReadElementHeader();
switch (name)
{
@@ -105,7 +107,7 @@ public class PersonMapper : ObjectIdMapperBase<Person>
{
var addrType = reader.ReadBsonType();
if (addrType == BsonType.EndOfDocument) break;
var addrName = reader.ReadElementHeader();
string addrName = reader.ReadElementHeader();
// We assume strict schema for benchmark speed, but should handle skipping
if (addrName == "street") person.HomeAddress.Street = reader.ReadString();
@@ -113,6 +115,7 @@ public class PersonMapper : ObjectIdMapperBase<Person>
else if (addrName == "zipcode") person.HomeAddress.ZipCode = reader.ReadString();
else reader.SkipValue(addrType);
}
break;
case "employmenthistory":
@@ -130,11 +133,20 @@ public class PersonMapper : ObjectIdMapperBase<Person>
{
var itemType = reader.ReadBsonType();
if (itemType == BsonType.EndOfDocument) break;
var itemName = reader.ReadElementHeader();
string itemName = reader.ReadElementHeader();
if (itemName == "companyname") workItem.CompanyName = reader.ReadString();
else if (itemName == "title") workItem.Title = reader.ReadString();
else if (itemName == "durationyears") workItem.DurationYears = reader.ReadInt32();
if (itemName == "companyname")
{
workItem.CompanyName = reader.ReadString();
}
else if (itemName == "title")
{
workItem.Title = reader.ReadString();
}
else if (itemName == "durationyears")
{
workItem.DurationYears = reader.ReadInt32();
}
else if (itemName == "tags")
{
reader.ReadDocumentSize(); // Enter Tags Array
@@ -149,10 +161,15 @@ public class PersonMapper : ObjectIdMapperBase<Person>
reader.SkipValue(tagType);
}
}
else reader.SkipValue(itemType);
else
{
reader.SkipValue(itemType);
}
}
person.EmploymentHistory.Add(workItem);
}
break;
default:
tests/CBDD.Tests.Benchmark/Infrastructure/BenchmarkTransactionHolder.cs
+15 -28
View File
@@ -1,4 +1,3 @@
using ZB.MOM.WW.CBDD.Core;
using ZB.MOM.WW.CBDD.Core.Storage;
using ZB.MOM.WW.CBDD.Core.Transactions;
@@ -11,7 +10,7 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
private ITransaction? _currentTransaction;
/// <summary>
/// Initializes a new instance of the <see cref="BenchmarkTransactionHolder"/> class.
/// Initializes a new instance of the <see cref="BenchmarkTransactionHolder" /> class.
/// </summary>
/// <param name="storage">The storage engine used to create transactions.</param>
public BenchmarkTransactionHolder(StorageEngine storage)
@@ -20,7 +19,15 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
}
/// <summary>
/// Gets the current active transaction or starts a new one.
/// Disposes this holder and rolls back any outstanding transaction.
/// </summary>
public void Dispose()
{
RollbackAndReset();
}
/// <summary>
/// Gets the current active transaction or starts a new one.
/// </summary>
/// <returns>The current active transaction.</returns>
public ITransaction GetCurrentTransactionOrStart()
@@ -28,16 +35,14 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
lock (_sync)
{
if (_currentTransaction == null || _currentTransaction.State != TransactionState.Active)
{
_currentTransaction = _storage.BeginTransaction();
}
return _currentTransaction;
}
}
/// <summary>
/// Gets the current active transaction or starts a new one asynchronously.
/// Gets the current active transaction or starts a new one asynchronously.
/// </summary>
/// <returns>A task that returns the current active transaction.</returns>
public Task<ITransaction> GetCurrentTransactionOrStartAsync()
@@ -46,22 +51,17 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
}
/// <summary>
/// Commits the current transaction when active and clears the holder.
/// Commits the current transaction when active and clears the holder.
/// </summary>
public void CommitAndReset()
{
lock (_sync)
{
if (_currentTransaction == null)
{
return;
}
if (_currentTransaction == null) return;
if (_currentTransaction.State == TransactionState.Active ||
_currentTransaction.State == TransactionState.Preparing)
{
_currentTransaction.Commit();
}
_currentTransaction.Dispose();
_currentTransaction = null;
@@ -69,33 +69,20 @@ internal sealed class BenchmarkTransactionHolder : ITransactionHolder, IDisposab
}
/// <summary>
/// Rolls back the current transaction when active and clears the holder.
/// Rolls back the current transaction when active and clears the holder.
/// </summary>
public void RollbackAndReset()
{
lock (_sync)
{
if (_currentTransaction == null)
{
return;
}
if (_currentTransaction == null) return;
if (_currentTransaction.State == TransactionState.Active ||
_currentTransaction.State == TransactionState.Preparing)
{
_currentTransaction.Rollback();
}
_currentTransaction.Dispose();
_currentTransaction = null;
}
}
/// <summary>
/// Disposes this holder and rolls back any outstanding transaction.
/// </summary>
public void Dispose()
{
RollbackAndReset();
}
}

Some files were not shown because too many files have changed in this diff Show More