Add enterprise docs structure and include pending core maintenance updates.

docs/README.md

@@ -0,0 +1,25 @@
+# CBDD Documentation
+
+This folder is the canonical source for internal operational and engineering documentation.
+
+## Core Documents
+
+- Architecture: [`architecture.md`](architecture.md)
+- Deployment: [`deployment.md`](deployment.md)
+- Operations runbook: [`runbook.md`](runbook.md)
+- Security controls: [`security.md`](security.md)
+- Access model: [`access.md`](access.md)
+- Troubleshooting guide: [`troubleshooting.md`](troubleshooting.md)
+
+## Major Features
+
+- Feature inventory: [`features/README.md`](features/README.md)
+
+## Architecture Decisions
+
+- Initial ADR: [`adr/0001-storage-engine-and-source-generation.md`](adr/0001-storage-engine-and-source-generation.md)
+
+## Reference Specifications
+
+- Engine RFC: [`../RFC.md`](../RFC.md)
+- C-BSON format spec: [`../C-BSON.md`](../C-BSON.md)

docs/access.md

@@ -0,0 +1,32 @@
+# Access And Permissions
+
+## Roles
+
+- Maintainer: merge authority, release authority, incident ownership.
+- Reviewer: approves pull requests and validates architecture/security impact.
+- Contributor: proposes changes through pull requests.
+- Consumer: integrates published package versions in downstream applications.
+
+## Least-Privilege Model
+
+- Limit maintainer privileges to required release and incident responders.
+- Use reviewer role for routine code review and documentation updates.
+- Restrict package publishing credentials to release maintainers.
+
+## Approval Workflow
+
+1. Contributor opens pull request.
+2. Reviewer validates tests, documentation, and risk impact.
+3. Maintainer approves merge for high-risk or release-impacting changes.
+4. Release maintainer publishes approved release artifacts.
+
+## Periodic Access Review
+
+1. Review maintainer and publisher access quarterly.
+2. Remove inactive accounts and obsolete credentials.
+3. Confirm access ownership in repository settings and package feed controls.
+
+## Emergency Access
+
+- Temporary elevated access requires a tracked incident issue.
+- Remove temporary access immediately after incident closure.

docs/adr/0001-storage-engine-and-source-generation.md

@@ -0,0 +1,32 @@
+# 0001 Storage Engine And Source Generation
+
+## Status
+
+Accepted
+
+## Context
+
+CBDD targets embedded workloads where predictable latency and low operational overhead are priorities. Runtime reflection and remote database dependencies increase startup and runtime variance for this workload profile.
+
+## Decision
+
+1. Use an embedded storage engine with page-based persistence and WAL-backed transactions.
+2. Use compile-time source generation for mapping instead of runtime reflection.
+3. Keep query and indexing execution in-process for deterministic behavior.
+
+## Consequences
+
+Positive:
+- Lower runtime allocation and startup overhead.
+- Strong control over transaction and recovery behavior.
+- Predictable deployment for local/offline workloads.
+
+Trade-offs:
+- Greater maintenance burden for custom storage/query engine internals.
+- Source generator complexity requires dedicated regression coverage.
+
+## Related Documents
+
+- [`../architecture.md`](../architecture.md)
+- [`../runbook.md`](../runbook.md)
+- [`../../RFC.md`](../../RFC.md)

docs/architecture.md

@@ -0,0 +1,46 @@
+# Architecture
+
+## System Context
+
+CBDD is an embedded database library consumed in-process by .NET applications. The host application owns process lifecycle, filesystem placement, and operational policy.
+
+External dependencies:
+- .NET runtime and SDK
+- Local filesystem
+- Optional CI and package registry for build/release
+
+## Containers And Major Components
+
+1. `CBDD.Core`
+- Owns storage engine, transaction protocol, WAL, indexing, query planning, and CDC plumbing.
+
+2. `CBDD.Bson`
+- Owns BSON document model and span-based serialization/deserialization primitives.
+
+3. `CBDD.SourceGenerators`
+- Generates mapping code at compile time for entity serialization and collection initialization.
+
+4. Consumer application
+- Defines entities, `DocumentDbContext` subclasses, and operational behavior.
+
+## Data Flow
+
+1. Consumer invokes collection API through `DocumentDbContext`.
+2. Mapper layer serializes entities to BSON via generated mappers.
+3. Storage engine writes page data and WAL entries.
+4. Index subsystem updates primary and secondary indexes.
+5. Transaction commit persists durable state and emits CDC notifications where enabled.
+6. Query path evaluates expression plans and uses indexes or scan fallback.
+
+## Reliability Model
+
+- Write-ahead logging enforces durability before logical commit completion.
+- Snapshot isolation supports concurrent reads with transactional correctness.
+- Recovery logic replays WAL on restart to restore committed state.
+
+## Cross References
+
+- Operational procedures: [`runbook.md`](runbook.md)
+- Deployment and rollback: [`deployment.md`](deployment.md)
+- Security controls: [`security.md`](security.md)
+- Detailed protocol reference: [`../RFC.md`](../RFC.md)

docs/deployment.md

@@ -0,0 +1,60 @@
+# Deployment
+
+## Scope
+
+This workflow covers releasing CBDD as an internal package and promoting a validated version for downstream consumers.
+
+## Promotion Path
+
+1. Development validation on feature branch.
+2. Merge to `main` after required quality gates.
+3. Tag release and publish package artifact.
+4. Consumer rollout to target services/tools.
+
+## Pre-Deployment Checklist
+
+1. Run repository fitness gate.
+```bash
+bash scripts/fitness-check.sh
+```
+2. Verify no pending incidents in current release window.
+3. Confirm release notes include behavioral changes and migration notes.
+4. Confirm package version bump strategy.
+
+## Release Procedure
+
+1. Build release artifacts.
+```bash
+dotnet build CBDD.slnx -c Release
+dotnet test CBDD.slnx -c Release
+```
+2. Pack publishable projects.
+```bash
+dotnet pack src/CBDD/ZB.MOM.WW.CBDD.csproj -c Release -o nupkgs
+```
+3. Publish package to approved internal feed.
+4. Create release tag and attach release notes.
+5. Notify downstream teams of version and rollout guidance.
+
+## Post-Deployment Validation
+
+1. Install released package in a smoke-test consumer.
+2. Validate insert, query, transaction commit, and recovery startup behavior.
+3. Verify CDC, vector, and geospatial capabilities when used by consuming teams.
+
+## Rollback Procedure
+
+Trigger rollback when release validation fails or production consumers detect regression.
+
+1. Stop further promotions of the failing version.
+2. Revert to previous known-good package version in consumer manifests.
+3. If package registry supports unlisting/yanking, unlist the bad version.
+4. Open incident issue with impact, timeline, and containment actions.
+5. Prepare and validate patch release before re-promotion.
+
+## Emergency Change Path
+
+1. Create hotfix branch from last good tag.
+2. Apply minimal fix and run full quality gates.
+3. Require maintainer approval.
+4. Publish patched version and communicate mandatory upgrade guidance.

docs/features/README.md

@@ -0,0 +1,10 @@
+# Feature Inventory
+
+The following documents are the canonical reference for major CBDD capabilities:
+
+- [Storage and transactions](storage-transactions.md)
+- [Query and indexing](query-and-indexing.md)
+- [Vector search](vector-search.md)
+- [Geospatial search](geospatial-search.md)
+- [Change data capture](change-data-capture.md)
+- [Source-generated mapping](source-generated-mapping.md)

docs/features/change-data-capture.md

@@ -0,0 +1,61 @@
+# Change Data Capture
+
+## Purpose And Business Outcome
+
+Expose transactional change events so consumers can react to committed inserts, updates, and deletes.
+
+## Scope And Non-Goals
+
+Scope:
+- Collection-level change stream subscriptions
+- Event publication after successful commit
+
+Non-goals:
+- Cross-process event transport guarantees
+- External message broker delivery semantics
+
+## User And System Workflows
+
+1. Consumer subscribes to a collection change stream.
+2. Application performs data mutations.
+3. On commit, CDC publishes events to active subscribers.
+4. Subscriber handlers process entity and metadata payloads.
+
+## Interfaces And APIs
+
+- `Watch(...)` collection API
+- `ChangeStreamObservable`
+- `ChangeStreamDispatcher`
+- `ChangeStreamEvent`
+
+## Permissions And Data Handling
+
+- CDC payloads can include document identifiers and entity data.
+- Restrict subscription access to trusted application components.
+
+## Dependencies And Failure Modes
+
+Dependencies:
+- Transaction commit lifecycle
+- Subscriber channel health
+
+Failure modes:
+- Subscriber backpressure or handler exceptions
+- Event handling assumptions that conflict with rollback behavior
+
+## Monitoring, Alerts, And Troubleshooting
+
+- Review CDC behavior in integration and scalability tests.
+- Follow [`../runbook.md`](../runbook.md) for incident response.
+- Follow [`../security.md`](../security.md) for event payload handling controls.
+- Use [`../troubleshooting.md`](../troubleshooting.md) for diagnosis guidance.
+
+## Rollout And Change Considerations
+
+- Behavioral CDC changes require explicit release-note callouts.
+- Maintain compatibility expectations for event payload shape.
+
+## Validation Guidance
+
+- Run `CdcTests` and `CdcScalabilityTests` before release.
+- Validate commit-only emission behavior in regression tests.

docs/features/geospatial-search.md

@@ -0,0 +1,60 @@
+# Geospatial Search
+
+## Purpose And Business Outcome
+
+Support location-aware queries such as nearest-neighbor and bounding-box lookups for geospatial workloads.
+
+## Scope And Non-Goals
+
+Scope:
+- Spatial index configuration
+- Proximity and bounding-box query operations
+
+Non-goals:
+- Full GIS projection transformations
+- External map tile services
+
+## User And System Workflows
+
+1. Consumer configures spatial index for coordinate fields.
+2. Coordinates are serialized and stored with entity payloads.
+3. Query path evaluates `Near` or `Within` predicates.
+4. Engine returns matching entities.
+
+## Interfaces And APIs
+
+- Spatial index APIs in model configuration
+- Query helpers under `GeoSpatialExtensions`
+- Index implementation in `RTreeIndex`
+
+## Permissions And Data Handling
+
+- Geolocation data may be regulated or privacy-sensitive.
+- Apply least-privilege access and retention limits.
+
+## Dependencies And Failure Modes
+
+Dependencies:
+- Correct coordinate format and units
+- Spatial index consistency
+
+Failure modes:
+- Invalid coordinate values
+- Unexpected results from bounding definitions or radius units
+
+## Monitoring, Alerts, And Troubleshooting
+
+- Validate geospatial paths through dedicated stress and correctness tests.
+- Use [`../runbook.md`](../runbook.md) for escalation.
+- Follow [`../security.md`](../security.md) for geolocation data protection controls.
+- Use [`../troubleshooting.md`](../troubleshooting.md#query-and-index-issues) for issue resolution.
+
+## Rollout And Change Considerations
+
+- Document coordinate and unit assumptions in release notes when behavior changes.
+- Validate backward compatibility for persisted spatial index pages.
+
+## Validation Guidance
+
+- Run `GeospatialTests` and `GeospatialStressTests` before release.
+- Include representative proximity/bounding queries in smoke checks.

docs/features/query-and-indexing.md

@@ -0,0 +1,62 @@
+# Query And Indexing
+
+## Purpose And Business Outcome
+
+Deliver predictable query correctness and performance using expression translation with index-aware execution.
+
+## Scope And Non-Goals
+
+Scope:
+- LINQ query translation
+- Primary and secondary index usage
+- Scan fallback where index optimization is not applicable
+
+Non-goals:
+- Full SQL compatibility
+- Distributed query federation
+
+## User And System Workflows
+
+1. Consumer submits LINQ expression.
+2. Query planner evaluates index opportunities.
+3. Engine executes index-backed or scan path.
+4. Results are materialized to consumer entities.
+
+## Interfaces And APIs
+
+- `AsQueryable()` and query provider components
+- `CollectionIndexManager`
+- Index implementations under `src/CBDD.Core/Indexing`
+
+## Permissions And Data Handling
+
+- Query access follows host application authorization policy.
+- Indexed data inherits the same sensitivity classification as source payloads.
+
+## Dependencies And Failure Modes
+
+Dependencies:
+- Valid index metadata and storage consistency
+- Expression visitor correctness
+
+Failure modes:
+- Incorrect predicate translation
+- Missing/ineffective indexes
+- Performance regressions due to scan-heavy paths
+
+## Monitoring, Alerts, And Troubleshooting
+
+- Track query-related regressions in automated tests.
+- Use [`../runbook.md`](../runbook.md) for incident handling.
+- Follow [`../security.md`](../security.md) for sensitive data and access constraints.
+- Use [`../troubleshooting.md`](../troubleshooting.md#query-and-index-issues) for remediation.
+
+## Rollout And Change Considerations
+
+- Query planner/index behavior changes require benchmark comparison and regression coverage.
+- Document breaking semantics in release notes.
+
+## Validation Guidance
+
+- Run query, index, and optimizer test suites in `tests/CBDD.Tests`.
+- Confirm coverage gate with `bash scripts/coverage-check.sh`.

docs/features/source-generated-mapping.md

@@ -0,0 +1,61 @@
+# Source-Generated Mapping
+
+## Purpose And Business Outcome
+
+Generate compile-time mapping code to reduce runtime overhead and reflection risk for serialization paths.
+
+## Scope And Non-Goals
+
+Scope:
+- Entity metadata analysis
+- Mapper source generation
+- Collection initialization helpers
+
+Non-goals:
+- Runtime dynamic mapping for unknown schemas
+- Support for unsupported C# patterns outside generator design
+
+## User And System Workflows
+
+1. Consumer defines entities and context patterns.
+2. Build invokes source generator.
+3. Generated mapper code is compiled into target project.
+4. Runtime serialization path uses generated code.
+
+## Interfaces And APIs
+
+- Source generator project under `src/CBDD.SourceGenerators`
+- Attributes in BSON and data-annotation mapping surface
+- Generated initialization methods for context collections
+
+## Permissions And Data Handling
+
+- Generated code can expose field-level mapping behavior.
+- Repository write permissions should be limited to trusted contributors.
+
+## Dependencies And Failure Modes
+
+Dependencies:
+- Roslyn source generator execution during build
+- Entity schema conventions
+
+Failure modes:
+- Missing generation due to invalid entity declarations
+- Serialization mismatch caused by attribute/model changes
+
+## Monitoring, Alerts, And Troubleshooting
+
+- Monitor build output for generator diagnostics.
+- Use [`../runbook.md`](../runbook.md) for escalation.
+- Follow [`../security.md`](../security.md) for review and control expectations.
+- Use [`../troubleshooting.md`](../troubleshooting.md#source-generation-issues) for remediation steps.
+
+## Rollout And Change Considerations
+
+- Generator behavioral changes require broad regression tests across entities.
+- Document any new constraints or unsupported patterns in release notes.
+
+## Validation Guidance
+
+- Run source generator and serialization tests in `tests/CBDD.Tests`.
+- Validate mapper generation with clean `dotnet build` in CI.

docs/features/storage-transactions.md

@@ -0,0 +1,63 @@
+# Storage And Transactions
+
+## Purpose And Business Outcome
+
+Provide durable, ACID-compliant local persistence for embedded workloads that need consistent commit and recovery semantics.
+
+## Scope And Non-Goals
+
+Scope:
+- Page-based storage
+- Write-ahead logging
+- Transaction lifecycle and commit/rollback semantics
+
+Non-goals:
+- Distributed transactions
+- Multi-node replication
+
+## User And System Workflows
+
+1. Application writes through `DocumentDbContext`.
+2. Engine records WAL entries.
+3. Commit persists pages and marks transaction durable.
+4. Recovery replays WAL to restore committed state after restart.
+
+## Interfaces And APIs
+
+- `DocumentDbContext`
+- `Transaction` and `ITransaction`
+- `WriteAheadLog`
+- Storage engine modules under `src/CBDD.Core/Storage`
+
+## Permissions And Data Handling
+
+- Database files require host-managed filesystem access controls.
+- Transaction data should be treated as sensitive if payloads contain regulated information.
+
+## Dependencies And Failure Modes
+
+Dependencies:
+- Local filesystem I/O
+- WAL and page file consistency
+
+Failure modes:
+- Interrupted writes
+- Corrupted WAL entries
+- Invalid page metadata after unsafe process termination
+
+## Monitoring, Alerts, And Troubleshooting
+
+- Use CI/test failures and incident issues as primary signals.
+- Follow [`../runbook.md`](../runbook.md) for triage.
+- Follow [`../security.md`](../security.md) for data handling and control requirements.
+- Use [`../troubleshooting.md`](../troubleshooting.md#data-file-and-recovery-issues) for recovery issues.
+
+## Rollout And Change Considerations
+
+- Any storage format or WAL behavior change requires migration and rollback validation.
+- Release notes must document backward compatibility impact.
+
+## Validation Guidance
+
+- Run transaction and recovery tests in `tests/CBDD.Tests`.
+- Execute `dotnet test CBDD.slnx -c Release` before merge.

docs/features/vector-search.md

@@ -0,0 +1,60 @@
+# Vector Search
+
+## Purpose And Business Outcome
+
+Enable similarity search for embedding-driven workloads directly in embedded storage.
+
+## Scope And Non-Goals
+
+Scope:
+- Vector index configuration
+- Approximate nearest-neighbor query execution
+
+Non-goals:
+- External model training
+- Cross-database vector federation
+
+## User And System Workflows
+
+1. Consumer registers vector index for embedding field.
+2. Documents persist embeddings in collection payloads.
+3. Query issues vector search request with `k` nearest neighbors.
+4. Engine returns ranked matches.
+
+## Interfaces And APIs
+
+- Vector index configuration via model builder
+- Query extensions under `VectorSearchExtensions`
+- Index implementation in `VectorSearchIndex`
+
+## Permissions And Data Handling
+
+- Embeddings may contain sensitive semantic information.
+- Apply host-level access restrictions and retention controls.
+
+## Dependencies And Failure Modes
+
+Dependencies:
+- Correct embedding dimensionality
+- Index parameter tuning for workload
+
+Failure modes:
+- Dimension mismatch between data and query vectors
+- Poor recall due to incorrect index configuration
+
+## Monitoring, Alerts, And Troubleshooting
+
+- Validate vector query quality during release smoke checks.
+- Use [`../runbook.md`](../runbook.md) for incident handling.
+- Follow [`../security.md`](../security.md) for embedding-data handling controls.
+- Use [`../troubleshooting.md`](../troubleshooting.md#query-and-index-issues) for vector query remediation.
+
+## Rollout And Change Considerations
+
+- Treat vector index parameter changes as performance-sensitive releases.
+- Document compatibility impact for existing persisted indexes.
+
+## Validation Guidance
+
+- Run vector search tests in `tests/CBDD.Tests/VectorSearchTests.cs`.
+- Add benchmark runs for large-vector workloads before release.

docs/runbook.md

@@ -0,0 +1,56 @@
+# Runbook
+
+## Purpose
+
+This runbook provides standard operations, incident triage, escalation, and recovery procedures for CBDD maintainers.
+
+## Signals And Entry Points
+
+- CI failures on `main`
+- Failing integration tests in consumer repositories
+- Regression issues labeled `incident`
+- Recovery or corruption reports from consumers
+
+## Alert Triage Procedure
+
+1. Capture incident context: version, environment, failing operation, and first failure timestamp.
+2. Classify severity:
+- `SEV-1`: data loss risk, persistent startup failure, or transaction correctness risk.
+- `SEV-2`: feature-level regression without confirmed data loss.
+- `SEV-3`: non-critical behavior or documentation defects.
+3. Create or update the incident issue with owner and current mitigation status.
+4. Reproduce with targeted tests in `/Users/dohertj2/Desktop/CBDD/tests/CBDD.Tests`.
+
+## Diagnostics
+
+1. Validate build and tests.
+```bash
+dotnet test CBDD.slnx -c Release
+```
+2. Run coverage threshold gate when behavior changed in core paths.
+```bash
+bash scripts/coverage-check.sh
+```
+3. For storage and recovery incidents, prioritize:
+- `StorageEngine.Recovery`
+- `WriteAheadLog`
+- transaction protocol tests
+
+## Escalation Path
+
+1. Initial owner: maintainer on incident issue.
+2. Escalate to release maintainer when severity is `SEV-1` or rollback is required.
+3. Communicate status updates on each milestone: triage complete, mitigation active, fix merged, validation complete.
+
+## Recovery Actions
+
+1. Contain impact by pinning consumers to last known-good package version.
+2. Apply rollback steps from [`deployment.md`](deployment.md#rollback-procedure).
+3. Validate repaired build with targeted and full regression suites.
+4. Publish fixed package and confirm consumer recovery.
+
+## Post-Incident Expectations
+
+1. Document root cause, blast radius, and timeline.
+2. Add regression tests to prevent recurrence.
+3. Record follow-up actions in issue tracker with owners and due dates.

docs/security.md

@@ -0,0 +1,34 @@
+# Security
+
+## Scope
+
+CBDD is an embedded data engine. Security controls are shared between the library and the host application that embeds it.
+
+## Authentication And Authorization Model
+
+- CBDD does not provide built-in user authentication.
+- Authorization is enforced by the host process and filesystem permissions.
+- Access to database files must be limited to trusted service identities.
+
+## Data Classification And Handling
+
+- Treat persisted database files as sensitive when they contain customer or regulated data.
+- Do not store secrets in source, fixtures, or benchmark datasets.
+- Apply environment-specific retention and backup controls outside this repository.
+
+## Storage And Cryptography Controls
+
+- CBDD enforces integrity through WAL and transactional semantics.
+- Encryption at rest and key management are host responsibilities.
+- If encryption is required, use filesystem or volume-level encryption managed by platform security controls.
+
+## Secure Coding Expectations
+
+1. Require code review for storage, WAL, indexing, query, and serialization changes.
+2. Add targeted tests for all security-relevant behavior changes.
+3. Run package vulnerability checks in fitness pipeline.
+
+## Incident Handling
+
+- Follow [`runbook.md`](runbook.md) for incident triage and escalation.
+- Label security-impacting issues with `security` and prioritize immediate containment.

docs/troubleshooting.md

@@ -0,0 +1,62 @@
+# Troubleshooting
+
+## Build And Test Failures
+
+### Symptom
+`dotnet build` or `dotnet test` fails locally or in CI.
+
+### Checks
+1. Verify .NET 10 SDK is installed.
+2. Run `dotnet restore`.
+3. Run `dotnet format --verify-no-changes`.
+4. Re-run tests with `dotnet test CBDD.slnx -c Release`.
+
+### Resolution
+- Fix reported compile/test failures before merge.
+- For flaky tests, isolate affected test class and open an issue with reproduction details.
+
+## Data File And Recovery Issues
+
+### Symptom
+Database startup fails or recovery path throws WAL/storage errors.
+
+### Checks
+1. Capture exact exception and stack trace.
+2. Reproduce with storage/recovery-focused tests.
+3. Validate rollback path from [`deployment.md`](deployment.md#rollback-procedure).
+
+### Resolution
+- Pin consumers to last known-good package.
+- Apply fix and add regression coverage in recovery/transaction tests.
+
+## Query And Index Issues
+
+### Symptom
+Unexpected query performance or incorrect query results.
+
+### Checks
+1. Verify relevant indexes are configured for query predicates.
+2. Reproduce with test cases in `tests/CBDD.Tests` for query/index modules.
+3. Validate behavior for scan fallback and expression translation.
+
+### Resolution
+- Add or adjust index definitions and query tests.
+- Document any changed query semantics in release notes.
+
+## Source Generation Issues
+
+### Symptom
+Generated mappers missing or serialization behavior differs from expectations.
+
+### Checks
+1. Verify entity attributes and accessibility are valid for source generation.
+2. Build solution to regenerate mapper output.
+3. Validate related tests in source generator test coverage.
+
+### Resolution
+- Update entity annotations or generator logic.
+- Add focused regression tests for unsupported pattern handling.
+
+## Escalation
+
+If troubleshooting steps do not resolve the issue, follow incident escalation in [`runbook.md`](runbook.md).