docs: update documentation for extraction functions migration

- Add ExtractionFunctions.md reference document
- Update database-schema spec with 11 extraction functions
- Update data-access spec to document extraction function approach
- Update search-processing spec with new query builder interface
- Add Database.Tests to Testing.md architecture doc
- Update DataFlow.md with extraction function flow
This commit is contained in:
Joseph Doherty
2026-01-06 14:54:10 -05:00
parent 35c1e6baf0
commit c6aeb20d9c
8 changed files with 602 additions and 345 deletions
+2
View File
@@ -19,6 +19,8 @@ The search flow mirrors the legacy pattern, modernized for ASP.NET Core:
4. SearchProcessorService (BackgroundService) polls
└─> Finds queued searches
└─> Passes SearchId to query builder
└─> SQL extraction functions read criteria from Search.Criteria JSON
└─> Executes query against local cache
└─> Generates Excel via ClosedXML
└─> Stores result in Search.Results (VARBINARY)
+33 -1
View File
@@ -121,7 +121,7 @@ The scoping tool cache database includes these primary tables:
| Table | Purpose |
|-------|---------|
| `Search` | User search requests, status, and results (Excel as VARBINARY) |
| `Search` | User search requests, status, criteria (JSON), and results (Excel as VARBINARY) |
| `DataUpdate` | Tracks last sync timestamp per data type |
| `WorkOrder_Curr` | Current work orders from JDE |
| `WorkOrder_Hist` | Historical work orders from JDE |
@@ -134,6 +134,38 @@ The scoping tool cache database includes these primary tables:
| `ProfitCenter` | Profit center reference data |
| `SchemaVersions` | DbUp tracking table (auto-created) |
## Search Criteria Extraction Functions
The database includes SQL functions that extract filter criteria from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, with all filter extraction happening in SQL Server.
### Scalar Functions (3)
| Function | Returns | Extracts |
|----------|---------|----------|
| `fn_GetSearchMinimumDt` | DATETIME2 | `$.MinimumDt` |
| `fn_GetSearchMaximumDt` | DATETIME2 | `$.MaximumDt` |
| `fn_GetSearchExtractMisData` | BIT | `$.ExtractMisData` |
### Table-Valued Functions (8)
| Function | Returns | Extracts |
|----------|---------|----------|
| `fn_GetSearchWorkOrders` | WorkOrderNumber | `$.WorkOrderNumbers` array |
| `fn_GetSearchItemNumbers` | ItemNumber | `$.ItemNumbers` array |
| `fn_GetSearchProfitCenters` | Code | `$.ProfitCenters` array |
| `fn_GetSearchWorkCenters` | Code | `$.WorkCenters` array |
| `fn_GetSearchOperatorIDs` | OperatorID | `$.OperatorIDs` array |
| `fn_GetSearchComponentLots` | LotNumber, ItemNumber | `$.ComponentLotNumbers` array |
| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision | `$.PartOperations` array |
### Validation Procedure
| Procedure | Purpose |
|-----------|---------|
| `usp_ValidateSearchCriteria` | Validates search exists and has valid JSON criteria |
Error codes: 50001 (not found), 50002 (no criteria), 50003 (invalid JSON)
## Example Migration Scripts
### 001_CreateSearchTable.sql
+85 -16
View File
@@ -5,22 +5,34 @@ The test project uses xUnit for the framework, Shouldly for assertions, and NSub
## Project Structure
```
JdeScoping.Tests/
├── Unit/
│ ├── Services/
│ │ ├── SearchServiceTests.cs
│ │ ├── ExcelExportServiceTests.cs
│ │ └── DataSyncOrchestratorTests.cs
├── Repositories/
│ │ ── SearchRepositoryTests.cs
└── Models/
└── SearchCriteriaTests.cs
└── Integration/
── ApiTests/
├── SearchControllerTests.cs
── LookupControllerTests.cs
└── RepositoryTests/
└── JdeRepositoryTests.cs
tests/
├── JdeScoping.Tests/
│ ├── Unit/
│ │ ├── Services/
│ │ │ ├── SearchServiceTests.cs
│ │ │ ├── ExcelExportServiceTests.cs
│ │ └── DataSyncOrchestratorTests.cs
│ │ ── Repositories/
│ │ └── SearchRepositoryTests.cs
└── Models/
│ │ └── SearchCriteriaTests.cs
── Integration/
├── ApiTests/
── SearchControllerTests.cs
│ └── LookupControllerTests.cs
└── RepositoryTests/
│ └── JdeRepositoryTests.cs
├── JdeScoping.Api.IntegrationTests/
│ └── ... (API integration tests)
└── JdeScoping.Database.Tests/
├── Infrastructure/
│ └── DatabaseTestBase.cs
├── Functions/
│ ├── ScalarFunctionTests.cs
│ ├── SimpleTableFunctionTests.cs
│ └── ComplexTableFunctionTests.cs
└── Procedures/
└── ValidateSearchCriteriaProcedureTests.cs
```
## Unit Tests
@@ -175,7 +187,64 @@ public class SearchRepositoryIntegrationTests : IDisposable
}
```
## Database Tests
The `JdeScoping.Database.Tests` project tests SQL functions and stored procedures against a live SQL Server instance.
### Test Infrastructure
Tests inherit from `DatabaseTestBase` which provides:
- Connection management via `DatabaseTestFixture`
- Automatic test data cleanup
- Helper methods for inserting test searches
```csharp
[Collection("DatabaseTests")]
public class ScalarFunctionTests : DatabaseTestBase
{
[Fact]
public async Task fn_GetSearchMinimumDt_ValidSearch_ReturnsDateTime()
{
// Arrange
var criteria = new SearchCriteria { MinimumDt = new DateTime(2024, 6, 15) };
var searchId = await InsertTestSearchAsync(criteria);
// Act
var result = await Connection.QuerySingleOrDefaultAsync<DateTime?>(
"SELECT dbo.fn_GetSearchMinimumDt(@SearchId)",
new { SearchId = searchId });
// Assert
result.Should().BeCloseTo(criteria.MinimumDt.Value, TimeSpan.FromSeconds(1));
}
}
```
### Test Categories
| Category | Tests | Description |
|----------|-------|-------------|
| Scalar Functions | 15 | Test `fn_GetSearchMinimumDt`, `fn_GetSearchMaximumDt`, `fn_GetSearchExtractMisData` |
| Simple Table Functions | 38 | Test array extraction for work orders, items, profit centers, work centers, operators |
| Complex Table Functions | 23 | Test object extraction for component lots and part operations |
| Validation Procedure | 6 | Test `usp_ValidateSearchCriteria` error handling |
**Total: 82 database tests**
### Running Database Tests
```bash
# Run all database tests
dotnet test tests/JdeScoping.Database.Tests
# Run specific test class
dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ScalarFunctionTests"
```
**Prerequisites**: SQL Server must be running on localhost:1434 with the ScopingTool database.
## Related Documentation
- [Core Project](./CoreProject.md)
- [Dependencies](./Dependencies.md)
- [Database](./Database.md)