docs: update documentation for extraction functions migration
- Add ExtractionFunctions.md reference document - Update database-schema spec with 11 extraction functions - Update data-access spec to document extraction function approach - Update search-processing spec with new query builder interface - Add Database.Tests to Testing.md architecture doc - Update DataFlow.md with extraction function flow
This commit is contained in:
Joseph Doherty
@@ -19,6 +19,8 @@ The search flow mirrors the legacy pattern, modernized for ASP.NET Core:
|
||||
|
||||
4. SearchProcessorService (BackgroundService) polls
|
||||
└─> Finds queued searches
|
||||
└─> Passes SearchId to query builder
|
||||
└─> SQL extraction functions read criteria from Search.Criteria JSON
|
||||
└─> Executes query against local cache
|
||||
└─> Generates Excel via ClosedXML
|
||||
└─> Stores result in Search.Results (VARBINARY)
|
||||
|
||||
@@ -121,7 +121,7 @@ The scoping tool cache database includes these primary tables:
|
||||
|
||||
| Table | Purpose |
|
||||
|-------|---------|
|
||||
| `Search` | User search requests, status, and results (Excel as VARBINARY) |
|
||||
| `Search` | User search requests, status, criteria (JSON), and results (Excel as VARBINARY) |
|
||||
| `DataUpdate` | Tracks last sync timestamp per data type |
|
||||
| `WorkOrder_Curr` | Current work orders from JDE |
|
||||
| `WorkOrder_Hist` | Historical work orders from JDE |
|
||||
@@ -134,6 +134,38 @@ The scoping tool cache database includes these primary tables:
|
||||
| `ProfitCenter` | Profit center reference data |
|
||||
| `SchemaVersions` | DbUp tracking table (auto-created) |
|
||||
|
||||
## Search Criteria Extraction Functions
|
||||
|
||||
The database includes SQL functions that extract filter criteria from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, with all filter extraction happening in SQL Server.
|
||||
|
||||
### Scalar Functions (3)
|
||||
|
||||
| Function | Returns | Extracts |
|
||||
|----------|---------|----------|
|
||||
| `fn_GetSearchMinimumDt` | DATETIME2 | `$.MinimumDt` |
|
||||
| `fn_GetSearchMaximumDt` | DATETIME2 | `$.MaximumDt` |
|
||||
| `fn_GetSearchExtractMisData` | BIT | `$.ExtractMisData` |
|
||||
|
||||
### Table-Valued Functions (8)
|
||||
|
||||
| Function | Returns | Extracts |
|
||||
|----------|---------|----------|
|
||||
| `fn_GetSearchWorkOrders` | WorkOrderNumber | `$.WorkOrderNumbers` array |
|
||||
| `fn_GetSearchItemNumbers` | ItemNumber | `$.ItemNumbers` array |
|
||||
| `fn_GetSearchProfitCenters` | Code | `$.ProfitCenters` array |
|
||||
| `fn_GetSearchWorkCenters` | Code | `$.WorkCenters` array |
|
||||
| `fn_GetSearchOperatorIDs` | OperatorID | `$.OperatorIDs` array |
|
||||
| `fn_GetSearchComponentLots` | LotNumber, ItemNumber | `$.ComponentLotNumbers` array |
|
||||
| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision | `$.PartOperations` array |
|
||||
|
||||
### Validation Procedure
|
||||
|
||||
| Procedure | Purpose |
|
||||
|-----------|---------|
|
||||
| `usp_ValidateSearchCriteria` | Validates search exists and has valid JSON criteria |
|
||||
|
||||
Error codes: 50001 (not found), 50002 (no criteria), 50003 (invalid JSON)
|
||||
|
||||
## Example Migration Scripts
|
||||
|
||||
### 001_CreateSearchTable.sql
|
||||
|
||||
@@ -5,22 +5,34 @@ The test project uses xUnit for the framework, Shouldly for assertions, and NSub
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
JdeScoping.Tests/
|
||||
├── Unit/
|
||||
│ ├── Services/
|
||||
│ │ ├── SearchServiceTests.cs
|
||||
│ │ ├── ExcelExportServiceTests.cs
|
||||
│ │ └── DataSyncOrchestratorTests.cs
|
||||
│ ├── Repositories/
|
||||
│ │ └── SearchRepositoryTests.cs
|
||||
│ └── Models/
|
||||
│ └── SearchCriteriaTests.cs
|
||||
└── Integration/
|
||||
├── ApiTests/
|
||||
│ ├── SearchControllerTests.cs
|
||||
│ └── LookupControllerTests.cs
|
||||
└── RepositoryTests/
|
||||
└── JdeRepositoryTests.cs
|
||||
tests/
|
||||
├── JdeScoping.Tests/
|
||||
│ ├── Unit/
|
||||
│ │ ├── Services/
|
||||
│ │ │ ├── SearchServiceTests.cs
|
||||
│ │ │ ├── ExcelExportServiceTests.cs
|
||||
│ │ │ └── DataSyncOrchestratorTests.cs
|
||||
│ │ ├── Repositories/
|
||||
│ │ │ └── SearchRepositoryTests.cs
|
||||
│ │ └── Models/
|
||||
│ │ └── SearchCriteriaTests.cs
|
||||
│ └── Integration/
|
||||
│ ├── ApiTests/
|
||||
│ │ ├── SearchControllerTests.cs
|
||||
│ │ └── LookupControllerTests.cs
|
||||
│ └── RepositoryTests/
|
||||
│ └── JdeRepositoryTests.cs
|
||||
├── JdeScoping.Api.IntegrationTests/
|
||||
│ └── ... (API integration tests)
|
||||
└── JdeScoping.Database.Tests/
|
||||
├── Infrastructure/
|
||||
│ └── DatabaseTestBase.cs
|
||||
├── Functions/
|
||||
│ ├── ScalarFunctionTests.cs
|
||||
│ ├── SimpleTableFunctionTests.cs
|
||||
│ └── ComplexTableFunctionTests.cs
|
||||
└── Procedures/
|
||||
└── ValidateSearchCriteriaProcedureTests.cs
|
||||
```
|
||||
|
||||
## Unit Tests
|
||||
@@ -175,7 +187,64 @@ public class SearchRepositoryIntegrationTests : IDisposable
|
||||
}
|
||||
```
|
||||
|
||||
## Database Tests
|
||||
|
||||
The `JdeScoping.Database.Tests` project tests SQL functions and stored procedures against a live SQL Server instance.
|
||||
|
||||
### Test Infrastructure
|
||||
|
||||
Tests inherit from `DatabaseTestBase` which provides:
|
||||
- Connection management via `DatabaseTestFixture`
|
||||
- Automatic test data cleanup
|
||||
- Helper methods for inserting test searches
|
||||
|
||||
```csharp
|
||||
[Collection("DatabaseTests")]
|
||||
public class ScalarFunctionTests : DatabaseTestBase
|
||||
{
|
||||
[Fact]
|
||||
public async Task fn_GetSearchMinimumDt_ValidSearch_ReturnsDateTime()
|
||||
{
|
||||
// Arrange
|
||||
var criteria = new SearchCriteria { MinimumDt = new DateTime(2024, 6, 15) };
|
||||
var searchId = await InsertTestSearchAsync(criteria);
|
||||
|
||||
// Act
|
||||
var result = await Connection.QuerySingleOrDefaultAsync<DateTime?>(
|
||||
"SELECT dbo.fn_GetSearchMinimumDt(@SearchId)",
|
||||
new { SearchId = searchId });
|
||||
|
||||
// Assert
|
||||
result.Should().BeCloseTo(criteria.MinimumDt.Value, TimeSpan.FromSeconds(1));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Test Categories
|
||||
|
||||
| Category | Tests | Description |
|
||||
|----------|-------|-------------|
|
||||
| Scalar Functions | 15 | Test `fn_GetSearchMinimumDt`, `fn_GetSearchMaximumDt`, `fn_GetSearchExtractMisData` |
|
||||
| Simple Table Functions | 38 | Test array extraction for work orders, items, profit centers, work centers, operators |
|
||||
| Complex Table Functions | 23 | Test object extraction for component lots and part operations |
|
||||
| Validation Procedure | 6 | Test `usp_ValidateSearchCriteria` error handling |
|
||||
|
||||
**Total: 82 database tests**
|
||||
|
||||
### Running Database Tests
|
||||
|
||||
```bash
|
||||
# Run all database tests
|
||||
dotnet test tests/JdeScoping.Database.Tests
|
||||
|
||||
# Run specific test class
|
||||
dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ScalarFunctionTests"
|
||||
```
|
||||
|
||||
**Prerequisites**: SQL Server must be running on localhost:1434 with the ScopingTool database.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Core Project](./CoreProject.md)
|
||||
- [Dependencies](./Dependencies.md)
|
||||
- [Database](./Database.md)
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
# Search Criteria Extraction Functions
|
||||
|
||||
SQL functions that extract filter values from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, simplifying the C# to SQL interface.
|
||||
|
||||
## Overview
|
||||
|
||||
| Type | Count | Purpose |
|
||||
|------|-------|---------|
|
||||
| Scalar Functions | 3 | Extract single values (dates, booleans) |
|
||||
| Table-Valued Functions | 8 | Extract arrays and object collections |
|
||||
| Validation Procedure | 1 | Pre-flight validation with error codes |
|
||||
|
||||
## Scalar Functions
|
||||
|
||||
### fn_GetSearchMinimumDt
|
||||
|
||||
Extracts the minimum date filter value.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchMinimumDt(@SearchId INT)
|
||||
RETURNS DATETIME2(7)
|
||||
```
|
||||
|
||||
**JSON Path**: `$.MinimumDt`
|
||||
|
||||
**Returns**: DATETIME2 value or NULL if not found/invalid
|
||||
|
||||
### fn_GetSearchMaximumDt
|
||||
|
||||
Extracts the maximum date filter value.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchMaximumDt(@SearchId INT)
|
||||
RETURNS DATETIME2(7)
|
||||
```
|
||||
|
||||
**JSON Path**: `$.MaximumDt`
|
||||
|
||||
**Returns**: DATETIME2 value or NULL if not found/invalid
|
||||
|
||||
### fn_GetSearchExtractMisData
|
||||
|
||||
Extracts the MIS data extraction flag.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchExtractMisData(@SearchId INT)
|
||||
RETURNS BIT
|
||||
```
|
||||
|
||||
**JSON Path**: `$.ExtractMisData`
|
||||
|
||||
**Returns**: 1 (true), 0 (false), or NULL if not found/invalid
|
||||
|
||||
## Table-Valued Functions (Simple Arrays)
|
||||
|
||||
### fn_GetSearchWorkOrders
|
||||
|
||||
Extracts work order number filter values.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
|
||||
RETURNS TABLE
|
||||
AS RETURN (
|
||||
SELECT j.WorkOrderNumber
|
||||
FROM ... OPENJSON(..., '$.WorkOrderNumbers') WITH (WorkOrderNumber BIGINT '$') j
|
||||
)
|
||||
```
|
||||
|
||||
**JSON Path**: `$.WorkOrderNumbers`
|
||||
|
||||
**Returns**: Table with `WorkOrderNumber BIGINT` column
|
||||
|
||||
### fn_GetSearchItemNumbers
|
||||
|
||||
Extracts item number filter values.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchItemNumbers(@SearchId INT)
|
||||
RETURNS TABLE
|
||||
```
|
||||
|
||||
**JSON Path**: `$.ItemNumbers`
|
||||
|
||||
**Returns**: Table with `ItemNumber VARCHAR(128)` column
|
||||
|
||||
### fn_GetSearchProfitCenters
|
||||
|
||||
Extracts profit center filter values.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchProfitCenters(@SearchId INT)
|
||||
RETURNS TABLE
|
||||
```
|
||||
|
||||
**JSON Path**: `$.ProfitCenters`
|
||||
|
||||
**Returns**: Table with `Code VARCHAR(12)` column
|
||||
|
||||
### fn_GetSearchWorkCenters
|
||||
|
||||
Extracts work center filter values.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchWorkCenters(@SearchId INT)
|
||||
RETURNS TABLE
|
||||
```
|
||||
|
||||
**JSON Path**: `$.WorkCenters`
|
||||
|
||||
**Returns**: Table with `Code VARCHAR(12)` column
|
||||
|
||||
### fn_GetSearchOperatorIDs
|
||||
|
||||
Extracts operator ID filter values.
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchOperatorIDs(@SearchId INT)
|
||||
RETURNS TABLE
|
||||
```
|
||||
|
||||
**JSON Path**: `$.OperatorIDs`
|
||||
|
||||
**Returns**: Table with `OperatorID VARCHAR(128)` column
|
||||
|
||||
## Table-Valued Functions (Complex Objects)
|
||||
|
||||
### fn_GetSearchComponentLots
|
||||
|
||||
Extracts component lot filter values (lot/item pairs).
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchComponentLots(@SearchId INT)
|
||||
RETURNS TABLE
|
||||
AS RETURN (
|
||||
SELECT j.LotNumber, j.ItemNumber
|
||||
FROM ... OPENJSON(..., '$.ComponentLotNumbers') WITH (
|
||||
LotNumber VARCHAR(30) '$.LotNumber',
|
||||
ItemNumber VARCHAR(128) '$.ItemNumber'
|
||||
) j
|
||||
)
|
||||
```
|
||||
|
||||
**JSON Path**: `$.ComponentLotNumbers`
|
||||
|
||||
**Returns**: Table with `LotNumber VARCHAR(30)`, `ItemNumber VARCHAR(128)` columns
|
||||
|
||||
### fn_GetSearchPartOperations
|
||||
|
||||
Extracts part operation filter values (item/operation/MIS combinations).
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchPartOperations(@SearchId INT)
|
||||
RETURNS TABLE
|
||||
AS RETURN (
|
||||
SELECT j.ItemNumber, j.OperationNumber, j.MisNumber, j.MisRevision
|
||||
FROM ... OPENJSON(..., '$.PartOperations') WITH (
|
||||
ItemNumber VARCHAR(128) '$.ItemNumber',
|
||||
OperationNumber VARCHAR(10) '$.OperationNumber',
|
||||
MisNumber VARCHAR(10) '$.MisNumber',
|
||||
MisRevision VARCHAR(10) '$.MisRevision'
|
||||
) j
|
||||
)
|
||||
```
|
||||
|
||||
**JSON Path**: `$.PartOperations`
|
||||
|
||||
**Returns**: Table with `ItemNumber`, `OperationNumber`, `MisNumber`, `MisRevision` columns
|
||||
|
||||
## Validation Procedure
|
||||
|
||||
### usp_ValidateSearchCriteria
|
||||
|
||||
Validates that a search exists and has valid JSON criteria.
|
||||
|
||||
```sql
|
||||
CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT)
|
||||
```
|
||||
|
||||
**Error Codes**:
|
||||
|
||||
| Code | Condition | Message |
|
||||
|------|-----------|---------|
|
||||
| 50001 | Search not found | "Search ID {id} not found" |
|
||||
| 50002 | Criteria NULL or empty | "Search ID {id} has no criteria" |
|
||||
| 50003 | Invalid JSON | "Search ID {id} has invalid JSON" |
|
||||
|
||||
**Usage**:
|
||||
```sql
|
||||
-- Throws on validation failure
|
||||
EXEC dbo.usp_ValidateSearchCriteria @SearchId = 123;
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
All extraction functions handle errors gracefully:
|
||||
|
||||
- **Scalar functions**: Return NULL for missing search, NULL criteria, or invalid JSON
|
||||
- **Table-valued functions**: Return empty result set for missing search, NULL criteria, or invalid JSON
|
||||
- **Validation procedure**: Throws errors via `THROW` statement for calling code to handle
|
||||
|
||||
## Design Pattern
|
||||
|
||||
Functions use a CTE pattern to pre-filter valid JSON before calling OPENJSON:
|
||||
|
||||
```sql
|
||||
WITH ValidSearch AS (
|
||||
SELECT Criteria
|
||||
FROM dbo.Search
|
||||
WHERE ID = @SearchId
|
||||
AND Criteria IS NOT NULL
|
||||
AND ISJSON(Criteria) = 1
|
||||
)
|
||||
SELECT ...
|
||||
FROM ValidSearch s
|
||||
CROSS APPLY OPENJSON(s.Criteria, '$.ArrayPath') ...
|
||||
```
|
||||
|
||||
This pattern prevents OPENJSON from running on invalid JSON, which would cause runtime errors.
|
||||
|
||||
## Migration Scripts
|
||||
|
||||
| Script | Contents |
|
||||
|--------|----------|
|
||||
| `045_CreateScalarExtractionFunctions.sql` | Scalar functions |
|
||||
| `046_CreateSimpleTableFunctions.sql` | Simple array TVFs |
|
||||
| `047_CreateComplexTableFunctions.sql` | Complex object TVFs |
|
||||
| `048_CreateValidateSearchCriteriaProcedure.sql` | Validation procedure |
|
||||
|
||||
## Testing
|
||||
|
||||
82 tests in `JdeScoping.Database.Tests` verify function behavior:
|
||||
|
||||
```bash
|
||||
dotnet test tests/JdeScoping.Database.Tests
|
||||
```
|
||||
|
||||
See [Testing](../Architecture/Testing.md) for details.
|
||||
Reference in New Issue
Block a user