diff --git a/PLANS/2026-01-06-search-criteria-extraction-design.md b/PLANS/2026-01-06-search-criteria-extraction-design.md new file mode 100644 index 0000000..95b2ca8 --- /dev/null +++ b/PLANS/2026-01-06-search-criteria-extraction-design.md @@ -0,0 +1,257 @@ +# Search Criteria SQL Extraction Functions - Design + +## Purpose + +Create SQL Server functions to extract values from the `Search.Criteria` JSON column, eliminating the need for C# to deserialize criteria and pass Table-Valued Parameters to SQL Server. The query builder will generate SQL that extracts filter values directly from the database. + +## Goals + +1. Create 11 SQL functions to extract scalar and table values from SearchCriteria JSON +2. Remove Table Type dependencies (7 TVP types) +3. Simplify C# query generation to pass only SearchId +4. Add comprehensive Database.Tests for the new functions +5. Update documentation and specifications + +## Architecture + +### SQL Server Version +- **Target:** SQL Server 2022 +- **JSON Functions:** `OPENJSON()`, `JSON_VALUE()`, `ISJSON()` + +### Function Types + +**Scalar Functions (3):** +| Function | Returns | JSON Path | +|----------|---------|-----------| +| `dbo.fn_GetSearchMinimumDt` | `DATETIME2(7)` | `$.MinimumDt` | +| `dbo.fn_GetSearchMaximumDt` | `DATETIME2(7)` | `$.MaximumDt` | +| `dbo.fn_GetSearchExtractMisData` | `BIT` | `$.ExtractMisData` | + +**Simple Table Functions (5):** +| Function | Returns | JSON Path | +|----------|---------|-----------| +| `dbo.fn_GetSearchWorkOrders` | `TABLE(WorkOrderNumber BIGINT)` | `$.WorkOrderNumbers` | +| `dbo.fn_GetSearchItemNumbers` | `TABLE(ItemNumber VARCHAR(128))` | `$.ItemNumbers` | +| `dbo.fn_GetSearchProfitCenters` | `TABLE(Code VARCHAR(12))` | `$.ProfitCenters` | +| `dbo.fn_GetSearchWorkCenters` | `TABLE(Code VARCHAR(12))` | `$.WorkCenters` | +| `dbo.fn_GetSearchOperatorIDs` | `TABLE(OperatorID VARCHAR(128))` | `$.OperatorIDs` | + +**Complex Table Functions (2):** +| Function | Returns | JSON Path | +|----------|---------|-----------| +| `dbo.fn_GetSearchComponentLots` | `TABLE(LotNumber VARCHAR(30), ItemNumber VARCHAR(128))` | `$.ComponentLotNumbers[*]` | +| `dbo.fn_GetSearchPartOperations` | `TABLE(ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10))` | `$.PartOperations[*]` | + +### Error Handling + +All functions validate inputs and throw errors for invalid conditions: + +| Error Code | Condition | Message Pattern | +|------------|-----------|-----------------| +| 50001 | SearchId not found | `Search ID {id} not found` | +| 50002 | Criteria is NULL or empty | `Search ID {id} has no criteria` | +| 50003 | Criteria is invalid JSON | `Search ID {id} has invalid JSON` | + +**Normal empty results (no error):** +- JSON valid but property missing +- JSON valid, property is `[]` or `null` + +### Multi-Statement TVF Pattern + +Table functions use multi-statement TVFs for proper error handling: + +```sql +CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT) +RETURNS @Results TABLE (WorkOrderNumber BIGINT NOT NULL) +AS +BEGIN + DECLARE @Criteria VARCHAR(MAX); + DECLARE @ErrorMsg NVARCHAR(400); + + -- Get criteria + SELECT @Criteria = Criteria + FROM dbo.Search + WHERE ID = @SearchId; + + -- Validate search exists + IF @@ROWCOUNT = 0 + BEGIN + SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' not found'); + THROW 50001, @ErrorMsg, 1; + END + + -- Validate criteria not null/empty + IF @Criteria IS NULL OR @Criteria = '' + BEGIN + SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria'); + THROW 50002, @ErrorMsg, 1; + END + + -- Validate JSON + IF ISJSON(@Criteria) = 0 + BEGIN + SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has invalid JSON'); + THROW 50003, @ErrorMsg, 1; + END + + -- Extract values (returns empty if property missing/null/empty array) + INSERT INTO @Results (WorkOrderNumber) + SELECT CAST(value AS BIGINT) + FROM OPENJSON(@Criteria, '$.WorkOrderNumbers'); + + RETURN; +END +``` + +## Migration Scripts + +### New Scripts + +**045_CreateScalarExtractionFunctions.sql** +- `fn_GetSearchMinimumDt` +- `fn_GetSearchMaximumDt` +- `fn_GetSearchExtractMisData` + +**046_CreateSimpleTableFunctions.sql** +- `fn_GetSearchWorkOrders` +- `fn_GetSearchItemNumbers` +- `fn_GetSearchProfitCenters` +- `fn_GetSearchWorkCenters` +- `fn_GetSearchOperatorIDs` + +**047_CreateComplexTableFunctions.sql** +- `fn_GetSearchComponentLots` +- `fn_GetSearchPartOperations` + +### Scripts to Delete + +Remove obsolete Table Type scripts: +- `033_CreateWorkOrderFilterParameterType.sql` +- `034_CreateItemNumberFilterParameterType.sql` +- `035_CreateProfitCenterFilterParameterType.sql` +- `036_CreateWorkCenterFilterParameterType.sql` +- `037_CreateOperatorFilterParameterType.sql` +- `038_CreateComponentLotFilterParameterType.sql` +- `039_CreateItemOperationMisFilterParameterType.sql` + +## C# Changes + +### Files to Delete + +``` +src/JdeScoping.DataAccess/ +├── Extensions/ +│ └── TableValuedParameterExtensions.cs +├── Models/FilterEntries/ +│ ├── WorkOrderFilterEntry.cs +│ ├── ItemNumberFilterEntry.cs +│ ├── ProfitCenterFilterEntry.cs +│ ├── WorkCenterFilterEntry.cs +│ ├── OperatorFilterEntry.cs +│ ├── ComponentLotFilterEntry.cs +│ └── ItemOperationMisFilterEntry.cs +``` + +### Files to Modify + +**`ISearchQueryBuilder.cs`** - New signature: +```csharp +SearchQueryResult BuildSearchQuery(int searchId); +SearchQueryResult BuildMisQuery(int searchId); +SearchQueryResult BuildMisNonMatchQuery(int searchId); +``` + +**`SqlKataSearchQueryBuilder.cs`** - Generate SQL using functions: +```sql +-- Instead of TVP temp table population: +INSERT INTO #P_WorkOrders SELECT * FROM dbo.fn_GetSearchWorkOrders(@SearchId) +``` + +**`SearchModel.cs`** - Simplify: +- Remove all `List<*FilterEntry>` properties +- Remove all `*FilterEnabled` computed properties +- Keep: `Id`, `UserName`, `Name`, timestamps, results + +**`SearchProcessor.cs`** - Pass `searchId` instead of filter lists + +## Test Structure + +``` +tests/JdeScoping.Database.Tests/ +├── Infrastructure/ +│ └── DatabaseTestBase.cs +├── Functions/ +│ ├── ScalarFunctionTests.cs +│ ├── SimpleTableFunctionTests.cs +│ └── ComplexTableFunctionTests.cs +``` + +### Test Categories + +**Happy Path:** Valid JSON, correct extraction, empty arrays +**Error Tests:** Non-existent SearchId, NULL criteria, invalid JSON +**Edge Cases:** Large arrays, special characters, NULL values in arrays, Unicode + +### Test Infrastructure + +Share with `Api.IntegrationTests` via `TestWebApplicationFactory` pattern. + +## Documentation Updates + +### OpenSpec Specifications +- Update `openspec/specs/data-access/spec.md` - Remove TVP references, add function requirements +- Update `openspec/specs/search-processing/spec.md` - Update query generation +- Update `openspec/specs/database-schema/spec.md` - Document extraction functions + +### DOCUMENTATION Folder +- Update architecture diagrams +- Add `DOCUMENTATION/Database/ExtractionFunctions.md` +- Update testing documentation + +## Implementation Order + +### Phase 1: SQL Functions +1. Create `045_CreateScalarExtractionFunctions.sql` +2. Create `046_CreateSimpleTableFunctions.sql` +3. Create `047_CreateComplexTableFunctions.sql` +4. Delete obsolete Table Type scripts (033-039) +5. Renumber remaining scripts (040-044 → 033-037) + +### Phase 2: Database Tests +6. Set up `DatabaseTestBase.cs` +7. Write `ScalarFunctionTests.cs` +8. Write `SimpleTableFunctionTests.cs` +9. Write `ComplexTableFunctionTests.cs` +10. Verify all tests pass + +### Phase 3: C# Refactor +11. Update `ISearchQueryBuilder` interface +12. Update `SqlKataSearchQueryBuilder` +13. Update `SearchProcessor` +14. Simplify `SearchModel` +15. Delete `TableValuedParameterExtensions.cs` +16. Delete `FilterEntries/*.cs` +17. Update/delete related tests + +### Phase 4: Integration & Verification +18. Run full test suite +19. Fix broken tests +20. Manual end-to-end verification + +### Phase 5: Documentation +21. Update OpenSpec specifications +22. Update architecture documentation +23. Add Database.Tests documentation +24. Create ExtractionFunctions.md reference + +## Acceptance Criteria + +- [ ] All 11 SQL functions created and working +- [ ] Error codes 50001/50002/50003 thrown for invalid inputs +- [ ] Empty results returned for missing/null/empty properties +- [ ] Table Type scripts removed +- [ ] C# TVP code removed +- [ ] Query builder uses SearchId parameter only +- [ ] Database.Tests passing (all categories) +- [ ] Existing integration tests passing +- [ ] Documentation updated