docs: add search criteria SQL extraction functions design
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,257 @@
|
|||||||
|
# Search Criteria SQL Extraction Functions - Design
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Create SQL Server functions to extract values from the `Search.Criteria` JSON column, eliminating the need for C# to deserialize criteria and pass Table-Valued Parameters to SQL Server. The query builder will generate SQL that extracts filter values directly from the database.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
1. Create 11 SQL functions to extract scalar and table values from SearchCriteria JSON
|
||||||
|
2. Remove Table Type dependencies (7 TVP types)
|
||||||
|
3. Simplify C# query generation to pass only SearchId
|
||||||
|
4. Add comprehensive Database.Tests for the new functions
|
||||||
|
5. Update documentation and specifications
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### SQL Server Version
|
||||||
|
- **Target:** SQL Server 2022
|
||||||
|
- **JSON Functions:** `OPENJSON()`, `JSON_VALUE()`, `ISJSON()`
|
||||||
|
|
||||||
|
### Function Types
|
||||||
|
|
||||||
|
**Scalar Functions (3):**
|
||||||
|
| Function | Returns | JSON Path |
|
||||||
|
|----------|---------|-----------|
|
||||||
|
| `dbo.fn_GetSearchMinimumDt` | `DATETIME2(7)` | `$.MinimumDt` |
|
||||||
|
| `dbo.fn_GetSearchMaximumDt` | `DATETIME2(7)` | `$.MaximumDt` |
|
||||||
|
| `dbo.fn_GetSearchExtractMisData` | `BIT` | `$.ExtractMisData` |
|
||||||
|
|
||||||
|
**Simple Table Functions (5):**
|
||||||
|
| Function | Returns | JSON Path |
|
||||||
|
|----------|---------|-----------|
|
||||||
|
| `dbo.fn_GetSearchWorkOrders` | `TABLE(WorkOrderNumber BIGINT)` | `$.WorkOrderNumbers` |
|
||||||
|
| `dbo.fn_GetSearchItemNumbers` | `TABLE(ItemNumber VARCHAR(128))` | `$.ItemNumbers` |
|
||||||
|
| `dbo.fn_GetSearchProfitCenters` | `TABLE(Code VARCHAR(12))` | `$.ProfitCenters` |
|
||||||
|
| `dbo.fn_GetSearchWorkCenters` | `TABLE(Code VARCHAR(12))` | `$.WorkCenters` |
|
||||||
|
| `dbo.fn_GetSearchOperatorIDs` | `TABLE(OperatorID VARCHAR(128))` | `$.OperatorIDs` |
|
||||||
|
|
||||||
|
**Complex Table Functions (2):**
|
||||||
|
| Function | Returns | JSON Path |
|
||||||
|
|----------|---------|-----------|
|
||||||
|
| `dbo.fn_GetSearchComponentLots` | `TABLE(LotNumber VARCHAR(30), ItemNumber VARCHAR(128))` | `$.ComponentLotNumbers[*]` |
|
||||||
|
| `dbo.fn_GetSearchPartOperations` | `TABLE(ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10))` | `$.PartOperations[*]` |
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
|
||||||
|
All functions validate inputs and throw errors for invalid conditions:
|
||||||
|
|
||||||
|
| Error Code | Condition | Message Pattern |
|
||||||
|
|------------|-----------|-----------------|
|
||||||
|
| 50001 | SearchId not found | `Search ID {id} not found` |
|
||||||
|
| 50002 | Criteria is NULL or empty | `Search ID {id} has no criteria` |
|
||||||
|
| 50003 | Criteria is invalid JSON | `Search ID {id} has invalid JSON` |
|
||||||
|
|
||||||
|
**Normal empty results (no error):**
|
||||||
|
- JSON valid but property missing
|
||||||
|
- JSON valid, property is `[]` or `null`
|
||||||
|
|
||||||
|
### Multi-Statement TVF Pattern
|
||||||
|
|
||||||
|
Table functions use multi-statement TVFs for proper error handling:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
|
||||||
|
RETURNS @Results TABLE (WorkOrderNumber BIGINT NOT NULL)
|
||||||
|
AS
|
||||||
|
BEGIN
|
||||||
|
DECLARE @Criteria VARCHAR(MAX);
|
||||||
|
DECLARE @ErrorMsg NVARCHAR(400);
|
||||||
|
|
||||||
|
-- Get criteria
|
||||||
|
SELECT @Criteria = Criteria
|
||||||
|
FROM dbo.Search
|
||||||
|
WHERE ID = @SearchId;
|
||||||
|
|
||||||
|
-- Validate search exists
|
||||||
|
IF @@ROWCOUNT = 0
|
||||||
|
BEGIN
|
||||||
|
SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' not found');
|
||||||
|
THROW 50001, @ErrorMsg, 1;
|
||||||
|
END
|
||||||
|
|
||||||
|
-- Validate criteria not null/empty
|
||||||
|
IF @Criteria IS NULL OR @Criteria = ''
|
||||||
|
BEGIN
|
||||||
|
SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria');
|
||||||
|
THROW 50002, @ErrorMsg, 1;
|
||||||
|
END
|
||||||
|
|
||||||
|
-- Validate JSON
|
||||||
|
IF ISJSON(@Criteria) = 0
|
||||||
|
BEGIN
|
||||||
|
SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has invalid JSON');
|
||||||
|
THROW 50003, @ErrorMsg, 1;
|
||||||
|
END
|
||||||
|
|
||||||
|
-- Extract values (returns empty if property missing/null/empty array)
|
||||||
|
INSERT INTO @Results (WorkOrderNumber)
|
||||||
|
SELECT CAST(value AS BIGINT)
|
||||||
|
FROM OPENJSON(@Criteria, '$.WorkOrderNumbers');
|
||||||
|
|
||||||
|
RETURN;
|
||||||
|
END
|
||||||
|
```
|
||||||
|
|
||||||
|
## Migration Scripts
|
||||||
|
|
||||||
|
### New Scripts
|
||||||
|
|
||||||
|
**045_CreateScalarExtractionFunctions.sql**
|
||||||
|
- `fn_GetSearchMinimumDt`
|
||||||
|
- `fn_GetSearchMaximumDt`
|
||||||
|
- `fn_GetSearchExtractMisData`
|
||||||
|
|
||||||
|
**046_CreateSimpleTableFunctions.sql**
|
||||||
|
- `fn_GetSearchWorkOrders`
|
||||||
|
- `fn_GetSearchItemNumbers`
|
||||||
|
- `fn_GetSearchProfitCenters`
|
||||||
|
- `fn_GetSearchWorkCenters`
|
||||||
|
- `fn_GetSearchOperatorIDs`
|
||||||
|
|
||||||
|
**047_CreateComplexTableFunctions.sql**
|
||||||
|
- `fn_GetSearchComponentLots`
|
||||||
|
- `fn_GetSearchPartOperations`
|
||||||
|
|
||||||
|
### Scripts to Delete
|
||||||
|
|
||||||
|
Remove obsolete Table Type scripts:
|
||||||
|
- `033_CreateWorkOrderFilterParameterType.sql`
|
||||||
|
- `034_CreateItemNumberFilterParameterType.sql`
|
||||||
|
- `035_CreateProfitCenterFilterParameterType.sql`
|
||||||
|
- `036_CreateWorkCenterFilterParameterType.sql`
|
||||||
|
- `037_CreateOperatorFilterParameterType.sql`
|
||||||
|
- `038_CreateComponentLotFilterParameterType.sql`
|
||||||
|
- `039_CreateItemOperationMisFilterParameterType.sql`
|
||||||
|
|
||||||
|
## C# Changes
|
||||||
|
|
||||||
|
### Files to Delete
|
||||||
|
|
||||||
|
```
|
||||||
|
src/JdeScoping.DataAccess/
|
||||||
|
├── Extensions/
|
||||||
|
│ └── TableValuedParameterExtensions.cs
|
||||||
|
├── Models/FilterEntries/
|
||||||
|
│ ├── WorkOrderFilterEntry.cs
|
||||||
|
│ ├── ItemNumberFilterEntry.cs
|
||||||
|
│ ├── ProfitCenterFilterEntry.cs
|
||||||
|
│ ├── WorkCenterFilterEntry.cs
|
||||||
|
│ ├── OperatorFilterEntry.cs
|
||||||
|
│ ├── ComponentLotFilterEntry.cs
|
||||||
|
│ └── ItemOperationMisFilterEntry.cs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Files to Modify
|
||||||
|
|
||||||
|
**`ISearchQueryBuilder.cs`** - New signature:
|
||||||
|
```csharp
|
||||||
|
SearchQueryResult BuildSearchQuery(int searchId);
|
||||||
|
SearchQueryResult BuildMisQuery(int searchId);
|
||||||
|
SearchQueryResult BuildMisNonMatchQuery(int searchId);
|
||||||
|
```
|
||||||
|
|
||||||
|
**`SqlKataSearchQueryBuilder.cs`** - Generate SQL using functions:
|
||||||
|
```sql
|
||||||
|
-- Instead of TVP temp table population:
|
||||||
|
INSERT INTO #P_WorkOrders SELECT * FROM dbo.fn_GetSearchWorkOrders(@SearchId)
|
||||||
|
```
|
||||||
|
|
||||||
|
**`SearchModel.cs`** - Simplify:
|
||||||
|
- Remove all `List<*FilterEntry>` properties
|
||||||
|
- Remove all `*FilterEnabled` computed properties
|
||||||
|
- Keep: `Id`, `UserName`, `Name`, timestamps, results
|
||||||
|
|
||||||
|
**`SearchProcessor.cs`** - Pass `searchId` instead of filter lists
|
||||||
|
|
||||||
|
## Test Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
tests/JdeScoping.Database.Tests/
|
||||||
|
├── Infrastructure/
|
||||||
|
│ └── DatabaseTestBase.cs
|
||||||
|
├── Functions/
|
||||||
|
│ ├── ScalarFunctionTests.cs
|
||||||
|
│ ├── SimpleTableFunctionTests.cs
|
||||||
|
│ └── ComplexTableFunctionTests.cs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test Categories
|
||||||
|
|
||||||
|
**Happy Path:** Valid JSON, correct extraction, empty arrays
|
||||||
|
**Error Tests:** Non-existent SearchId, NULL criteria, invalid JSON
|
||||||
|
**Edge Cases:** Large arrays, special characters, NULL values in arrays, Unicode
|
||||||
|
|
||||||
|
### Test Infrastructure
|
||||||
|
|
||||||
|
Share with `Api.IntegrationTests` via `TestWebApplicationFactory` pattern.
|
||||||
|
|
||||||
|
## Documentation Updates
|
||||||
|
|
||||||
|
### OpenSpec Specifications
|
||||||
|
- Update `openspec/specs/data-access/spec.md` - Remove TVP references, add function requirements
|
||||||
|
- Update `openspec/specs/search-processing/spec.md` - Update query generation
|
||||||
|
- Update `openspec/specs/database-schema/spec.md` - Document extraction functions
|
||||||
|
|
||||||
|
### DOCUMENTATION Folder
|
||||||
|
- Update architecture diagrams
|
||||||
|
- Add `DOCUMENTATION/Database/ExtractionFunctions.md`
|
||||||
|
- Update testing documentation
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
### Phase 1: SQL Functions
|
||||||
|
1. Create `045_CreateScalarExtractionFunctions.sql`
|
||||||
|
2. Create `046_CreateSimpleTableFunctions.sql`
|
||||||
|
3. Create `047_CreateComplexTableFunctions.sql`
|
||||||
|
4. Delete obsolete Table Type scripts (033-039)
|
||||||
|
5. Renumber remaining scripts (040-044 → 033-037)
|
||||||
|
|
||||||
|
### Phase 2: Database Tests
|
||||||
|
6. Set up `DatabaseTestBase.cs`
|
||||||
|
7. Write `ScalarFunctionTests.cs`
|
||||||
|
8. Write `SimpleTableFunctionTests.cs`
|
||||||
|
9. Write `ComplexTableFunctionTests.cs`
|
||||||
|
10. Verify all tests pass
|
||||||
|
|
||||||
|
### Phase 3: C# Refactor
|
||||||
|
11. Update `ISearchQueryBuilder` interface
|
||||||
|
12. Update `SqlKataSearchQueryBuilder`
|
||||||
|
13. Update `SearchProcessor`
|
||||||
|
14. Simplify `SearchModel`
|
||||||
|
15. Delete `TableValuedParameterExtensions.cs`
|
||||||
|
16. Delete `FilterEntries/*.cs`
|
||||||
|
17. Update/delete related tests
|
||||||
|
|
||||||
|
### Phase 4: Integration & Verification
|
||||||
|
18. Run full test suite
|
||||||
|
19. Fix broken tests
|
||||||
|
20. Manual end-to-end verification
|
||||||
|
|
||||||
|
### Phase 5: Documentation
|
||||||
|
21. Update OpenSpec specifications
|
||||||
|
22. Update architecture documentation
|
||||||
|
23. Add Database.Tests documentation
|
||||||
|
24. Create ExtractionFunctions.md reference
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [ ] All 11 SQL functions created and working
|
||||||
|
- [ ] Error codes 50001/50002/50003 thrown for invalid inputs
|
||||||
|
- [ ] Empty results returned for missing/null/empty properties
|
||||||
|
- [ ] Table Type scripts removed
|
||||||
|
- [ ] C# TVP code removed
|
||||||
|
- [ ] Query builder uses SearchId parameter only
|
||||||
|
- [ ] Database.Tests passing (all categories)
|
||||||
|
- [ ] Existing integration tests passing
|
||||||
|
- [ ] Documentation updated
|
||||||
Reference in New Issue
Block a user