docs: add search criteria SQL extraction functions design
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,257 @@
|
||||
# Search Criteria SQL Extraction Functions - Design
|
||||
|
||||
## Purpose
|
||||
|
||||
Create SQL Server functions to extract values from the `Search.Criteria` JSON column, eliminating the need for C# to deserialize criteria and pass Table-Valued Parameters to SQL Server. The query builder will generate SQL that extracts filter values directly from the database.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Create 11 SQL functions to extract scalar and table values from SearchCriteria JSON
|
||||
2. Remove Table Type dependencies (7 TVP types)
|
||||
3. Simplify C# query generation to pass only SearchId
|
||||
4. Add comprehensive Database.Tests for the new functions
|
||||
5. Update documentation and specifications
|
||||
|
||||
## Architecture
|
||||
|
||||
### SQL Server Version
|
||||
- **Target:** SQL Server 2022
|
||||
- **JSON Functions:** `OPENJSON()`, `JSON_VALUE()`, `ISJSON()`
|
||||
|
||||
### Function Types
|
||||
|
||||
**Scalar Functions (3):**
|
||||
| Function | Returns | JSON Path |
|
||||
|----------|---------|-----------|
|
||||
| `dbo.fn_GetSearchMinimumDt` | `DATETIME2(7)` | `$.MinimumDt` |
|
||||
| `dbo.fn_GetSearchMaximumDt` | `DATETIME2(7)` | `$.MaximumDt` |
|
||||
| `dbo.fn_GetSearchExtractMisData` | `BIT` | `$.ExtractMisData` |
|
||||
|
||||
**Simple Table Functions (5):**
|
||||
| Function | Returns | JSON Path |
|
||||
|----------|---------|-----------|
|
||||
| `dbo.fn_GetSearchWorkOrders` | `TABLE(WorkOrderNumber BIGINT)` | `$.WorkOrderNumbers` |
|
||||
| `dbo.fn_GetSearchItemNumbers` | `TABLE(ItemNumber VARCHAR(128))` | `$.ItemNumbers` |
|
||||
| `dbo.fn_GetSearchProfitCenters` | `TABLE(Code VARCHAR(12))` | `$.ProfitCenters` |
|
||||
| `dbo.fn_GetSearchWorkCenters` | `TABLE(Code VARCHAR(12))` | `$.WorkCenters` |
|
||||
| `dbo.fn_GetSearchOperatorIDs` | `TABLE(OperatorID VARCHAR(128))` | `$.OperatorIDs` |
|
||||
|
||||
**Complex Table Functions (2):**
|
||||
| Function | Returns | JSON Path |
|
||||
|----------|---------|-----------|
|
||||
| `dbo.fn_GetSearchComponentLots` | `TABLE(LotNumber VARCHAR(30), ItemNumber VARCHAR(128))` | `$.ComponentLotNumbers[*]` |
|
||||
| `dbo.fn_GetSearchPartOperations` | `TABLE(ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10))` | `$.PartOperations[*]` |
|
||||
|
||||
### Error Handling
|
||||
|
||||
All functions validate inputs and throw errors for invalid conditions:
|
||||
|
||||
| Error Code | Condition | Message Pattern |
|
||||
|------------|-----------|-----------------|
|
||||
| 50001 | SearchId not found | `Search ID {id} not found` |
|
||||
| 50002 | Criteria is NULL or empty | `Search ID {id} has no criteria` |
|
||||
| 50003 | Criteria is invalid JSON | `Search ID {id} has invalid JSON` |
|
||||
|
||||
**Normal empty results (no error):**
|
||||
- JSON valid but property missing
|
||||
- JSON valid, property is `[]` or `null`
|
||||
|
||||
### Multi-Statement TVF Pattern
|
||||
|
||||
Table functions use multi-statement TVFs for proper error handling:
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
|
||||
RETURNS @Results TABLE (WorkOrderNumber BIGINT NOT NULL)
|
||||
AS
|
||||
BEGIN
|
||||
DECLARE @Criteria VARCHAR(MAX);
|
||||
DECLARE @ErrorMsg NVARCHAR(400);
|
||||
|
||||
-- Get criteria
|
||||
SELECT @Criteria = Criteria
|
||||
FROM dbo.Search
|
||||
WHERE ID = @SearchId;
|
||||
|
||||
-- Validate search exists
|
||||
IF @@ROWCOUNT = 0
|
||||
BEGIN
|
||||
SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' not found');
|
||||
THROW 50001, @ErrorMsg, 1;
|
||||
END
|
||||
|
||||
-- Validate criteria not null/empty
|
||||
IF @Criteria IS NULL OR @Criteria = ''
|
||||
BEGIN
|
||||
SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria');
|
||||
THROW 50002, @ErrorMsg, 1;
|
||||
END
|
||||
|
||||
-- Validate JSON
|
||||
IF ISJSON(@Criteria) = 0
|
||||
BEGIN
|
||||
SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has invalid JSON');
|
||||
THROW 50003, @ErrorMsg, 1;
|
||||
END
|
||||
|
||||
-- Extract values (returns empty if property missing/null/empty array)
|
||||
INSERT INTO @Results (WorkOrderNumber)
|
||||
SELECT CAST(value AS BIGINT)
|
||||
FROM OPENJSON(@Criteria, '$.WorkOrderNumbers');
|
||||
|
||||
RETURN;
|
||||
END
|
||||
```
|
||||
|
||||
## Migration Scripts
|
||||
|
||||
### New Scripts
|
||||
|
||||
**045_CreateScalarExtractionFunctions.sql**
|
||||
- `fn_GetSearchMinimumDt`
|
||||
- `fn_GetSearchMaximumDt`
|
||||
- `fn_GetSearchExtractMisData`
|
||||
|
||||
**046_CreateSimpleTableFunctions.sql**
|
||||
- `fn_GetSearchWorkOrders`
|
||||
- `fn_GetSearchItemNumbers`
|
||||
- `fn_GetSearchProfitCenters`
|
||||
- `fn_GetSearchWorkCenters`
|
||||
- `fn_GetSearchOperatorIDs`
|
||||
|
||||
**047_CreateComplexTableFunctions.sql**
|
||||
- `fn_GetSearchComponentLots`
|
||||
- `fn_GetSearchPartOperations`
|
||||
|
||||
### Scripts to Delete
|
||||
|
||||
Remove obsolete Table Type scripts:
|
||||
- `033_CreateWorkOrderFilterParameterType.sql`
|
||||
- `034_CreateItemNumberFilterParameterType.sql`
|
||||
- `035_CreateProfitCenterFilterParameterType.sql`
|
||||
- `036_CreateWorkCenterFilterParameterType.sql`
|
||||
- `037_CreateOperatorFilterParameterType.sql`
|
||||
- `038_CreateComponentLotFilterParameterType.sql`
|
||||
- `039_CreateItemOperationMisFilterParameterType.sql`
|
||||
|
||||
## C# Changes
|
||||
|
||||
### Files to Delete
|
||||
|
||||
```
|
||||
src/JdeScoping.DataAccess/
|
||||
├── Extensions/
|
||||
│ └── TableValuedParameterExtensions.cs
|
||||
├── Models/FilterEntries/
|
||||
│ ├── WorkOrderFilterEntry.cs
|
||||
│ ├── ItemNumberFilterEntry.cs
|
||||
│ ├── ProfitCenterFilterEntry.cs
|
||||
│ ├── WorkCenterFilterEntry.cs
|
||||
│ ├── OperatorFilterEntry.cs
|
||||
│ ├── ComponentLotFilterEntry.cs
|
||||
│ └── ItemOperationMisFilterEntry.cs
|
||||
```
|
||||
|
||||
### Files to Modify
|
||||
|
||||
**`ISearchQueryBuilder.cs`** - New signature:
|
||||
```csharp
|
||||
SearchQueryResult BuildSearchQuery(int searchId);
|
||||
SearchQueryResult BuildMisQuery(int searchId);
|
||||
SearchQueryResult BuildMisNonMatchQuery(int searchId);
|
||||
```
|
||||
|
||||
**`SqlKataSearchQueryBuilder.cs`** - Generate SQL using functions:
|
||||
```sql
|
||||
-- Instead of TVP temp table population:
|
||||
INSERT INTO #P_WorkOrders SELECT * FROM dbo.fn_GetSearchWorkOrders(@SearchId)
|
||||
```
|
||||
|
||||
**`SearchModel.cs`** - Simplify:
|
||||
- Remove all `List<*FilterEntry>` properties
|
||||
- Remove all `*FilterEnabled` computed properties
|
||||
- Keep: `Id`, `UserName`, `Name`, timestamps, results
|
||||
|
||||
**`SearchProcessor.cs`** - Pass `searchId` instead of filter lists
|
||||
|
||||
## Test Structure
|
||||
|
||||
```
|
||||
tests/JdeScoping.Database.Tests/
|
||||
├── Infrastructure/
|
||||
│ └── DatabaseTestBase.cs
|
||||
├── Functions/
|
||||
│ ├── ScalarFunctionTests.cs
|
||||
│ ├── SimpleTableFunctionTests.cs
|
||||
│ └── ComplexTableFunctionTests.cs
|
||||
```
|
||||
|
||||
### Test Categories
|
||||
|
||||
**Happy Path:** Valid JSON, correct extraction, empty arrays
|
||||
**Error Tests:** Non-existent SearchId, NULL criteria, invalid JSON
|
||||
**Edge Cases:** Large arrays, special characters, NULL values in arrays, Unicode
|
||||
|
||||
### Test Infrastructure
|
||||
|
||||
Share with `Api.IntegrationTests` via `TestWebApplicationFactory` pattern.
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
### OpenSpec Specifications
|
||||
- Update `openspec/specs/data-access/spec.md` - Remove TVP references, add function requirements
|
||||
- Update `openspec/specs/search-processing/spec.md` - Update query generation
|
||||
- Update `openspec/specs/database-schema/spec.md` - Document extraction functions
|
||||
|
||||
### DOCUMENTATION Folder
|
||||
- Update architecture diagrams
|
||||
- Add `DOCUMENTATION/Database/ExtractionFunctions.md`
|
||||
- Update testing documentation
|
||||
|
||||
## Implementation Order
|
||||
|
||||
### Phase 1: SQL Functions
|
||||
1. Create `045_CreateScalarExtractionFunctions.sql`
|
||||
2. Create `046_CreateSimpleTableFunctions.sql`
|
||||
3. Create `047_CreateComplexTableFunctions.sql`
|
||||
4. Delete obsolete Table Type scripts (033-039)
|
||||
5. Renumber remaining scripts (040-044 → 033-037)
|
||||
|
||||
### Phase 2: Database Tests
|
||||
6. Set up `DatabaseTestBase.cs`
|
||||
7. Write `ScalarFunctionTests.cs`
|
||||
8. Write `SimpleTableFunctionTests.cs`
|
||||
9. Write `ComplexTableFunctionTests.cs`
|
||||
10. Verify all tests pass
|
||||
|
||||
### Phase 3: C# Refactor
|
||||
11. Update `ISearchQueryBuilder` interface
|
||||
12. Update `SqlKataSearchQueryBuilder`
|
||||
13. Update `SearchProcessor`
|
||||
14. Simplify `SearchModel`
|
||||
15. Delete `TableValuedParameterExtensions.cs`
|
||||
16. Delete `FilterEntries/*.cs`
|
||||
17. Update/delete related tests
|
||||
|
||||
### Phase 4: Integration & Verification
|
||||
18. Run full test suite
|
||||
19. Fix broken tests
|
||||
20. Manual end-to-end verification
|
||||
|
||||
### Phase 5: Documentation
|
||||
21. Update OpenSpec specifications
|
||||
22. Update architecture documentation
|
||||
23. Add Database.Tests documentation
|
||||
24. Create ExtractionFunctions.md reference
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] All 11 SQL functions created and working
|
||||
- [ ] Error codes 50001/50002/50003 thrown for invalid inputs
|
||||
- [ ] Empty results returned for missing/null/empty properties
|
||||
- [ ] Table Type scripts removed
|
||||
- [ ] C# TVP code removed
|
||||
- [ ] Query builder uses SearchId parameter only
|
||||
- [ ] Database.Tests passing (all categories)
|
||||
- [ ] Existing integration tests passing
|
||||
- [ ] Documentation updated
|
||||
Reference in New Issue
Block a user