# Search Criteria SQL Extraction Functions - Design

## Purpose

Create SQL Server functions to extract values from the `Search.Criteria` JSON column, eliminating the need for C# to deserialize criteria and pass Table-Valued Parameters to SQL Server. The query builder will generate SQL that extracts filter values directly from the database.

## Goals

1. Create 11 SQL functions to extract scalar and table values from SearchCriteria JSON
2. Remove Table Type dependencies (7 TVP types)
3. Simplify C# query generation to pass only SearchId
4. Add comprehensive Database.Tests for the new functions
5. Update documentation and specifications

## Architecture

### SQL Server Version
- **Target:** SQL Server 2022
- **JSON Functions:** `OPENJSON()`, `JSON_VALUE()`, `ISJSON()`

### Function Types

**Scalar Functions (3):**
| Function | Returns | JSON Path |
|----------|---------|-----------|
| `dbo.fn_GetSearchMinimumDt` | `DATETIME2(7)` | `$.MinimumDt` |
| `dbo.fn_GetSearchMaximumDt` | `DATETIME2(7)` | `$.MaximumDt` |
| `dbo.fn_GetSearchExtractMisData` | `BIT` | `$.ExtractMisData` |

**Simple Table Functions (5):**
| Function | Returns | JSON Path |
|----------|---------|-----------|
| `dbo.fn_GetSearchWorkOrders` | `TABLE(WorkOrderNumber BIGINT)` | `$.WorkOrderNumbers` |
| `dbo.fn_GetSearchItemNumbers` | `TABLE(ItemNumber VARCHAR(128))` | `$.ItemNumbers` |
| `dbo.fn_GetSearchProfitCenters` | `TABLE(Code VARCHAR(12))` | `$.ProfitCenters` |
| `dbo.fn_GetSearchWorkCenters` | `TABLE(Code VARCHAR(12))` | `$.WorkCenters` |
| `dbo.fn_GetSearchOperatorIDs` | `TABLE(OperatorID VARCHAR(128))` | `$.OperatorIDs` |

**Complex Table Functions (2):**
| Function | Returns | JSON Path |
|----------|---------|-----------|
| `dbo.fn_GetSearchComponentLots` | `TABLE(LotNumber VARCHAR(30), ItemNumber VARCHAR(128))` | `$.ComponentLotNumbers[*]` |
| `dbo.fn_GetSearchPartOperations` | `TABLE(ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10))` | `$.PartOperations[*]` |

### Error Handling

All functions validate inputs and throw errors for invalid conditions:

| Error Code | Condition | Message Pattern |
|------------|-----------|-----------------|
| 50001 | SearchId not found | `Search ID {id} not found` |
| 50002 | Criteria is NULL or empty | `Search ID {id} has no criteria` |
| 50003 | Criteria is invalid JSON | `Search ID {id} has invalid JSON` |

**Normal empty results (no error):**
- JSON valid but property missing
- JSON valid, property is `[]` or `null`

### Multi-Statement TVF Pattern

Table functions use multi-statement TVFs for proper error handling:

```sql
CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
RETURNS @Results TABLE (WorkOrderNumber BIGINT NOT NULL)
AS
BEGIN
    DECLARE @Criteria VARCHAR(MAX);
    DECLARE @ErrorMsg NVARCHAR(400);

    -- Get criteria
    SELECT @Criteria = Criteria
    FROM dbo.Search
    WHERE ID = @SearchId;

    -- Validate search exists
    IF @@ROWCOUNT = 0
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' not found');
        THROW 50001, @ErrorMsg, 1;
    END

    -- Validate criteria not null/empty
    IF @Criteria IS NULL OR @Criteria = ''
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria');
        THROW 50002, @ErrorMsg, 1;
    END

    -- Validate JSON
    IF ISJSON(@Criteria) = 0
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has invalid JSON');
        THROW 50003, @ErrorMsg, 1;
    END

    -- Extract values (returns empty if property missing/null/empty array)
    INSERT INTO @Results (WorkOrderNumber)
    SELECT CAST(value AS BIGINT)
    FROM OPENJSON(@Criteria, '$.WorkOrderNumbers');

    RETURN;
END
```

## Migration Scripts

### New Scripts

**045_CreateScalarExtractionFunctions.sql**
- `fn_GetSearchMinimumDt`
- `fn_GetSearchMaximumDt`
- `fn_GetSearchExtractMisData`

**046_CreateSimpleTableFunctions.sql**
- `fn_GetSearchWorkOrders`
- `fn_GetSearchItemNumbers`
- `fn_GetSearchProfitCenters`
- `fn_GetSearchWorkCenters`
- `fn_GetSearchOperatorIDs`

**047_CreateComplexTableFunctions.sql**
- `fn_GetSearchComponentLots`
- `fn_GetSearchPartOperations`

### Scripts to Delete

Remove obsolete Table Type scripts:
- `033_CreateWorkOrderFilterParameterType.sql`
- `034_CreateItemNumberFilterParameterType.sql`
- `035_CreateProfitCenterFilterParameterType.sql`
- `036_CreateWorkCenterFilterParameterType.sql`
- `037_CreateOperatorFilterParameterType.sql`
- `038_CreateComponentLotFilterParameterType.sql`
- `039_CreateItemOperationMisFilterParameterType.sql`

## C# Changes

### Files to Delete

```
src/JdeScoping.DataAccess/
├── Extensions/
│   └── TableValuedParameterExtensions.cs
├── Models/FilterEntries/
│   ├── WorkOrderFilterEntry.cs
│   ├── ItemNumberFilterEntry.cs
│   ├── ProfitCenterFilterEntry.cs
│   ├── WorkCenterFilterEntry.cs
│   ├── OperatorFilterEntry.cs
│   ├── ComponentLotFilterEntry.cs
│   └── ItemOperationMisFilterEntry.cs
```

### Files to Modify

**`ISearchQueryBuilder.cs`** - New signature:
```csharp
SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);
```

**`SqlKataSearchQueryBuilder.cs`** - Generate SQL using functions:
```sql
-- Instead of TVP temp table population:
INSERT INTO #P_WorkOrders SELECT * FROM dbo.fn_GetSearchWorkOrders(@SearchId)
```

**`SearchModel.cs`** - Simplify:
- Remove all `List<*FilterEntry>` properties
- Remove all `*FilterEnabled` computed properties
- Keep: `Id`, `UserName`, `Name`, timestamps, results

**`SearchProcessor.cs`** - Pass `searchId` instead of filter lists

## Test Structure

```
tests/JdeScoping.Database.Tests/
├── Infrastructure/
│   └── DatabaseTestBase.cs
├── Functions/
│   ├── ScalarFunctionTests.cs
│   ├── SimpleTableFunctionTests.cs
│   └── ComplexTableFunctionTests.cs
```

### Test Categories

**Happy Path:** Valid JSON, correct extraction, empty arrays
**Error Tests:** Non-existent SearchId, NULL criteria, invalid JSON
**Edge Cases:** Large arrays, special characters, NULL values in arrays, Unicode

### Test Infrastructure

Share with `Api.IntegrationTests` via `TestWebApplicationFactory` pattern.

## Documentation Updates

### OpenSpec Specifications
- Update `openspec/specs/data-access/spec.md` - Remove TVP references, add function requirements
- Update `openspec/specs/search-processing/spec.md` - Update query generation
- Update `openspec/specs/database-schema/spec.md` - Document extraction functions

### DOCUMENTATION Folder
- Update architecture diagrams
- Add `DOCUMENTATION/Database/ExtractionFunctions.md`
- Update testing documentation

## Implementation Order

### Phase 1: SQL Functions
1. Create `045_CreateScalarExtractionFunctions.sql`
2. Create `046_CreateSimpleTableFunctions.sql`
3. Create `047_CreateComplexTableFunctions.sql`
4. Delete obsolete Table Type scripts (033-039)
5. Renumber remaining scripts (040-044 → 033-037)

### Phase 2: Database Tests
6. Set up `DatabaseTestBase.cs`
7. Write `ScalarFunctionTests.cs`
8. Write `SimpleTableFunctionTests.cs`
9. Write `ComplexTableFunctionTests.cs`
10. Verify all tests pass

### Phase 3: C# Refactor
11. Update `ISearchQueryBuilder` interface
12. Update `SqlKataSearchQueryBuilder`
13. Update `SearchProcessor`
14. Simplify `SearchModel`
15. Delete `TableValuedParameterExtensions.cs`
16. Delete `FilterEntries/*.cs`
17. Update/delete related tests

### Phase 4: Integration & Verification
18. Run full test suite
19. Fix broken tests
20. Manual end-to-end verification

### Phase 5: Documentation
21. Update OpenSpec specifications
22. Update architecture documentation
23. Add Database.Tests documentation
24. Create ExtractionFunctions.md reference

## Acceptance Criteria

- [ ] All 11 SQL functions created and working
- [ ] Error codes 50001/50002/50003 thrown for invalid inputs
- [ ] Empty results returned for missing/null/empty properties
- [ ] Table Type scripts removed
- [ ] C# TVP code removed
- [ ] Query builder uses SearchId parameter only
- [ ] Database.Tests passing (all categories)
- [ ] Existing integration tests passing
- [ ] Documentation updated