jdescopingtool/PLANS/2026-01-06-search-criteria-extraction-design.md

Search Criteria SQL Extraction Functions - Design

Purpose

Create SQL Server functions to extract values from the Search.Criteria JSON column, eliminating the need for C# to deserialize criteria and pass Table-Valued Parameters to SQL Server. The query builder will generate SQL that extracts filter values directly from the database.

Goals

1. Create 11 SQL functions to extract scalar and table values from SearchCriteria JSON
2. Remove Table Type dependencies (7 TVP types)
3. Simplify C# query generation to pass only SearchId
4. Add comprehensive Database.Tests for the new functions
5. Update documentation and specifications

Architecture

SQL Server Version

• Target: SQL Server 2022
• JSON Functions: OPENJSON(), JSON_VALUE(), ISJSON()

Function Types

Scalar Functions (3):

Function	Returns	JSON Path
dbo.fn_GetSearchMinimumDt	DATETIME2(7)	$.MinimumDt
dbo.fn_GetSearchMaximumDt	DATETIME2(7)	$.MaximumDt
dbo.fn_GetSearchExtractMisData	BIT	$.ExtractMisData

Simple Table Functions (5):

Function	Returns	JSON Path
dbo.fn_GetSearchWorkOrders	TABLE(WorkOrderNumber BIGINT)	$.WorkOrderNumbers
dbo.fn_GetSearchItemNumbers	TABLE(ItemNumber VARCHAR(128))	$.ItemNumbers
dbo.fn_GetSearchProfitCenters	TABLE(Code VARCHAR(12))	$.ProfitCenters
dbo.fn_GetSearchWorkCenters	TABLE(Code VARCHAR(12))	$.WorkCenters
dbo.fn_GetSearchOperatorIDs	TABLE(OperatorID VARCHAR(128))	$.OperatorIDs

Complex Table Functions (2):

Function	Returns	JSON Path
dbo.fn_GetSearchComponentLots	TABLE(LotNumber VARCHAR(30), ItemNumber VARCHAR(128))	$.ComponentLotNumbers[*]
dbo.fn_GetSearchPartOperations	TABLE(ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10))	$.PartOperations[*]

Error Handling

All functions validate inputs and throw errors for invalid conditions:

Error Code	Condition	Message Pattern
50001	SearchId not found	Search ID {id} not found
50002	Criteria is NULL or empty	Search ID {id} has no criteria
50003	Criteria is invalid JSON	Search ID {id} has invalid JSON

Normal empty results (no error):

• JSON valid but property missing
• JSON valid, property is [] or null

Multi-Statement TVF Pattern

Table functions use multi-statement TVFs for proper error handling:

CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
RETURNS @Results TABLE (WorkOrderNumber BIGINT NOT NULL)
AS
BEGIN
    DECLARE @Criteria VARCHAR(MAX);
    DECLARE @ErrorMsg NVARCHAR(400);

    -- Get criteria
    SELECT @Criteria = Criteria
    FROM dbo.Search
    WHERE ID = @SearchId;

    -- Validate search exists
    IF @@ROWCOUNT = 0
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' not found');
        THROW 50001, @ErrorMsg, 1;
    END

    -- Validate criteria not null/empty
    IF @Criteria IS NULL OR @Criteria = ''
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria');
        THROW 50002, @ErrorMsg, 1;
    END

    -- Validate JSON
    IF ISJSON(@Criteria) = 0
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has invalid JSON');
        THROW 50003, @ErrorMsg, 1;
    END

    -- Extract values (returns empty if property missing/null/empty array)
    INSERT INTO @Results (WorkOrderNumber)
    SELECT CAST(value AS BIGINT)
    FROM OPENJSON(@Criteria, '$.WorkOrderNumbers');

    RETURN;
END

Migration Scripts

New Scripts

045_CreateScalarExtractionFunctions.sql

• fn_GetSearchMinimumDt
• fn_GetSearchMaximumDt
• fn_GetSearchExtractMisData

046_CreateSimpleTableFunctions.sql

• fn_GetSearchWorkOrders
• fn_GetSearchItemNumbers
• fn_GetSearchProfitCenters
• fn_GetSearchWorkCenters
• fn_GetSearchOperatorIDs

047_CreateComplexTableFunctions.sql

• fn_GetSearchComponentLots
• fn_GetSearchPartOperations

Scripts to Delete

Remove obsolete Table Type scripts:

• 033_CreateWorkOrderFilterParameterType.sql
• 034_CreateItemNumberFilterParameterType.sql
• 035_CreateProfitCenterFilterParameterType.sql
• 036_CreateWorkCenterFilterParameterType.sql
• 037_CreateOperatorFilterParameterType.sql
• 038_CreateComponentLotFilterParameterType.sql
• 039_CreateItemOperationMisFilterParameterType.sql

C# Changes

Files to Delete

src/JdeScoping.DataAccess/
├── Extensions/
│   └── TableValuedParameterExtensions.cs
├── Models/FilterEntries/
│   ├── WorkOrderFilterEntry.cs
│   ├── ItemNumberFilterEntry.cs
│   ├── ProfitCenterFilterEntry.cs
│   ├── WorkCenterFilterEntry.cs
│   ├── OperatorFilterEntry.cs
│   ├── ComponentLotFilterEntry.cs
│   └── ItemOperationMisFilterEntry.cs

Files to Modify

ISearchQueryBuilder.cs - New signature:

SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);

SqlKataSearchQueryBuilder.cs - Generate SQL using functions:

-- Instead of TVP temp table population:
INSERT INTO #P_WorkOrders SELECT * FROM dbo.fn_GetSearchWorkOrders(@SearchId)

SearchModel.cs - Simplify:

• Remove all List<*FilterEntry> properties
• Remove all *FilterEnabled computed properties
• Keep: Id, UserName, Name, timestamps, results

SearchProcessor.cs - Pass searchId instead of filter lists

Test Structure

tests/JdeScoping.Database.Tests/
├── Infrastructure/
│   └── DatabaseTestBase.cs
├── Functions/
│   ├── ScalarFunctionTests.cs
│   ├── SimpleTableFunctionTests.cs
│   └── ComplexTableFunctionTests.cs

Test Categories

Happy Path: Valid JSON, correct extraction, empty arrays Error Tests: Non-existent SearchId, NULL criteria, invalid JSON Edge Cases: Large arrays, special characters, NULL values in arrays, Unicode

Test Infrastructure

Share with Api.IntegrationTests via TestWebApplicationFactory pattern.

Documentation Updates

OpenSpec Specifications

• Update openspec/specs/data-access/spec.md - Remove TVP references, add function requirements
• Update openspec/specs/search-processing/spec.md - Update query generation
• Update openspec/specs/database-schema/spec.md - Document extraction functions

DOCUMENTATION Folder

• Update architecture diagrams
• Add DOCUMENTATION/Database/ExtractionFunctions.md
• Update testing documentation

Implementation Order

Phase 1: SQL Functions

1. Create 045_CreateScalarExtractionFunctions.sql
2. Create 046_CreateSimpleTableFunctions.sql
3. Create 047_CreateComplexTableFunctions.sql
4. Delete obsolete Table Type scripts (033-039)
5. Renumber remaining scripts (040-044 → 033-037)

Phase 2: Database Tests

6. Set up DatabaseTestBase.cs
7. Write ScalarFunctionTests.cs
8. Write SimpleTableFunctionTests.cs
9. Write ComplexTableFunctionTests.cs
10. Verify all tests pass

Phase 3: C# Refactor

11. Update ISearchQueryBuilder interface
12. Update SqlKataSearchQueryBuilder
13. Update SearchProcessor
14. Simplify SearchModel
15. Delete TableValuedParameterExtensions.cs
16. Delete FilterEntries/*.cs
17. Update/delete related tests

Phase 4: Integration & Verification

18. Run full test suite
19. Fix broken tests
20. Manual end-to-end verification

Phase 5: Documentation

21. Update OpenSpec specifications
22. Update architecture documentation
23. Add Database.Tests documentation
24. Create ExtractionFunctions.md reference

Acceptance Criteria

• All 11 SQL functions created and working
• Error codes 50001/50002/50003 thrown for invalid inputs
• Empty results returned for missing/null/empty properties
• Table Type scripts removed
• C# TVP code removed
• Query builder uses SearchId parameter only
• Database.Tests passing (all categories)
• Existing integration tests passing
• Documentation updated