jdescopingtool/PLANS/2026-01-06-search-criteria-extraction-design.md

Search Criteria SQL Extraction Functions - Design

Purpose

Create SQL Server functions to extract values from the Search.Criteria JSON column, eliminating the need for C# to deserialize criteria and pass Table-Valued Parameters to SQL Server. The query builder will generate SQL that extracts filter values directly from the database.

Goals

1. Create 11 SQL functions to extract scalar and table values from SearchCriteria JSON
2. Remove Table Type dependencies (7 TVP types)
3. Simplify C# query generation to pass only SearchId
4. Add comprehensive Database.Tests for the new functions
5. Update documentation and specifications

Architecture

SQL Server Version

• Target: SQL Server 2022
• JSON Functions: OPENJSON(), JSON_VALUE(), ISJSON(), TRY_CONVERT()

Design Decision: Inline TVFs + Stored Procedure Validation

Codex Review Findings:

1. THROW cannot be used in UDFs - SQL Server restriction applies to all function types
2. Multi-statement TVFs have poor performance - Table variables lack statistics, causing bad cardinality estimates
3. Inline TVFs are optimal - Can be inlined into query plans with proper optimization

Chosen Pattern:

• Inline TVFs for extraction (performance-critical, used in query builder)
• Validation stored procedure for error handling when needed
• C# validates search exists before calling query builder (defense in depth)

Function Types

Scalar Functions (3):

Function	Returns	JSON Path
dbo.fn_GetSearchMinimumDt	DATETIME2(7)	$.MinimumDt
dbo.fn_GetSearchMaximumDt	DATETIME2(7)	$.MaximumDt
dbo.fn_GetSearchExtractMisData	BIT	$.ExtractMisData

Simple Table Functions (5) - Inline TVFs:

Function	Returns	JSON Path
dbo.fn_GetSearchWorkOrders	TABLE(WorkOrderNumber BIGINT)	$.WorkOrderNumbers
dbo.fn_GetSearchItemNumbers	TABLE(ItemNumber VARCHAR(128))	$.ItemNumbers
dbo.fn_GetSearchProfitCenters	TABLE(Code VARCHAR(12))	$.ProfitCenters
dbo.fn_GetSearchWorkCenters	TABLE(Code VARCHAR(12))	$.WorkCenters
dbo.fn_GetSearchOperatorIDs	TABLE(OperatorID VARCHAR(128))	$.OperatorIDs

Complex Table Functions (2) - Inline TVFs with OPENJSON...WITH:

Function	Returns	JSON Path
dbo.fn_GetSearchComponentLots	TABLE(LotNumber VARCHAR(30), ItemNumber VARCHAR(128))	$.ComponentLotNumbers[*]
dbo.fn_GetSearchPartOperations	TABLE(ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10))	$.PartOperations[*]

Validation Stored Procedure (1):

Procedure	Purpose
dbo.usp_ValidateSearchCriteria	Validates search exists and has valid JSON, throws errors

Error Handling Strategy

Two-tier approach:

1. Inline TVFs (graceful): Return empty results for all edge cases

   • Search not found → empty
   • Criteria NULL/empty → empty
   • Invalid JSON → empty (ISJSON guard)
   • Property missing → empty
   • Bad data types → filtered out (TRY_CONVERT)

2. Validation Procedure (strict): Throws errors for invalid conditions

   • Used when explicit validation needed
   • Called before query execution if strict mode required

Error Code	Condition	Message Pattern
50001	SearchId not found	Search ID {id} not found
50002	Criteria is NULL or empty	Search ID {id} has no criteria
50003	Criteria is invalid JSON	Search ID {id} has invalid JSON

Inline TVF Pattern (Simple Arrays)

CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
RETURNS TABLE
AS
RETURN
(
    SELECT TRY_CONVERT(BIGINT, j.[value]) AS WorkOrderNumber
    FROM dbo.Search s
    CROSS APPLY OPENJSON(s.Criteria, '$.WorkOrderNumbers') j
    WHERE s.ID = @SearchId
      AND s.Criteria IS NOT NULL
      AND ISJSON(s.Criteria) = 1
      AND TRY_CONVERT(BIGINT, j.[value]) IS NOT NULL
);
GO

Inline TVF Pattern (Complex Objects with OPENJSON...WITH)

CREATE FUNCTION dbo.fn_GetSearchComponentLots(@SearchId INT)
RETURNS TABLE
AS
RETURN
(
    SELECT j.LotNumber, j.ItemNumber
    FROM dbo.Search s
    CROSS APPLY OPENJSON(s.Criteria, '$.ComponentLotNumbers')
        WITH (
            LotNumber VARCHAR(30) '$.LotNumber',
            ItemNumber VARCHAR(128) '$.ItemNumber'
        ) j
    WHERE s.ID = @SearchId
      AND s.Criteria IS NOT NULL
      AND ISJSON(s.Criteria) = 1
);
GO

Scalar Function Pattern

CREATE FUNCTION dbo.fn_GetSearchMinimumDt(@SearchId INT)
RETURNS DATETIME2(7)
AS
BEGIN
    DECLARE @Result DATETIME2(7);

    SELECT @Result = TRY_CONVERT(DATETIME2(7), JSON_VALUE(s.Criteria, '$.MinimumDt'))
    FROM dbo.Search s
    WHERE s.ID = @SearchId
      AND s.Criteria IS NOT NULL
      AND ISJSON(s.Criteria) = 1;

    RETURN @Result;
END
GO

Validation Stored Procedure

CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT)
AS
BEGIN
    SET NOCOUNT ON;
    DECLARE @Criteria VARCHAR(MAX);
    DECLARE @ErrorMsg NVARCHAR(400);

    SELECT @Criteria = Criteria
    FROM dbo.Search
    WHERE ID = @SearchId;

    IF @@ROWCOUNT = 0
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' not found');
        THROW 50001, @ErrorMsg, 1;
    END

    IF @Criteria IS NULL OR @Criteria = ''
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria');
        THROW 50002, @ErrorMsg, 1;
    END

    IF ISJSON(@Criteria) = 0
    BEGIN
        SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has invalid JSON');
        THROW 50003, @ErrorMsg, 1;
    END
END
GO

Migration Scripts

New Scripts

045_CreateScalarExtractionFunctions.sql

• fn_GetSearchMinimumDt (scalar)
• fn_GetSearchMaximumDt (scalar)
• fn_GetSearchExtractMisData (scalar)

046_CreateSimpleTableFunctions.sql (inline TVFs)

• fn_GetSearchWorkOrders
• fn_GetSearchItemNumbers
• fn_GetSearchProfitCenters
• fn_GetSearchWorkCenters
• fn_GetSearchOperatorIDs

047_CreateComplexTableFunctions.sql (inline TVFs with OPENJSON...WITH)

• fn_GetSearchComponentLots
• fn_GetSearchPartOperations

048_CreateValidateSearchCriteriaProcedure.sql

• usp_ValidateSearchCriteria (stored procedure with THROW)

Scripts to Delete

Remove obsolete Table Type scripts:

• 033_CreateWorkOrderFilterParameterType.sql
• 034_CreateItemNumberFilterParameterType.sql
• 035_CreateProfitCenterFilterParameterType.sql
• 036_CreateWorkCenterFilterParameterType.sql
• 037_CreateOperatorFilterParameterType.sql
• 038_CreateComponentLotFilterParameterType.sql
• 039_CreateItemOperationMisFilterParameterType.sql

C# Changes

Files to Delete

src/JdeScoping.DataAccess/
├── Extensions/
│   └── TableValuedParameterExtensions.cs
├── Models/FilterEntries/
│   ├── WorkOrderFilterEntry.cs
│   ├── ItemNumberFilterEntry.cs
│   ├── ProfitCenterFilterEntry.cs
│   ├── WorkCenterFilterEntry.cs
│   ├── OperatorFilterEntry.cs
│   ├── ComponentLotFilterEntry.cs
│   └── ItemOperationMisFilterEntry.cs

Files to Modify

ISearchQueryBuilder.cs - New signature:

SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);

SqlKataSearchQueryBuilder.cs - Generate SQL using functions:

-- Instead of TVP temp table population:
INSERT INTO #P_WorkOrders SELECT * FROM dbo.fn_GetSearchWorkOrders(@SearchId)

SearchModel.cs - Simplify:

• Remove all List<*FilterEntry> properties
• Remove all *FilterEnabled computed properties
• Keep: Id, UserName, Name, timestamps, results

SearchProcessor.cs - Pass searchId instead of filter lists

Test Structure

tests/JdeScoping.Database.Tests/
├── Infrastructure/
│   └── DatabaseTestBase.cs
├── Functions/
│   ├── ScalarFunctionTests.cs
│   ├── SimpleTableFunctionTests.cs
│   └── ComplexTableFunctionTests.cs
└── Procedures/
    └── ValidateSearchCriteriaProcedureTests.cs

Test Categories

Inline TVF Tests (graceful behavior):

• Valid JSON → correct extraction
• Empty array → empty result
• Missing property → empty result
• Search not found → empty result
• NULL criteria → empty result
• Invalid JSON → empty result
• Bad data types → filtered out (TRY_CONVERT)

Validation Procedure Tests (strict behavior):

• Valid search → no error, completes successfully
• Search not found → throws error 50001
• NULL criteria → throws error 50002
• Empty criteria → throws error 50002
• Invalid JSON → throws error 50003

Edge Cases:

• Large arrays (1000+ items)
• Special characters in string values
• NULL values within arrays
• Unicode characters
• Nested JSON objects

Test Infrastructure

Share with Api.IntegrationTests via TestWebApplicationFactory pattern.

Documentation Updates

OpenSpec Specifications

• Update openspec/specs/data-access/spec.md - Remove TVP references, add function requirements
• Update openspec/specs/search-processing/spec.md - Update query generation
• Update openspec/specs/database-schema/spec.md - Document extraction functions

DOCUMENTATION Folder

• Update architecture diagrams
• Add DOCUMENTATION/Database/ExtractionFunctions.md
• Update testing documentation

Implementation Order

Phase 1: SQL Functions & Procedure

1. Create 045_CreateScalarExtractionFunctions.sql (3 scalar functions)
2. Create 046_CreateSimpleTableFunctions.sql (5 inline TVFs with CTE pattern)
3. Create 047_CreateComplexTableFunctions.sql (2 inline TVFs with OPENJSON...WITH)
4. Create 048_CreateValidateSearchCriteriaProcedure.sql (validation procedure)
5. Delete obsolete Table Type scripts (033-039) - keep gaps, don't renumber

Phase 2: Database Tests

6. Set up DatabaseTestBase.cs with xUnit Collection for isolation
7. Write ScalarFunctionTests.cs
8. Write SimpleTableFunctionTests.cs
9. Write ComplexTableFunctionTests.cs
10. Write ValidateSearchCriteriaProcedureTests.cs
11. Verify all tests pass

Phase 3: C# Refactor

12. Update ISearchQueryBuilder interface
13. Update SqlKataSearchQueryBuilder
14. Update SearchProcessor
15. Simplify SearchModel
16. Delete TableValuedParameterExtensions.cs
17. Delete FilterEntries/*.cs
18. Update/delete related tests

Phase 4: Integration & Verification

19. Run full test suite
20. Fix broken tests
21. Manual end-to-end verification

Phase 5: Documentation

22. Update OpenSpec specifications
23. Update architecture documentation
24. Add Database.Tests documentation
25. Create ExtractionFunctions.md reference

Acceptance Criteria

• All 11 SQL inline TVFs/scalar functions created and working
• Validation stored procedure created with THROW for errors
• Inline TVFs return empty results for all edge cases (graceful)
• Validation procedure throws 50001/50002/50003 for invalid inputs (strict)
• Table Type scripts (033-039) removed
• C# TVP code removed (TableValuedParameterExtensions, FilterEntries)
• Query builder uses SearchId parameter only
• Database.Tests passing (functions + procedure)
• Existing integration tests passing
• Documentation updated