# Search Criteria SQL Extraction Functions - Implementation Plan > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. **Goal:** Create SQL functions to extract SearchCriteria JSON values, eliminating C# TVP code. **Architecture:** Inline TVFs for performance-critical extraction, validation stored procedure for strict error handling. C# query builder passes SearchId only. **Tech Stack:** SQL Server 2022, OPENJSON, TRY_CONVERT, xUnit, Dapper --- ## Codex Review Findings (Applied) **Critical fixes applied:** 1. **CROSS APPLY OPENJSON before WHERE** - Use CTE with pre-filtered search to prevent OPENJSON from running on invalid JSON 2. **Use OPENJSON...WITH for simple arrays** - Avoids double TRY_CONVERT, cleaner syntax **Design decisions:** 1. **Script numbering** - Keep new scripts as 045-048, do NOT renumber existing 040-044 (avoids migration drift) 2. **Test isolation** - Use xUnit Collection attribute to disable parallel execution 3. **Date format** - Document that ISO 8601 format without timezone is expected (System.Text.Json default) --- ## Task 1: Create Scalar Extraction Functions **Files:** - Create: `src/JdeScoping.Database/Scripts/045_CreateScalarExtractionFunctions.sql` **Step 1: Write the SQL script** ```sql -- Migration: 045_CreateScalarExtractionFunctions -- Scalar functions to extract simple values from Search.Criteria JSON -- fn_GetSearchMinimumDt: Extract MinimumDt datetime IF OBJECT_ID('dbo.fn_GetSearchMinimumDt', 'FN') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchMinimumDt; GO CREATE FUNCTION dbo.fn_GetSearchMinimumDt(@SearchId INT) RETURNS DATETIME2(7) AS BEGIN DECLARE @Result DATETIME2(7); SELECT @Result = TRY_CONVERT(DATETIME2(7), JSON_VALUE(s.Criteria, '$.MinimumDt')) FROM dbo.Search s WHERE s.ID = @SearchId AND s.Criteria IS NOT NULL AND ISJSON(s.Criteria) = 1; RETURN @Result; END GO -- fn_GetSearchMaximumDt: Extract MaximumDt datetime IF OBJECT_ID('dbo.fn_GetSearchMaximumDt', 'FN') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchMaximumDt; GO CREATE FUNCTION dbo.fn_GetSearchMaximumDt(@SearchId INT) RETURNS DATETIME2(7) AS BEGIN DECLARE @Result DATETIME2(7); SELECT @Result = TRY_CONVERT(DATETIME2(7), JSON_VALUE(s.Criteria, '$.MaximumDt')) FROM dbo.Search s WHERE s.ID = @SearchId AND s.Criteria IS NOT NULL AND ISJSON(s.Criteria) = 1; RETURN @Result; END GO -- fn_GetSearchExtractMisData: Extract ExtractMisData boolean IF OBJECT_ID('dbo.fn_GetSearchExtractMisData', 'FN') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchExtractMisData; GO CREATE FUNCTION dbo.fn_GetSearchExtractMisData(@SearchId INT) RETURNS BIT AS BEGIN DECLARE @Result BIT; SELECT @Result = TRY_CONVERT(BIT, JSON_VALUE(s.Criteria, '$.ExtractMisData')) FROM dbo.Search s WHERE s.ID = @SearchId AND s.Criteria IS NOT NULL AND ISJSON(s.Criteria) = 1; RETURN @Result; END GO ``` **Step 2: Verify script syntax** Run: `sqlcmd -S localhost,1434 -U sa -P -d JdeScoping -i src/JdeScoping.Database/Scripts/045_CreateScalarExtractionFunctions.sql` Expected: Commands completed successfully **Step 3: Commit** ```bash git add src/JdeScoping.Database/Scripts/045_CreateScalarExtractionFunctions.sql git commit -m "feat(db): add scalar extraction functions for SearchCriteria JSON" ``` --- ## Task 2: Create Simple Table Extraction Functions **Files:** - Create: `src/JdeScoping.Database/Scripts/046_CreateSimpleTableFunctions.sql` **Step 1: Write the SQL script** Uses CTE pattern to pre-filter valid JSON before OPENJSON (prevents runtime errors on invalid JSON). Uses OPENJSON...WITH for type conversion (single conversion, cleaner syntax). ```sql -- Migration: 046_CreateSimpleTableFunctions -- Inline TVFs to extract simple arrays from Search.Criteria JSON -- Pattern: CTE pre-filters valid JSON, OPENJSON...WITH for type-safe extraction -- fn_GetSearchWorkOrders: Extract WorkOrderNumbers array IF OBJECT_ID('dbo.fn_GetSearchWorkOrders', 'IF') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchWorkOrders; GO CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT) RETURNS TABLE AS RETURN ( WITH ValidSearch AS ( SELECT Criteria FROM dbo.Search WHERE ID = @SearchId AND Criteria IS NOT NULL AND ISJSON(Criteria) = 1 ) SELECT j.WorkOrderNumber FROM ValidSearch s CROSS APPLY OPENJSON(s.Criteria, '$.WorkOrderNumbers') WITH (WorkOrderNumber BIGINT '$') j WHERE j.WorkOrderNumber IS NOT NULL ); GO -- fn_GetSearchItemNumbers: Extract ItemNumbers array IF OBJECT_ID('dbo.fn_GetSearchItemNumbers', 'IF') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchItemNumbers; GO CREATE FUNCTION dbo.fn_GetSearchItemNumbers(@SearchId INT) RETURNS TABLE AS RETURN ( WITH ValidSearch AS ( SELECT Criteria FROM dbo.Search WHERE ID = @SearchId AND Criteria IS NOT NULL AND ISJSON(Criteria) = 1 ) SELECT j.ItemNumber FROM ValidSearch s CROSS APPLY OPENJSON(s.Criteria, '$.ItemNumbers') WITH (ItemNumber VARCHAR(128) '$') j WHERE j.ItemNumber IS NOT NULL ); GO -- fn_GetSearchProfitCenters: Extract ProfitCenters array IF OBJECT_ID('dbo.fn_GetSearchProfitCenters', 'IF') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchProfitCenters; GO CREATE FUNCTION dbo.fn_GetSearchProfitCenters(@SearchId INT) RETURNS TABLE AS RETURN ( WITH ValidSearch AS ( SELECT Criteria FROM dbo.Search WHERE ID = @SearchId AND Criteria IS NOT NULL AND ISJSON(Criteria) = 1 ) SELECT j.Code FROM ValidSearch s CROSS APPLY OPENJSON(s.Criteria, '$.ProfitCenters') WITH (Code VARCHAR(12) '$') j WHERE j.Code IS NOT NULL ); GO -- fn_GetSearchWorkCenters: Extract WorkCenters array IF OBJECT_ID('dbo.fn_GetSearchWorkCenters', 'IF') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchWorkCenters; GO CREATE FUNCTION dbo.fn_GetSearchWorkCenters(@SearchId INT) RETURNS TABLE AS RETURN ( WITH ValidSearch AS ( SELECT Criteria FROM dbo.Search WHERE ID = @SearchId AND Criteria IS NOT NULL AND ISJSON(Criteria) = 1 ) SELECT j.Code FROM ValidSearch s CROSS APPLY OPENJSON(s.Criteria, '$.WorkCenters') WITH (Code VARCHAR(12) '$') j WHERE j.Code IS NOT NULL ); GO -- fn_GetSearchOperatorIDs: Extract OperatorIDs array IF OBJECT_ID('dbo.fn_GetSearchOperatorIDs', 'IF') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchOperatorIDs; GO CREATE FUNCTION dbo.fn_GetSearchOperatorIDs(@SearchId INT) RETURNS TABLE AS RETURN ( WITH ValidSearch AS ( SELECT Criteria FROM dbo.Search WHERE ID = @SearchId AND Criteria IS NOT NULL AND ISJSON(Criteria) = 1 ) SELECT j.OperatorID FROM ValidSearch s CROSS APPLY OPENJSON(s.Criteria, '$.OperatorIDs') WITH (OperatorID VARCHAR(128) '$') j WHERE j.OperatorID IS NOT NULL ); GO ``` **Step 2: Verify script syntax** Run: `sqlcmd -S localhost,1434 -U sa -P -d JdeScoping -i src/JdeScoping.Database/Scripts/046_CreateSimpleTableFunctions.sql` Expected: Commands completed successfully **Step 3: Commit** ```bash git add src/JdeScoping.Database/Scripts/046_CreateSimpleTableFunctions.sql git commit -m "feat(db): add inline TVFs for simple array extraction from SearchCriteria" ``` --- ## Task 3: Create Complex Table Extraction Functions **Files:** - Create: `src/JdeScoping.Database/Scripts/047_CreateComplexTableFunctions.sql` **Step 1: Write the SQL script** Uses same CTE pattern to pre-filter valid JSON before OPENJSON. ```sql -- Migration: 047_CreateComplexTableFunctions -- Inline TVFs to extract complex object arrays from Search.Criteria JSON -- Pattern: CTE pre-filters valid JSON, OPENJSON...WITH for type-safe extraction -- fn_GetSearchComponentLots: Extract ComponentLotNumbers array of objects IF OBJECT_ID('dbo.fn_GetSearchComponentLots', 'IF') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchComponentLots; GO CREATE FUNCTION dbo.fn_GetSearchComponentLots(@SearchId INT) RETURNS TABLE AS RETURN ( WITH ValidSearch AS ( SELECT Criteria FROM dbo.Search WHERE ID = @SearchId AND Criteria IS NOT NULL AND ISJSON(Criteria) = 1 ) SELECT j.LotNumber, j.ItemNumber FROM ValidSearch s CROSS APPLY OPENJSON(s.Criteria, '$.ComponentLotNumbers') WITH ( LotNumber VARCHAR(30) '$.LotNumber', ItemNumber VARCHAR(128) '$.ItemNumber' ) j ); GO -- fn_GetSearchPartOperations: Extract PartOperations array of objects IF OBJECT_ID('dbo.fn_GetSearchPartOperations', 'IF') IS NOT NULL DROP FUNCTION dbo.fn_GetSearchPartOperations; GO CREATE FUNCTION dbo.fn_GetSearchPartOperations(@SearchId INT) RETURNS TABLE AS RETURN ( WITH ValidSearch AS ( SELECT Criteria FROM dbo.Search WHERE ID = @SearchId AND Criteria IS NOT NULL AND ISJSON(Criteria) = 1 ) SELECT j.ItemNumber, j.OperationNumber, j.MisNumber, j.MisRevision FROM ValidSearch s CROSS APPLY OPENJSON(s.Criteria, '$.PartOperations') WITH ( ItemNumber VARCHAR(128) '$.ItemNumber', OperationNumber VARCHAR(10) '$.OperationNumber', MisNumber VARCHAR(10) '$.MisNumber', MisRevision VARCHAR(10) '$.MisRevision' ) j ); GO ``` **Step 2: Verify script syntax** Run: `sqlcmd -S localhost,1434 -U sa -P -d JdeScoping -i src/JdeScoping.Database/Scripts/047_CreateComplexTableFunctions.sql` Expected: Commands completed successfully **Step 3: Commit** ```bash git add src/JdeScoping.Database/Scripts/047_CreateComplexTableFunctions.sql git commit -m "feat(db): add inline TVFs for complex object extraction from SearchCriteria" ``` --- ## Task 4: Create Validation Stored Procedure **Files:** - Create: `src/JdeScoping.Database/Scripts/048_CreateValidateSearchCriteriaProcedure.sql` **Step 1: Write the SQL script** ```sql -- Migration: 048_CreateValidateSearchCriteriaProcedure -- Stored procedure for strict validation with THROW errors IF OBJECT_ID('dbo.usp_ValidateSearchCriteria', 'P') IS NOT NULL DROP PROCEDURE dbo.usp_ValidateSearchCriteria; GO CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT) AS BEGIN SET NOCOUNT ON; DECLARE @Criteria VARCHAR(MAX); DECLARE @ErrorMsg NVARCHAR(400); -- Get criteria for the search SELECT @Criteria = Criteria FROM dbo.Search WHERE ID = @SearchId; -- Validate search exists IF @@ROWCOUNT = 0 BEGIN SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' not found'); THROW 50001, @ErrorMsg, 1; END -- Validate criteria not null IF @Criteria IS NULL BEGIN SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria'); THROW 50002, @ErrorMsg, 1; END -- Validate criteria not empty IF @Criteria = '' BEGIN SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has no criteria'); THROW 50002, @ErrorMsg, 1; END -- Validate JSON format IF ISJSON(@Criteria) = 0 BEGIN SET @ErrorMsg = CONCAT('Search ID ', @SearchId, ' has invalid JSON'); THROW 50003, @ErrorMsg, 1; END -- If we get here, validation passed RETURN 0; END GO ``` **Step 2: Verify script syntax** Run: `sqlcmd -S localhost,1434 -U sa -P -d JdeScoping -i src/JdeScoping.Database/Scripts/048_CreateValidateSearchCriteriaProcedure.sql` Expected: Commands completed successfully **Step 3: Commit** ```bash git add src/JdeScoping.Database/Scripts/048_CreateValidateSearchCriteriaProcedure.sql git commit -m "feat(db): add validation stored procedure with THROW errors" ``` --- ## Task 5: Delete Obsolete Table Type Scripts **Files:** - Delete: `src/JdeScoping.Database/Scripts/033_CreateWorkOrderFilterParameterType.sql` - Delete: `src/JdeScoping.Database/Scripts/034_CreateItemNumberFilterParameterType.sql` - Delete: `src/JdeScoping.Database/Scripts/035_CreateProfitCenterFilterParameterType.sql` - Delete: `src/JdeScoping.Database/Scripts/036_CreateWorkCenterFilterParameterType.sql` - Delete: `src/JdeScoping.Database/Scripts/037_CreateOperatorFilterParameterType.sql` - Delete: `src/JdeScoping.Database/Scripts/038_CreateComponentLotFilterParameterType.sql` - Delete: `src/JdeScoping.Database/Scripts/039_CreateItemOperationMisFilterParameterType.sql` **Step 1: Delete the files** ```bash rm src/JdeScoping.Database/Scripts/033_CreateWorkOrderFilterParameterType.sql rm src/JdeScoping.Database/Scripts/034_CreateItemNumberFilterParameterType.sql rm src/JdeScoping.Database/Scripts/035_CreateProfitCenterFilterParameterType.sql rm src/JdeScoping.Database/Scripts/036_CreateWorkCenterFilterParameterType.sql rm src/JdeScoping.Database/Scripts/037_CreateOperatorFilterParameterType.sql rm src/JdeScoping.Database/Scripts/038_CreateComponentLotFilterParameterType.sql rm src/JdeScoping.Database/Scripts/039_CreateItemOperationMisFilterParameterType.sql ``` **Step 2: Commit** ```bash git add -A git commit -m "chore(db): remove obsolete Table Type scripts (replaced by extraction functions)" ``` --- ## Task 6: Set Up Database Test Infrastructure **Files:** - Create: `tests/JdeScoping.Database.Tests/Infrastructure/DatabaseTestBase.cs` - Modify: `tests/JdeScoping.Database.Tests/JdeScoping.Database.Tests.csproj` - Delete: `tests/JdeScoping.Database.Tests/Placeholder.cs` **Step 1: Update project file to add dependencies** Add references to test infrastructure and Dapper: ```xml ``` **Step 2: Create DatabaseTestBase.cs** Uses xUnit Collection to disable parallel test execution (prevents cross-test contamination). ```csharp using System.Text.Json; using Dapper; using JdeScoping.Core.Models.Search; using Microsoft.Data.SqlClient; namespace JdeScoping.Database.Tests.Infrastructure; /// /// xUnit Collection definition to disable parallel execution for database tests. /// All test classes using [Collection("DatabaseTests")] run sequentially. /// [CollectionDefinition("DatabaseTests")] public class DatabaseTestCollection : ICollectionFixture { } /// /// Shared fixture for database tests. Ensures database is available. /// public class DatabaseTestFixture : IAsyncLifetime { public const string ConnectionString = "Server=localhost,1434;Database=JdeScoping;User Id=sa;Password=YourPassword123!;TrustServerCertificate=True;"; public async Task InitializeAsync() { // Verify database is accessible using var connection = new SqlConnection(ConnectionString); await connection.OpenAsync(); } public Task DisposeAsync() => Task.CompletedTask; } /// /// Base class for database function tests. /// Each test gets its own connection and cleans up created data. /// [Collection("DatabaseTests")] public abstract class DatabaseTestBase : IAsyncLifetime { protected SqlConnection Connection { get; private set; } = null!; private readonly List _createdSearchIds = []; public async Task InitializeAsync() { Connection = new SqlConnection(DatabaseTestFixture.ConnectionString); await Connection.OpenAsync(); } public async Task DisposeAsync() { // Clean up test data foreach (var id in _createdSearchIds) { await Connection.ExecuteAsync("DELETE FROM dbo.Search WHERE ID = @Id", new { Id = id }); } await Connection.DisposeAsync(); } /// /// Insert a test search with the given criteria and return the ID. /// protected async Task InsertTestSearchAsync(SearchCriteria criteria, string userName = "testuser", string name = "Test Search") { var criteriaJson = JsonSerializer.Serialize(criteria); var id = await Connection.QuerySingleAsync( @"INSERT INTO dbo.Search (UserName, Name, Status, Criteria) OUTPUT INSERTED.ID VALUES (@UserName, @Name, 0, @Criteria)", new { UserName = userName, Name = name, Criteria = criteriaJson }); _createdSearchIds.Add(id); return id; } /// /// Insert a test search with raw criteria JSON (for testing invalid JSON scenarios). /// protected async Task InsertTestSearchWithRawCriteriaAsync(string? criteriaJson, string userName = "testuser", string name = "Test Search") { var id = await Connection.QuerySingleAsync( @"INSERT INTO dbo.Search (UserName, Name, Status, Criteria) OUTPUT INSERTED.ID VALUES (@UserName, @Name, 0, @Criteria)", new { UserName = userName, Name = name, Criteria = criteriaJson }); _createdSearchIds.Add(id); return id; } } ``` **Step 3: Delete placeholder file** ```bash rm tests/JdeScoping.Database.Tests/Placeholder.cs ``` **Step 4: Commit** ```bash git add -A git commit -m "feat(tests): add DatabaseTestBase infrastructure for SQL function tests" ``` --- ## Task 8: Write Scalar Function Tests **Files:** - Create: `tests/JdeScoping.Database.Tests/Functions/ScalarFunctionTests.cs` **Step 1: Write the tests** ```csharp using Dapper; using FluentAssertions; using JdeScoping.Core.Models.Search; using JdeScoping.Database.Tests.Infrastructure; namespace JdeScoping.Database.Tests.Functions; public class ScalarFunctionTests : DatabaseTestBase { [Fact] public async Task fn_GetSearchMinimumDt_ValidSearch_ReturnsDateTime() { // Arrange var expectedDate = new DateTime(2024, 6, 15, 10, 30, 0); var criteria = new SearchCriteria { MinimumDt = expectedDate }; var searchId = await InsertTestSearchAsync(criteria); // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchMinimumDt(@SearchId)", new { SearchId = searchId }); // Assert result.Should().BeCloseTo(expectedDate, TimeSpan.FromSeconds(1)); } [Fact] public async Task fn_GetSearchMinimumDt_NullValue_ReturnsNull() { // Arrange var criteria = new SearchCriteria { MinimumDt = null }; var searchId = await InsertTestSearchAsync(criteria); // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchMinimumDt(@SearchId)", new { SearchId = searchId }); // Assert result.Should().BeNull(); } [Fact] public async Task fn_GetSearchMinimumDt_SearchNotFound_ReturnsNull() { // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchMinimumDt(@SearchId)", new { SearchId = 99999 }); // Assert result.Should().BeNull(); } [Fact] public async Task fn_GetSearchMinimumDt_InvalidJson_ReturnsNull() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync("not valid json"); // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchMinimumDt(@SearchId)", new { SearchId = searchId }); // Assert result.Should().BeNull(); } [Fact] public async Task fn_GetSearchMaximumDt_ValidSearch_ReturnsDateTime() { // Arrange var expectedDate = new DateTime(2024, 12, 31, 23, 59, 59); var criteria = new SearchCriteria { MaximumDt = expectedDate }; var searchId = await InsertTestSearchAsync(criteria); // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchMaximumDt(@SearchId)", new { SearchId = searchId }); // Assert result.Should().BeCloseTo(expectedDate, TimeSpan.FromSeconds(1)); } [Fact] public async Task fn_GetSearchExtractMisData_True_ReturnsTrue() { // Arrange var criteria = new SearchCriteria { ExtractMisData = true }; var searchId = await InsertTestSearchAsync(criteria); // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchExtractMisData(@SearchId)", new { SearchId = searchId }); // Assert result.Should().BeTrue(); } [Fact] public async Task fn_GetSearchExtractMisData_False_ReturnsFalse() { // Arrange var criteria = new SearchCriteria { ExtractMisData = false }; var searchId = await InsertTestSearchAsync(criteria); // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchExtractMisData(@SearchId)", new { SearchId = searchId }); // Assert result.Should().BeFalse(); } [Fact] public async Task fn_GetSearchExtractMisData_NullCriteria_ReturnsNull() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync(null); // Act var result = await Connection.QuerySingleOrDefaultAsync( "SELECT dbo.fn_GetSearchExtractMisData(@SearchId)", new { SearchId = searchId }); // Assert result.Should().BeNull(); } } ``` **Step 2: Run tests to verify they pass** Run: `dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ScalarFunctionTests" -v normal` Expected: All tests pass **Step 3: Commit** ```bash git add tests/JdeScoping.Database.Tests/Functions/ScalarFunctionTests.cs git commit -m "test(db): add scalar function tests for MinimumDt, MaximumDt, ExtractMisData" ``` --- ## Task 9: Write Simple Table Function Tests **Files:** - Create: `tests/JdeScoping.Database.Tests/Functions/SimpleTableFunctionTests.cs` **Step 1: Write the tests** ```csharp using Dapper; using FluentAssertions; using JdeScoping.Core.Models.Search; using JdeScoping.Database.Tests.Infrastructure; namespace JdeScoping.Database.Tests.Functions; public class SimpleTableFunctionTests : DatabaseTestBase { // WorkOrders [Fact] public async Task fn_GetSearchWorkOrders_ValidSearch_ReturnsWorkOrders() { // Arrange var criteria = new SearchCriteria { WorkOrderNumbers = [12345, 67890, 11111] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEquivalentTo([12345L, 67890L, 11111L]); } [Fact] public async Task fn_GetSearchWorkOrders_EmptyArray_ReturnsEmpty() { // Arrange var criteria = new SearchCriteria { WorkOrderNumbers = [] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEmpty(); } [Fact] public async Task fn_GetSearchWorkOrders_SearchNotFound_ReturnsEmpty() { // Act var results = await Connection.QueryAsync( "SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)", new { SearchId = 99999 }); // Assert results.Should().BeEmpty(); } // ItemNumbers [Fact] public async Task fn_GetSearchItemNumbers_ValidSearch_ReturnsItemNumbers() { // Arrange var criteria = new SearchCriteria { ItemNumbers = ["ITEM001", "ITEM002", "ITEM003"] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEquivalentTo(["ITEM001", "ITEM002", "ITEM003"]); } [Fact] public async Task fn_GetSearchItemNumbers_SpecialCharacters_ReturnsCorrectly() { // Arrange var criteria = new SearchCriteria { ItemNumbers = ["ITEM-001", "ITEM_002", "ITEM.003"] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEquivalentTo(["ITEM-001", "ITEM_002", "ITEM.003"]); } // ProfitCenters [Fact] public async Task fn_GetSearchProfitCenters_ValidSearch_ReturnsCodes() { // Arrange var criteria = new SearchCriteria { ProfitCenters = ["PC01", "PC02"] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT Code FROM dbo.fn_GetSearchProfitCenters(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEquivalentTo(["PC01", "PC02"]); } // WorkCenters [Fact] public async Task fn_GetSearchWorkCenters_ValidSearch_ReturnsCodes() { // Arrange var criteria = new SearchCriteria { WorkCenters = ["WC001", "WC002", "WC003"] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT Code FROM dbo.fn_GetSearchWorkCenters(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEquivalentTo(["WC001", "WC002", "WC003"]); } // OperatorIDs [Fact] public async Task fn_GetSearchOperatorIDs_ValidSearch_ReturnsOperatorIDs() { // Arrange var criteria = new SearchCriteria { OperatorIDs = ["OP001", "OP002"] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT OperatorID FROM dbo.fn_GetSearchOperatorIDs(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEquivalentTo(["OP001", "OP002"]); } // Edge cases [Fact] public async Task fn_GetSearchWorkOrders_InvalidJson_ReturnsEmpty() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync("not valid json"); // Act var results = await Connection.QueryAsync( "SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEmpty(); } [Fact] public async Task fn_GetSearchWorkOrders_NullCriteria_ReturnsEmpty() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync(null); // Act var results = await Connection.QueryAsync( "SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEmpty(); } [Fact] public async Task fn_GetSearchWorkOrders_LargeArray_ReturnsAll() { // Arrange var workOrders = Enumerable.Range(1, 1000).Select(i => (long)i).ToList(); var criteria = new SearchCriteria { WorkOrderNumbers = workOrders }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync( "SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)", new { SearchId = searchId }); // Assert results.Should().HaveCount(1000); results.Should().BeEquivalentTo(workOrders); } } ``` **Step 2: Run tests to verify they pass** Run: `dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~SimpleTableFunctionTests" -v normal` Expected: All tests pass **Step 3: Commit** ```bash git add tests/JdeScoping.Database.Tests/Functions/SimpleTableFunctionTests.cs git commit -m "test(db): add simple table function tests for array extraction" ``` --- ## Task 10: Write Complex Table Function Tests **Files:** - Create: `tests/JdeScoping.Database.Tests/Functions/ComplexTableFunctionTests.cs` **Step 1: Write the tests** ```csharp using Dapper; using FluentAssertions; using JdeScoping.Core.Models.Search; using JdeScoping.Core.ViewModels; using JdeScoping.Database.Tests.Infrastructure; namespace JdeScoping.Database.Tests.Functions; public class ComplexTableFunctionTests : DatabaseTestBase { // ComponentLots [Fact] public async Task fn_GetSearchComponentLots_ValidSearch_ReturnsLots() { // Arrange var criteria = new SearchCriteria { ComponentLotNumbers = [ new LotViewModel { LotNumber = "LOT001", ItemNumber = "ITEM001" }, new LotViewModel { LotNumber = "LOT002", ItemNumber = "ITEM002" } ] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync<(string LotNumber, string ItemNumber)>( "SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId)", new { SearchId = searchId }); // Assert results.Should().HaveCount(2); results.Should().Contain(x => x.LotNumber == "LOT001" && x.ItemNumber == "ITEM001"); results.Should().Contain(x => x.LotNumber == "LOT002" && x.ItemNumber == "ITEM002"); } [Fact] public async Task fn_GetSearchComponentLots_EmptyArray_ReturnsEmpty() { // Arrange var criteria = new SearchCriteria { ComponentLotNumbers = [] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync<(string LotNumber, string ItemNumber)>( "SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEmpty(); } [Fact] public async Task fn_GetSearchComponentLots_SearchNotFound_ReturnsEmpty() { // Act var results = await Connection.QueryAsync<(string LotNumber, string ItemNumber)>( "SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId)", new { SearchId = 99999 }); // Assert results.Should().BeEmpty(); } // PartOperations [Fact] public async Task fn_GetSearchPartOperations_ValidSearch_ReturnsOperations() { // Arrange var criteria = new SearchCriteria { PartOperations = [ new PartOperationViewModel { ItemNumber = "ITEM001", OperationNumber = "10", MisNumber = "MIS001", MisRevision = "A" }, new PartOperationViewModel { ItemNumber = "ITEM002", OperationNumber = "20", MisNumber = "MIS002", MisRevision = "B" } ] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync<(string ItemNumber, string OperationNumber, string MisNumber, string MisRevision)>( "SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId)", new { SearchId = searchId }); // Assert results.Should().HaveCount(2); results.Should().Contain(x => x.ItemNumber == "ITEM001" && x.OperationNumber == "10" && x.MisNumber == "MIS001" && x.MisRevision == "A"); results.Should().Contain(x => x.ItemNumber == "ITEM002" && x.OperationNumber == "20" && x.MisNumber == "MIS002" && x.MisRevision == "B"); } [Fact] public async Task fn_GetSearchPartOperations_NullOptionalFields_ReturnsNulls() { // Arrange var criteria = new SearchCriteria { PartOperations = [ new PartOperationViewModel { ItemNumber = "ITEM001", OperationNumber = "10", MisNumber = null, MisRevision = null } ] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync<(string ItemNumber, string OperationNumber, string? MisNumber, string? MisRevision)>( "SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId)", new { SearchId = searchId }); // Assert results.Should().HaveCount(1); var result = results.First(); result.ItemNumber.Should().Be("ITEM001"); result.OperationNumber.Should().Be("10"); result.MisNumber.Should().BeNull(); result.MisRevision.Should().BeNull(); } [Fact] public async Task fn_GetSearchPartOperations_InvalidJson_ReturnsEmpty() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync("not valid json"); // Act var results = await Connection.QueryAsync<(string ItemNumber, string OperationNumber, string MisNumber, string MisRevision)>( "SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId)", new { SearchId = searchId }); // Assert results.Should().BeEmpty(); } // Unicode test [Fact] public async Task fn_GetSearchComponentLots_UnicodeCharacters_ReturnsCorrectly() { // Arrange var criteria = new SearchCriteria { ComponentLotNumbers = [ new LotViewModel { LotNumber = "LOT-日本語", ItemNumber = "アイテム001" } ] }; var searchId = await InsertTestSearchAsync(criteria); // Act var results = await Connection.QueryAsync<(string LotNumber, string ItemNumber)>( "SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId)", new { SearchId = searchId }); // Assert results.Should().HaveCount(1); results.First().LotNumber.Should().Be("LOT-日本語"); results.First().ItemNumber.Should().Be("アイテム001"); } } ``` **Step 2: Run tests to verify they pass** Run: `dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ComplexTableFunctionTests" -v normal` Expected: All tests pass **Step 3: Commit** ```bash git add tests/JdeScoping.Database.Tests/Functions/ComplexTableFunctionTests.cs git commit -m "test(db): add complex table function tests for ComponentLots and PartOperations" ``` --- ## Task 11: Write Validation Procedure Tests **Files:** - Create: `tests/JdeScoping.Database.Tests/Procedures/ValidateSearchCriteriaProcedureTests.cs` **Step 1: Write the tests** ```csharp using Dapper; using FluentAssertions; using JdeScoping.Core.Models.Search; using JdeScoping.Database.Tests.Infrastructure; using Microsoft.Data.SqlClient; namespace JdeScoping.Database.Tests.Procedures; public class ValidateSearchCriteriaProcedureTests : DatabaseTestBase { [Fact] public async Task usp_ValidateSearchCriteria_ValidSearch_Succeeds() { // Arrange var criteria = new SearchCriteria { WorkOrderNumbers = [12345] }; var searchId = await InsertTestSearchAsync(criteria); // Act var act = async () => await Connection.ExecuteAsync( "EXEC dbo.usp_ValidateSearchCriteria @SearchId", new { SearchId = searchId }); // Assert await act.Should().NotThrowAsync(); } [Fact] public async Task usp_ValidateSearchCriteria_SearchNotFound_ThrowsError50001() { // Act var act = async () => await Connection.ExecuteAsync( "EXEC dbo.usp_ValidateSearchCriteria @SearchId", new { SearchId = 99999 }); // Assert var ex = await act.Should().ThrowAsync(); ex.Which.Number.Should().Be(50001); ex.Which.Message.Should().Contain("Search ID 99999 not found"); } [Fact] public async Task usp_ValidateSearchCriteria_NullCriteria_ThrowsError50002() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync(null); // Act var act = async () => await Connection.ExecuteAsync( "EXEC dbo.usp_ValidateSearchCriteria @SearchId", new { SearchId = searchId }); // Assert var ex = await act.Should().ThrowAsync(); ex.Which.Number.Should().Be(50002); ex.Which.Message.Should().Contain("has no criteria"); } [Fact] public async Task usp_ValidateSearchCriteria_EmptyCriteria_ThrowsError50002() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync(""); // Act var act = async () => await Connection.ExecuteAsync( "EXEC dbo.usp_ValidateSearchCriteria @SearchId", new { SearchId = searchId }); // Assert var ex = await act.Should().ThrowAsync(); ex.Which.Number.Should().Be(50002); ex.Which.Message.Should().Contain("has no criteria"); } [Fact] public async Task usp_ValidateSearchCriteria_InvalidJson_ThrowsError50003() { // Arrange var searchId = await InsertTestSearchWithRawCriteriaAsync("not valid json {{{"); // Act var act = async () => await Connection.ExecuteAsync( "EXEC dbo.usp_ValidateSearchCriteria @SearchId", new { SearchId = searchId }); // Assert var ex = await act.Should().ThrowAsync(); ex.Which.Number.Should().Be(50003); ex.Which.Message.Should().Contain("has invalid JSON"); } [Fact] public async Task usp_ValidateSearchCriteria_EmptyJsonObject_Succeeds() { // Arrange - empty JSON object is valid var searchId = await InsertTestSearchWithRawCriteriaAsync("{}"); // Act var act = async () => await Connection.ExecuteAsync( "EXEC dbo.usp_ValidateSearchCriteria @SearchId", new { SearchId = searchId }); // Assert await act.Should().NotThrowAsync(); } } ``` **Step 2: Run tests to verify they pass** Run: `dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ValidateSearchCriteriaProcedureTests" -v normal` Expected: All tests pass **Step 3: Commit** ```bash git add tests/JdeScoping.Database.Tests/Procedures/ValidateSearchCriteriaProcedureTests.cs git commit -m "test(db): add validation procedure tests for error handling" ``` --- ## Task 12: Run All Database Tests **Step 1: Run all tests** Run: `dotnet test tests/JdeScoping.Database.Tests -v normal` Expected: All tests pass **Step 2: If any failures, fix and re-run** --- ## Task 13: Update ISearchQueryBuilder Interface **Files:** - Modify: `src/JdeScoping.DataAccess/Interfaces/ISearchQueryBuilder.cs` **Step 1: Read current interface** **Step 2: Update to use searchId parameter** ```csharp using JdeScoping.DataAccess.Models; namespace JdeScoping.DataAccess.Interfaces; /// /// Builds SQL queries for search operations. /// public interface ISearchQueryBuilder { /// /// Builds the main search query using extraction functions. /// /// The search ID to extract criteria from. /// Query result with SQL and parameters. SearchQueryResult BuildSearchQuery(int searchId); /// /// Builds the MIS data extraction query. /// /// The search ID to extract criteria from. /// Query result with SQL and parameters. SearchQueryResult BuildMisQuery(int searchId); /// /// Builds the MIS non-match extraction query. /// /// The search ID. /// Query result with SQL and parameters. SearchQueryResult BuildMisNonMatchQuery(int searchId); } ``` **Step 3: Commit** ```bash git add src/JdeScoping.DataAccess/Interfaces/ISearchQueryBuilder.cs git commit -m "refactor(data-access): update ISearchQueryBuilder to use searchId parameter" ``` --- ## Task 14: Update SqlKataSearchQueryBuilder **Files:** - Modify: `src/JdeScoping.DataAccess/QueryBuilders/SqlKataSearchQueryBuilder.cs` **Step 1: Read current implementation** **Step 2: Update to use extraction functions** Replace TVP temp table population with function calls. The key change is in filter handler setup: Instead of: ```sql INSERT INTO #P_WorkOrders SELECT * FROM @p_WorkOrders ``` Now generates: ```sql INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId) ``` **Step 3: Remove filter handler dependencies** The query builder no longer needs filter handlers that populate from SearchModel. **Step 4: Commit** ```bash git add src/JdeScoping.DataAccess/QueryBuilders/SqlKataSearchQueryBuilder.cs git commit -m "refactor(data-access): update SqlKataSearchQueryBuilder to use extraction functions" ``` --- ## Task 15: Update SearchProcessor **Files:** - Modify: `src/JdeScoping.DataAccess/Services/SearchProcessor.cs` **Step 1: Read current implementation** **Step 2: Simplify to pass searchId only** Remove code that: - Deserializes SearchCriteria - Populates SearchModel filter lists - Creates TVP parameters Replace with: - Pass searchId to query builder - Query builder generates SQL that calls extraction functions **Step 3: Commit** ```bash git add src/JdeScoping.DataAccess/Services/SearchProcessor.cs git commit -m "refactor(data-access): simplify SearchProcessor to pass searchId only" ``` --- ## Task 16: Simplify SearchModel **Files:** - Modify: `src/JdeScoping.DataAccess/Models/SearchModel.cs` **Step 1: Remove filter properties** Remove: - `WorkOrderFilter` and `WorkOrderFilterEnabled` - `ItemNumberFilter` and `ItemNumberFilterEnabled` - `ProfitCenterFilter` and `ProfitCenterFilterEnabled` - `WorkCenterFilter` and `WorkCenterFilterEnabled` - `OperatorFilter` and `OperatorFilterEnabled` - `ComponentLotFilter` and `ComponentLotFilterEnabled` - `ItemOperationMisFilter` and `ItemOperationMisFilterEnabled` - `MinimumDt`, `MaximumDt`, `TimespanFilterEnabled` - `ExtractMisData` Keep: - `Id`, `UserName`, `Name` - `SubmitDt`, `StartDt`, `EndDt` - `Results`, `MisResults`, `MisNonMatchResults` **Step 2: Commit** ```bash git add src/JdeScoping.DataAccess/Models/SearchModel.cs git commit -m "refactor(data-access): simplify SearchModel by removing filter properties" ``` --- ## Task 17: Delete TableValuedParameterExtensions **Files:** - Delete: `src/JdeScoping.DataAccess/Extensions/TableValuedParameterExtensions.cs` **Step 1: Delete the file** ```bash rm src/JdeScoping.DataAccess/Extensions/TableValuedParameterExtensions.cs ``` **Step 2: Commit** ```bash git add -A git commit -m "chore(data-access): delete obsolete TableValuedParameterExtensions" ``` --- ## Task 18: Delete FilterEntries **Files:** - Delete: `src/JdeScoping.DataAccess/Models/FilterEntries/` directory **Step 1: List files to delete** ```bash ls src/JdeScoping.DataAccess/Models/FilterEntries/ ``` **Step 2: Delete the directory** ```bash rm -rf src/JdeScoping.DataAccess/Models/FilterEntries/ ``` **Step 3: Commit** ```bash git add -A git commit -m "chore(data-access): delete obsolete FilterEntries models" ``` --- ## Task 19: Update Related Tests **Files:** - Modify/Delete tests that use old SearchModel or TVP code **Step 1: Find affected tests** ```bash grep -r "WorkOrderFilter\|TableValuedParameter\|FilterEntry" tests/ ``` **Step 2: Update or delete affected tests** - `TableValuedParameterExtensionsTests.cs` - DELETE - `WorkOrderFilterHandlerTests.cs` - DELETE or UPDATE - `SqlKataSearchQueryBuilderTests.cs` - UPDATE to use new signature **Step 3: Run all tests to verify** ```bash dotnet test ``` **Step 4: Commit** ```bash git add -A git commit -m "test: update tests for new SearchQueryBuilder signature" ``` --- ## Task 20: Run Full Test Suite **Step 1: Build solution** ```bash dotnet build ``` **Step 2: Run all tests** ```bash dotnet test ``` **Step 3: Fix any failures** --- ## Task 21: Manual End-to-End Verification **Step 1: Start the application** **Step 2: Create a search with various criteria** **Step 3: Submit the search and verify it processes correctly** **Step 4: Check the results are correct** --- ## Task 22: Update OpenSpec Specifications **Files:** - Modify: `openspec/specs/data-access/spec.md` - Modify: `openspec/specs/search-processing/spec.md` - Modify: `openspec/specs/database-schema/spec.md` **Step 1: Update data-access spec** Remove TVP references, add extraction function requirements. **Step 2: Update search-processing spec** Update query generation requirements. **Step 3: Update database-schema spec** Document the 11 extraction functions and validation procedure. **Step 4: Run validation** ```bash openspec validate --specs ``` **Step 5: Commit** ```bash git add openspec/ git commit -m "docs(specs): update specifications for extraction functions" ``` --- ## Task 23: Update Architecture Documentation **Files:** - Modify: `DOCUMENTATION/Architecture/` (if exists) **Step 1: Update data flow diagrams** Remove TVP step, add SQL extraction step. **Step 2: Commit** ```bash git add DOCUMENTATION/ git commit -m "docs: update architecture documentation for extraction functions" ``` --- ## Task 24: Add Database Tests Documentation **Files:** - Create: `DOCUMENTATION/Testing/DatabaseTests.md` **Step 1: Document test infrastructure** - DatabaseTestBase usage - Connection string configuration - Test categories **Step 2: Commit** ```bash git add DOCUMENTATION/ git commit -m "docs: add Database.Tests documentation" ``` --- ## Task 25: Create ExtractionFunctions Reference **Files:** - Create: `DOCUMENTATION/Database/ExtractionFunctions.md` **Step 1: Document all functions** - Function signatures - Parameters - Return types - Examples - Error handling behavior **Step 2: Commit** ```bash git add DOCUMENTATION/ git commit -m "docs: add ExtractionFunctions reference documentation" ``` --- ## Task 26: Final Verification and Cleanup **Step 1: Run full test suite one more time** ```bash dotnet test ``` **Step 2: Verify all acceptance criteria are met** **Step 3: Create summary commit if needed** ```bash git log --oneline -20 ```