docs: update documentation for extraction functions migration

- Add ExtractionFunctions.md reference document
- Update database-schema spec with 11 extraction functions
- Update data-access spec to document extraction function approach
- Update search-processing spec with new query builder interface
- Add Database.Tests to Testing.md architecture doc
- Update DataFlow.md with extraction function flow

DOCUMENTATION/Architecture/DataFlow.md

@@ -19,6 +19,8 @@ The search flow mirrors the legacy pattern, modernized for ASP.NET Core:
 
 4. SearchProcessorService (BackgroundService) polls
    └─> Finds queued searches
+   └─> Passes SearchId to query builder
+   └─> SQL extraction functions read criteria from Search.Criteria JSON
    └─> Executes query against local cache
    └─> Generates Excel via ClosedXML
    └─> Stores result in Search.Results (VARBINARY)

DOCUMENTATION/Architecture/Database.md

@@ -121,7 +121,7 @@ The scoping tool cache database includes these primary tables:
 
 | Table | Purpose |
 |-------|---------|
-| `Search` | User search requests, status, and results (Excel as VARBINARY) |
+| `Search` | User search requests, status, criteria (JSON), and results (Excel as VARBINARY) |
 | `DataUpdate` | Tracks last sync timestamp per data type |
 | `WorkOrder_Curr` | Current work orders from JDE |
 | `WorkOrder_Hist` | Historical work orders from JDE |
@@ -134,6 +134,38 @@ The scoping tool cache database includes these primary tables:
 | `ProfitCenter` | Profit center reference data |
 | `SchemaVersions` | DbUp tracking table (auto-created) |
 
+## Search Criteria Extraction Functions
+
+The database includes SQL functions that extract filter criteria from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, with all filter extraction happening in SQL Server.
+
+### Scalar Functions (3)
+
+| Function | Returns | Extracts |
+|----------|---------|----------|
+| `fn_GetSearchMinimumDt` | DATETIME2 | `$.MinimumDt` |
+| `fn_GetSearchMaximumDt` | DATETIME2 | `$.MaximumDt` |
+| `fn_GetSearchExtractMisData` | BIT | `$.ExtractMisData` |
+
+### Table-Valued Functions (8)
+
+| Function | Returns | Extracts |
+|----------|---------|----------|
+| `fn_GetSearchWorkOrders` | WorkOrderNumber | `$.WorkOrderNumbers` array |
+| `fn_GetSearchItemNumbers` | ItemNumber | `$.ItemNumbers` array |
+| `fn_GetSearchProfitCenters` | Code | `$.ProfitCenters` array |
+| `fn_GetSearchWorkCenters` | Code | `$.WorkCenters` array |
+| `fn_GetSearchOperatorIDs` | OperatorID | `$.OperatorIDs` array |
+| `fn_GetSearchComponentLots` | LotNumber, ItemNumber | `$.ComponentLotNumbers` array |
+| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision | `$.PartOperations` array |
+
+### Validation Procedure
+
+| Procedure | Purpose |
+|-----------|---------|
+| `usp_ValidateSearchCriteria` | Validates search exists and has valid JSON criteria |
+
+Error codes: 50001 (not found), 50002 (no criteria), 50003 (invalid JSON)
+
 ## Example Migration Scripts
 
 ### 001_CreateSearchTable.sql

DOCUMENTATION/Architecture/Testing.md

@@ -5,22 +5,34 @@ The test project uses xUnit for the framework, Shouldly for assertions, and NSub
 ## Project Structure
 
 ```
-JdeScoping.Tests/
+tests/
-├── Unit/
+├── JdeScoping.Tests/
-│   ├── Services/
+│   ├── Unit/
-│   │   ├── SearchServiceTests.cs
+│   │   ├── Services/
-│   │   ├── ExcelExportServiceTests.cs
+│   │   │   ├── SearchServiceTests.cs
-│   │   └── DataSyncOrchestratorTests.cs
+│   │   │   ├── ExcelExportServiceTests.cs
-│   ├── Repositories/
+│   │   │   └── DataSyncOrchestratorTests.cs
-│   │   └── SearchRepositoryTests.cs
+│   │   ├── Repositories/
-│   └── Models/
+│   │   │   └── SearchRepositoryTests.cs
-│       └── SearchCriteriaTests.cs
+│   │   └── Models/
-└── Integration/
+│   │       └── SearchCriteriaTests.cs
-    ├── ApiTests/
+│   └── Integration/
-    │   ├── SearchControllerTests.cs
+│       ├── ApiTests/
-    │   └── LookupControllerTests.cs
+│       │   ├── SearchControllerTests.cs
-    └── RepositoryTests/
+│       │   └── LookupControllerTests.cs
-        └── JdeRepositoryTests.cs
+│       └── RepositoryTests/
+│           └── JdeRepositoryTests.cs
+├── JdeScoping.Api.IntegrationTests/
+│   └── ... (API integration tests)
+└── JdeScoping.Database.Tests/
+    ├── Infrastructure/
+    │   └── DatabaseTestBase.cs
+    ├── Functions/
+    │   ├── ScalarFunctionTests.cs
+    │   ├── SimpleTableFunctionTests.cs
+    │   └── ComplexTableFunctionTests.cs
+    └── Procedures/
+        └── ValidateSearchCriteriaProcedureTests.cs
 ```
 
 ## Unit Tests
@@ -175,7 +187,64 @@ public class SearchRepositoryIntegrationTests : IDisposable
 }
 ```
 
+## Database Tests
+
+The `JdeScoping.Database.Tests` project tests SQL functions and stored procedures against a live SQL Server instance.
+
+### Test Infrastructure
+
+Tests inherit from `DatabaseTestBase` which provides:
+- Connection management via `DatabaseTestFixture`
+- Automatic test data cleanup
+- Helper methods for inserting test searches
+
+```csharp
+[Collection("DatabaseTests")]
+public class ScalarFunctionTests : DatabaseTestBase
+{
+    [Fact]
+    public async Task fn_GetSearchMinimumDt_ValidSearch_ReturnsDateTime()
+    {
+        // Arrange
+        var criteria = new SearchCriteria { MinimumDt = new DateTime(2024, 6, 15) };
+        var searchId = await InsertTestSearchAsync(criteria);
+
+        // Act
+        var result = await Connection.QuerySingleOrDefaultAsync<DateTime?>(
+            "SELECT dbo.fn_GetSearchMinimumDt(@SearchId)",
+            new { SearchId = searchId });
+
+        // Assert
+        result.Should().BeCloseTo(criteria.MinimumDt.Value, TimeSpan.FromSeconds(1));
+    }
+}
+```
+
+### Test Categories
+
+| Category | Tests | Description |
+|----------|-------|-------------|
+| Scalar Functions | 15 | Test `fn_GetSearchMinimumDt`, `fn_GetSearchMaximumDt`, `fn_GetSearchExtractMisData` |
+| Simple Table Functions | 38 | Test array extraction for work orders, items, profit centers, work centers, operators |
+| Complex Table Functions | 23 | Test object extraction for component lots and part operations |
+| Validation Procedure | 6 | Test `usp_ValidateSearchCriteria` error handling |
+
+**Total: 82 database tests**
+
+### Running Database Tests
+
+```bash
+# Run all database tests
+dotnet test tests/JdeScoping.Database.Tests
+
+# Run specific test class
+dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ScalarFunctionTests"
+```
+
+**Prerequisites**: SQL Server must be running on localhost:1434 with the ScopingTool database.
+
 ## Related Documentation
 
 - [Core Project](./CoreProject.md)
 - [Dependencies](./Dependencies.md)
+- [Database](./Database.md)

DOCUMENTATION/Database/ExtractionFunctions.md

@@ -0,0 +1,237 @@
+# Search Criteria Extraction Functions
+
+SQL functions that extract filter values from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, simplifying the C# to SQL interface.
+
+## Overview
+
+| Type | Count | Purpose |
+|------|-------|---------|
+| Scalar Functions | 3 | Extract single values (dates, booleans) |
+| Table-Valued Functions | 8 | Extract arrays and object collections |
+| Validation Procedure | 1 | Pre-flight validation with error codes |
+
+## Scalar Functions
+
+### fn_GetSearchMinimumDt
+
+Extracts the minimum date filter value.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchMinimumDt(@SearchId INT)
+RETURNS DATETIME2(7)
+```
+
+**JSON Path**: `$.MinimumDt`
+
+**Returns**: DATETIME2 value or NULL if not found/invalid
+
+### fn_GetSearchMaximumDt
+
+Extracts the maximum date filter value.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchMaximumDt(@SearchId INT)
+RETURNS DATETIME2(7)
+```
+
+**JSON Path**: `$.MaximumDt`
+
+**Returns**: DATETIME2 value or NULL if not found/invalid
+
+### fn_GetSearchExtractMisData
+
+Extracts the MIS data extraction flag.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchExtractMisData(@SearchId INT)
+RETURNS BIT
+```
+
+**JSON Path**: `$.ExtractMisData`
+
+**Returns**: 1 (true), 0 (false), or NULL if not found/invalid
+
+## Table-Valued Functions (Simple Arrays)
+
+### fn_GetSearchWorkOrders
+
+Extracts work order number filter values.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
+RETURNS TABLE
+AS RETURN (
+    SELECT j.WorkOrderNumber
+    FROM ... OPENJSON(..., '$.WorkOrderNumbers') WITH (WorkOrderNumber BIGINT '$') j
+)
+```
+
+**JSON Path**: `$.WorkOrderNumbers`
+
+**Returns**: Table with `WorkOrderNumber BIGINT` column
+
+### fn_GetSearchItemNumbers
+
+Extracts item number filter values.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchItemNumbers(@SearchId INT)
+RETURNS TABLE
+```
+
+**JSON Path**: `$.ItemNumbers`
+
+**Returns**: Table with `ItemNumber VARCHAR(128)` column
+
+### fn_GetSearchProfitCenters
+
+Extracts profit center filter values.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchProfitCenters(@SearchId INT)
+RETURNS TABLE
+```
+
+**JSON Path**: `$.ProfitCenters`
+
+**Returns**: Table with `Code VARCHAR(12)` column
+
+### fn_GetSearchWorkCenters
+
+Extracts work center filter values.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchWorkCenters(@SearchId INT)
+RETURNS TABLE
+```
+
+**JSON Path**: `$.WorkCenters`
+
+**Returns**: Table with `Code VARCHAR(12)` column
+
+### fn_GetSearchOperatorIDs
+
+Extracts operator ID filter values.
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchOperatorIDs(@SearchId INT)
+RETURNS TABLE
+```
+
+**JSON Path**: `$.OperatorIDs`
+
+**Returns**: Table with `OperatorID VARCHAR(128)` column
+
+## Table-Valued Functions (Complex Objects)
+
+### fn_GetSearchComponentLots
+
+Extracts component lot filter values (lot/item pairs).
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchComponentLots(@SearchId INT)
+RETURNS TABLE
+AS RETURN (
+    SELECT j.LotNumber, j.ItemNumber
+    FROM ... OPENJSON(..., '$.ComponentLotNumbers') WITH (
+        LotNumber VARCHAR(30) '$.LotNumber',
+        ItemNumber VARCHAR(128) '$.ItemNumber'
+    ) j
+)
+```
+
+**JSON Path**: `$.ComponentLotNumbers`
+
+**Returns**: Table with `LotNumber VARCHAR(30)`, `ItemNumber VARCHAR(128)` columns
+
+### fn_GetSearchPartOperations
+
+Extracts part operation filter values (item/operation/MIS combinations).
+
+```sql
+CREATE FUNCTION dbo.fn_GetSearchPartOperations(@SearchId INT)
+RETURNS TABLE
+AS RETURN (
+    SELECT j.ItemNumber, j.OperationNumber, j.MisNumber, j.MisRevision
+    FROM ... OPENJSON(..., '$.PartOperations') WITH (
+        ItemNumber VARCHAR(128) '$.ItemNumber',
+        OperationNumber VARCHAR(10) '$.OperationNumber',
+        MisNumber VARCHAR(10) '$.MisNumber',
+        MisRevision VARCHAR(10) '$.MisRevision'
+    ) j
+)
+```
+
+**JSON Path**: `$.PartOperations`
+
+**Returns**: Table with `ItemNumber`, `OperationNumber`, `MisNumber`, `MisRevision` columns
+
+## Validation Procedure
+
+### usp_ValidateSearchCriteria
+
+Validates that a search exists and has valid JSON criteria.
+
+```sql
+CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT)
+```
+
+**Error Codes**:
+
+| Code | Condition | Message |
+|------|-----------|---------|
+| 50001 | Search not found | "Search ID {id} not found" |
+| 50002 | Criteria NULL or empty | "Search ID {id} has no criteria" |
+| 50003 | Invalid JSON | "Search ID {id} has invalid JSON" |
+
+**Usage**:
+```sql
+-- Throws on validation failure
+EXEC dbo.usp_ValidateSearchCriteria @SearchId = 123;
+```
+
+## Error Handling
+
+All extraction functions handle errors gracefully:
+
+- **Scalar functions**: Return NULL for missing search, NULL criteria, or invalid JSON
+- **Table-valued functions**: Return empty result set for missing search, NULL criteria, or invalid JSON
+- **Validation procedure**: Throws errors via `THROW` statement for calling code to handle
+
+## Design Pattern
+
+Functions use a CTE pattern to pre-filter valid JSON before calling OPENJSON:
+
+```sql
+WITH ValidSearch AS (
+    SELECT Criteria
+    FROM dbo.Search
+    WHERE ID = @SearchId
+      AND Criteria IS NOT NULL
+      AND ISJSON(Criteria) = 1
+)
+SELECT ...
+FROM ValidSearch s
+CROSS APPLY OPENJSON(s.Criteria, '$.ArrayPath') ...
+```
+
+This pattern prevents OPENJSON from running on invalid JSON, which would cause runtime errors.
+
+## Migration Scripts
+
+| Script | Contents |
+|--------|----------|
+| `045_CreateScalarExtractionFunctions.sql` | Scalar functions |
+| `046_CreateSimpleTableFunctions.sql` | Simple array TVFs |
+| `047_CreateComplexTableFunctions.sql` | Complex object TVFs |
+| `048_CreateValidateSearchCriteriaProcedure.sql` | Validation procedure |
+
+## Testing
+
+82 tests in `JdeScoping.Database.Tests` verify function behavior:
+
+```bash
+dotnet test tests/JdeScoping.Database.Tests
+```
+
+See [Testing](../Architecture/Testing.md) for details.

Tools/CacheConverter/Program.cs

@@ -1,5 +1,4 @@
 using System.Data;
-using System.Text;
 using System.Text.Json;
 using System.Text.RegularExpressions;
 using ProtoBuf.Data;
@@ -102,55 +101,32 @@ foreach (var jsonFile in jsonFiles)
         }
 
         // Stream JSON and write to protobuf in batches
-        using var inputFs = new FileStream(jsonFile, FileMode.Open, FileAccess.Read, FileShare.Read, 256 * 1024, FileOptions.SequentialScan);
+        await using var inputFs = new FileStream(jsonFile, FileMode.Open, FileAccess.Read, FileShare.Read, 256 * 1024, FileOptions.SequentialScan | FileOptions.Asynchronous);
-        using var decompressStream = new DecompressionStream(inputFs);
+        await using var decompressStream = new DecompressionStream(inputFs);
-        using var outputFs = new FileStream(outputFile, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024);
+        await using var outputFs = new FileStream(outputFile, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024, FileOptions.Asynchronous);
-        using var compressStream = new CompressionStream(outputFs, level: 3);
+        await using var compressStream = new CompressionStream(outputFs, level: 3);
 
         int rowCount = 0;
         int batchCount = 0;
 
-        // Stream JSON records one at a time
+        // True streaming: DeserializeAsyncEnumerable streams each array element without loading entire JSON
-        var buffer = new byte[4096];
+        var jsonOptions = new JsonSerializerOptions { PropertyNameCaseInsensitive = true };
-        using var memoryStream = new MemoryStream();
+        await foreach (var element in JsonSerializer.DeserializeAsyncEnumerable<JsonElement>(
-
+            decompressStream,
-        int bytesRead;
+            jsonOptions))
-        while ((bytesRead = decompressStream.Read(buffer, 0, buffer.Length)) > 0)
         {
-            memoryStream.Write(buffer, 0, bytesRead);
+            var row = dataTable.NewRow();
-        }
+            ReadJsonElement(element, row, dataTable);
+            dataTable.Rows.Add(row);
+            rowCount++;
 
-        memoryStream.Position = 0;
+            // Write batch when we hit the batch size
-        var jsonReader = new Utf8JsonReader(memoryStream.ToArray(), new JsonReaderOptions { AllowTrailingCommas = true });
+            if (dataTable.Rows.Count >= BatchSize)
-
-        // Skip to start of array
-        while (jsonReader.Read())
-        {
-            if (jsonReader.TokenType == JsonTokenType.StartArray)
-                break;
-        }
-
-        // Read each object in the array
-        while (jsonReader.Read())
-        {
-            if (jsonReader.TokenType == JsonTokenType.EndArray)
-                break;
-
-            if (jsonReader.TokenType == JsonTokenType.StartObject)
             {
-                var row = dataTable.NewRow();
+                using var reader = dataTable.CreateDataReader();
-                ReadJsonObject(ref jsonReader, row, dataTable);
+                DataSerializer.Serialize(compressStream, reader);
-                dataTable.Rows.Add(row);
+                dataTable.Clear();
-                rowCount++;
+                batchCount++;
-
-                // Write batch when we hit the batch size
-                if (dataTable.Rows.Count >= BatchSize)
-                {
-                    using var reader = dataTable.CreateDataReader();
-                    DataSerializer.Serialize(compressStream, reader);
-                    dataTable.Clear();
-                    batchCount++;
-                }
             }
         }
 
@@ -162,7 +138,7 @@ foreach (var jsonFile in jsonFiles)
             batchCount++;
         }
 
-        compressStream.Flush();
+        await compressStream.FlushAsync();
 
         var newSize = new FileInfo(outputFile).Length;
         totalNewSize += newSize;
@@ -243,68 +219,58 @@ static Type MapSqlTypeToNet(string sqlType) => sqlType.ToUpperInvariant() switch
 };
 
 /// <summary>
-/// Read a JSON object into a DataRow using streaming reader.
+/// Read a JsonElement (object) into a DataRow.
 /// </summary>
-static void ReadJsonObject(ref Utf8JsonReader reader, DataRow row, DataTable table)
+static void ReadJsonElement(JsonElement element, DataRow row, DataTable table)
 {
-    while (reader.Read())
+    foreach (var property in element.EnumerateObject())
     {
-        if (reader.TokenType == JsonTokenType.EndObject)
+        // Find matching column (case-insensitive)
-            break;
+        DataColumn? column = null;
-
+        foreach (DataColumn col in table.Columns)
-        if (reader.TokenType == JsonTokenType.PropertyName)
         {
-            var propertyName = reader.GetString()!;
+            if (col.ColumnName.Equals(property.Name, StringComparison.OrdinalIgnoreCase))
-            reader.Read(); // Move to value
-
-            // Find matching column (case-insensitive)
-            DataColumn? column = null;
-            foreach (DataColumn col in table.Columns)
             {
-                if (col.ColumnName.Equals(propertyName, StringComparison.OrdinalIgnoreCase))
+                column = col;
-                {
+                break;
-                    column = col;
-                    break;
-                }
             }
-
-            if (column == null)
-            {
-                // Skip unknown property
-                SkipJsonValue(ref reader);
-                continue;
-            }
-
-            row[column] = ReadJsonValue(ref reader, column.DataType);
         }
+
+        if (column == null)
+        {
+            // Skip unknown property
+            continue;
+        }
+
+        row[column] = ReadJsonElementValue(property.Value, column.DataType);
     }
 }
 
 /// <summary>
-/// Read a JSON value and convert to the target .NET type.
+/// Read a JSON value from JsonElement and convert to the target .NET type.
 /// </summary>
-static object ReadJsonValue(ref Utf8JsonReader reader, Type targetType)
+static object ReadJsonElementValue(JsonElement element, Type targetType)
 {
-    if (reader.TokenType == JsonTokenType.Null)
+    if (element.ValueKind == JsonValueKind.Null || element.ValueKind == JsonValueKind.Undefined)
         return DBNull.Value;
 
     if (targetType == typeof(string))
     {
-        return reader.TokenType switch
+        return element.ValueKind switch
         {
-            JsonTokenType.String => reader.GetString() ?? (object)DBNull.Value,
+            JsonValueKind.String => element.GetString() ?? (object)DBNull.Value,
-            JsonTokenType.Number => reader.GetDecimal().ToString(),
+            JsonValueKind.Number => element.GetDecimal().ToString(),
-            JsonTokenType.True => "true",
+            JsonValueKind.True => "true",
-            JsonTokenType.False => "false",
+            JsonValueKind.False => "false",
             _ => DBNull.Value
         };
     }
 
     if (targetType == typeof(DateTime))
     {
-        if (reader.TokenType == JsonTokenType.String)
+        if (element.ValueKind == JsonValueKind.String)
         {
-            var str = reader.GetString();
+            var str = element.GetString();
             if (str != null && DateTime.TryParse(str, out var dt))
                 return dt;
         }
@@ -313,71 +279,45 @@ static object ReadJsonValue(ref Utf8JsonReader reader, Type targetType)
 
     if (targetType == typeof(long))
     {
-        return reader.TokenType switch
+        return element.ValueKind switch
         {
-            JsonTokenType.Number => reader.GetInt64(),
+            JsonValueKind.Number => element.GetInt64(),
-            JsonTokenType.String when long.TryParse(reader.GetString(), out var val) => val,
+            JsonValueKind.String when long.TryParse(element.GetString(), out var val) => val,
             _ => DBNull.Value
         };
     }
 
     if (targetType == typeof(int))
     {
-        return reader.TokenType switch
+        return element.ValueKind switch
         {
-            JsonTokenType.Number => reader.GetInt32(),
+            JsonValueKind.Number => element.GetInt32(),
-            JsonTokenType.String when int.TryParse(reader.GetString(), out var val) => val,
+            JsonValueKind.String when int.TryParse(element.GetString(), out var val) => val,
             _ => DBNull.Value
         };
     }
 
     if (targetType == typeof(decimal))
     {
-        return reader.TokenType switch
+        return element.ValueKind switch
         {
-            JsonTokenType.Number => reader.GetDecimal(),
+            JsonValueKind.Number => element.GetDecimal(),
-            JsonTokenType.String when decimal.TryParse(reader.GetString(), out var val) => val,
+            JsonValueKind.String when decimal.TryParse(element.GetString(), out var val) => val,
             _ => DBNull.Value
         };
     }
 
     if (targetType == typeof(bool))
     {
-        return reader.TokenType switch
+        return element.ValueKind switch
         {
-            JsonTokenType.True => true,
+            JsonValueKind.True => true,
-            JsonTokenType.False => false,
+            JsonValueKind.False => false,
-            JsonTokenType.Number => reader.GetInt32() != 0,
+            JsonValueKind.Number => element.GetInt32() != 0,
-            JsonTokenType.String => reader.GetString()?.Equals("true", StringComparison.OrdinalIgnoreCase) ?? false,
+            JsonValueKind.String => element.GetString()?.Equals("true", StringComparison.OrdinalIgnoreCase) ?? false,
             _ => DBNull.Value
         };
     }
 
     return DBNull.Value;
 }
-
-/// <summary>
-/// Skip a JSON value (used for unknown properties).
-/// </summary>
-static void SkipJsonValue(ref Utf8JsonReader reader)
-{
-    if (reader.TokenType == JsonTokenType.StartObject)
-    {
-        int depth = 1;
-        while (depth > 0 && reader.Read())
-        {
-            if (reader.TokenType == JsonTokenType.StartObject) depth++;
-            else if (reader.TokenType == JsonTokenType.EndObject) depth--;
-        }
-    }
-    else if (reader.TokenType == JsonTokenType.StartArray)
-    {
-        int depth = 1;
-        while (depth > 0 && reader.Read())
-        {
-            if (reader.TokenType == JsonTokenType.StartArray) depth++;
-            else if (reader.TokenType == JsonTokenType.EndArray) depth--;
-        }
-    }
-    // Simple values are already consumed by the Read() call
-}

openspec/specs/data-access/spec.md

@@ -467,40 +467,65 @@ catch (OracleException ex)
 
 ---
 
-### Requirement: Table-valued parameter support
+### Requirement: Search criteria extraction via SQL functions
 
-The system SHALL use DataTable for SQL Server table-valued parameters in lookup methods.
+The system SHALL use SQL extraction functions to retrieve filter criteria directly from the `Search.Criteria` JSON column.
 
 #### Implementation Pattern
 
+Search query building now uses SearchId to invoke extraction functions rather than passing filter values from C#:
+
 ```csharp
-public async Task<List<Item>> LookupItemsAsync(List<string> itemNumbers, CancellationToken ct = default)
+public SearchQueryResult BuildSearchQuery(int searchId)
 {
-    await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
+    // Query builder generates SQL that calls extraction functions
+    // Example generated SQL fragment:
+    // INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
+    // INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
 
-    var table = new DataTable();
+    return new SearchQueryResult(sql, new { SearchId = searchId });
-    table.Columns.Add("ItemNumber", typeof(string));
-    foreach (var itemNumber in itemNumbers)
-    {
-        table.Rows.Add(itemNumber);
-    }
-
-    var parameters = new DynamicParameters();
-    parameters.Add("@itemNumbers", table.AsTableValuedParameter("dbo.ItemNumberFilterParameter"));
-
-    return (await connection.QueryAsync<Item>(
-        LotFinderQueries.SQL_LOOKUP_ITEMS,
-        parameters,
-        commandTimeout: _options.Value.DefaultTimeoutSeconds))
-        .AsList();
 }
 ```
 
 #### Business Rules
 
-- TVPs SHALL use `AsTableValuedParameter` extension with correct type name
+- Query builder SHALL accept only `searchId` parameter (not full criteria object)
+- SQL queries SHALL call extraction functions to populate temporary filter tables
+- Extraction functions handle JSON parsing and validation in SQL Server
+- Invalid JSON or missing criteria results in empty filter sets (no errors thrown)
+
+#### Available Extraction Functions
+
+| Function | Returns |
+|----------|---------|
+| `fn_GetSearchMinimumDt` | DATETIME2 scalar |
+| `fn_GetSearchMaximumDt` | DATETIME2 scalar |
+| `fn_GetSearchExtractMisData` | BIT scalar |
+| `fn_GetSearchWorkOrders` | WorkOrderNumber table |
+| `fn_GetSearchItemNumbers` | ItemNumber table |
+| `fn_GetSearchProfitCenters` | Code table |
+| `fn_GetSearchWorkCenters` | Code table |
+| `fn_GetSearchOperatorIDs` | OperatorID table |
+| `fn_GetSearchComponentLots` | LotNumber, ItemNumber table |
+| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision table |
+
+#### Scenario: Build search query with extraction functions
+
+- **WHEN** `BuildSearchQuery(123)` is called for a search with work order filter
+- **THEN** generated SQL includes `SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(123)`
+- **AND** only the `@SearchId` parameter is passed to the query
+
+---
+
+### Requirement: Table-valued parameter support for lookups
+
+The system SHALL use DataTable for SQL Server table-valued parameters in reference data lookup methods.
+
+#### Business Rules
+
+- TVPs are used for batch lookups (LookupItemsAsync, LookupWorkordersAsync, etc.)
+- TVPs are NOT used for search query execution (replaced by extraction functions)
 - DataTable column names SHALL match TVP type column names
-- TVPs enable efficient batch lookups with single database round-trip
 
 #### Scenario: Batch lookup with TVP
 

openspec/specs/database-schema/spec.md

@@ -1079,6 +1079,90 @@ var results = await context.WorkOrders
 
 ## Functions
 
+### Requirement: Search Criteria Extraction Functions
+
+The system SHALL provide SQL functions to extract search criteria values from the `Search.Criteria` JSON column.
+
+#### Scalar Functions
+
+| Function | Return Type | Description |
+|----------|-------------|-------------|
+| `fn_GetSearchMinimumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MinimumDt` value |
+| `fn_GetSearchMaximumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MaximumDt` value |
+| `fn_GetSearchExtractMisData(@SearchId INT)` | BIT | Extracts `$.ExtractMisData` value |
+
+#### Table-Valued Functions (Simple Arrays)
+
+| Function | Return Columns | Description |
+|----------|----------------|-------------|
+| `fn_GetSearchWorkOrders(@SearchId INT)` | WorkOrderNumber BIGINT | Extracts `$.WorkOrderNumbers` array |
+| `fn_GetSearchItemNumbers(@SearchId INT)` | ItemNumber VARCHAR(128) | Extracts `$.ItemNumbers` array |
+| `fn_GetSearchProfitCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.ProfitCenters` array |
+| `fn_GetSearchWorkCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.WorkCenters` array |
+| `fn_GetSearchOperatorIDs(@SearchId INT)` | OperatorID VARCHAR(128) | Extracts `$.OperatorIDs` array |
+
+#### Table-Valued Functions (Complex Objects)
+
+| Function | Return Columns | Description |
+|----------|----------------|-------------|
+| `fn_GetSearchComponentLots(@SearchId INT)` | LotNumber VARCHAR(30), ItemNumber VARCHAR(128) | Extracts `$.ComponentLotNumbers` array |
+| `fn_GetSearchPartOperations(@SearchId INT)` | ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10) | Extracts `$.PartOperations` array |
+
+#### Business Rules
+
+- Scalar functions return NULL if search not found, criteria is NULL, or JSON is invalid
+- Table-valued functions return empty result set if search not found, criteria is NULL, or JSON is invalid
+- TVFs use CTE pattern to pre-filter valid JSON before OPENJSON (prevents runtime errors)
+- TVFs use `OPENJSON...WITH` syntax for type-safe extraction
+
+#### Scenario: Extract work order filter values
+
+- **WHEN** Search ID 123 has Criteria containing `{"WorkOrderNumbers":[12345,67890]}`
+- **THEN** `SELECT * FROM dbo.fn_GetSearchWorkOrders(123)` returns two rows with WorkOrderNumber values 12345 and 67890
+
+#### Scenario: Handle invalid JSON gracefully
+
+- **WHEN** Search ID 456 has Criteria containing invalid JSON text
+- **THEN** extraction functions return NULL (scalar) or empty result set (TVF) without throwing errors
+
+---
+
+### Requirement: Validate Search Criteria Procedure
+
+The system SHALL provide a stored procedure for strict validation of search criteria with error reporting.
+
+#### Procedure Signature
+
+```sql
+CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT)
+```
+
+#### Error Codes
+
+| Error Code | Condition | Message Pattern |
+|------------|-----------|-----------------|
+| 50001 | Search ID not found | "Search ID {id} not found" |
+| 50002 | Criteria is NULL or empty | "Search ID {id} has no criteria" |
+| 50003 | Criteria is not valid JSON | "Search ID {id} has invalid JSON" |
+
+#### Business Rules
+
+- Procedure throws errors using `THROW` statement for calling code to handle
+- Procedure returns 0 (success) when validation passes
+- Used for pre-flight validation before query execution
+
+#### Scenario: Validate valid search
+
+- **WHEN** `usp_ValidateSearchCriteria` is called for a search with valid JSON criteria
+- **THEN** procedure returns 0 (success) without throwing
+
+#### Scenario: Validate missing search
+
+- **WHEN** `usp_ValidateSearchCriteria` is called for non-existent search ID 99999
+- **THEN** procedure throws error 50001 with message "Search ID 99999 not found"
+
+---
+
 ### Requirement: MatchMIS function
 
 The system SHALL provide a table-valued function to match work order steps to MIS data.

openspec/specs/search-processing/spec.md

@@ -86,7 +86,7 @@ The legacy system uses a T4 text template (`QueryTemplate.tt`) for SQL generatio
 
 **NuGet Package:** `SqlKata` and `SqlKata.Execution`
 
-**Primary Query Building with SqlKata:**
+**Primary Query Building with SqlKata and Extraction Functions:**
 
 ```csharp
 using SqlKata;
@@ -94,7 +94,9 @@ using SqlKata.Compilers;
 
 public interface ISearchQueryBuilder
 {
-    SearchQueryResult BuildSearchQuery(SearchModel model);
+    SearchQueryResult BuildSearchQuery(int searchId);
+    SearchQueryResult BuildMisQuery(int searchId);
+    SearchQueryResult BuildMisNonMatchQuery(int searchId);
 }
 
 public record SearchQueryResult(string Sql, IDictionary<string, object> Parameters);
@@ -103,127 +105,50 @@ public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder
 {
     private readonly SqlServerCompiler _compiler = new();
 
-    public SearchQueryResult BuildSearchQuery(SearchModel model)
+    public SearchQueryResult BuildSearchQuery(int searchId)
     {
+        // Query uses SQL extraction functions to get criteria values
+        // No need to pass filter collections from C# - just the SearchId
+
+        // Generated SQL populates temp tables from extraction functions:
+        // INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
+        // INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
+        // etc.
+
         var query = new Query("WorkOrder_Curr as wo")
             .Select("wo.*", "wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode");
 
-        // Conditional joins based on active filters
+        // Conditional joins based on temp table population (determined at runtime)
-        if (model.OperatorFilterEnabled)
+        // Query builder generates SQL that checks IF EXISTS on temp tables
-        {
-            query.Join("WorkOrderTime_Curr as wot", "wot.WorkOrderNumber", "wo.WorkOrderNumber");
-        }
 
-        if (model.ProfitCenterFilterEnabled || model.WorkCenterFilterEnabled)
-        {
-            query.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber");
-        }
-
-        if (model.ComponentLotFilterEnabled)
-        {
-            query.Join("WorkOrderComponent_Curr as woc", "woc.WorkOrderNumber", "wo.WorkOrderNumber");
-        }
-
-        // Conditional WHERE clauses
-        if (model.TimespanFilterEnabled)
-        {
-            query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
-        }
-
-        if (model.WorkOrderFilterEnabled)
-        {
-            query.WhereIn("wo.WorkOrderNumber", model.WorkOrderFilter.Select(w => w.WorkOrderNumber));
-        }
-
-        if (model.ItemNumberFilterEnabled)
-        {
-            query.WhereIn("wo.ItemNumber", model.ItemNumberFilter.Select(i => i.ItemNumber));
-        }
-
-        if (model.OperatorFilterEnabled)
-        {
-            query.WhereIn("wot.OperatorAN", model.OperatorFilter.Select(o => o.AddressNumber));
-        }
-
-        // Compile to SQL + parameters
         var compiled = _compiler.Compile(query);
-        return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
+        return new SearchQueryResult(compiled.Sql, new Dictionary<string, object> { ["SearchId"] = searchId });
     }
 }
 ```
 
-**Complex Query Composition with SqlKata:**
+**Extraction Function Integration:**
 
-```csharp
+The query builder generates SQL that populates temporary filter tables from extraction functions:
-public sealed class ComposableSearchQueryBuilder
-{
-    private readonly SqlServerCompiler _compiler = new();
 
-    public SearchQueryResult BuildFullSearchQuery(SearchModel model)
+```sql
-    {
+-- Generated SQL fragment example:
-        // Build composable query parts
+-- Populate temp tables from extraction functions
-        var baseQuery = BuildBaseWorkOrderQuery(model);
+INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId);
-        var filterQuery = ApplyFilters(baseQuery, model);
+INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId);
-        var joinedQuery = ApplyConditionalJoins(filterQuery, model);
+INSERT INTO #P_ProfitCenters SELECT Code FROM dbo.fn_GetSearchProfitCenters(@SearchId);
+INSERT INTO #P_WorkCenters SELECT Code FROM dbo.fn_GetSearchWorkCenters(@SearchId);
+INSERT INTO #P_OperatorIDs SELECT OperatorID FROM dbo.fn_GetSearchOperatorIDs(@SearchId);
+INSERT INTO #P_ComponentLots SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId);
+INSERT INTO #P_PartOperations SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId);
 
-        var compiled = _compiler.Compile(joinedQuery);
+-- Check which filters are active based on temp table population
-        return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
+DECLARE @HasWorkOrderFilter BIT = CASE WHEN EXISTS(SELECT 1 FROM #P_WorkOrders) THEN 1 ELSE 0 END;
-    }
+-- etc.
-
-    private Query BuildBaseWorkOrderQuery(SearchModel model)
-    {
-        return new Query("WorkOrder_Curr as wo")
-            .Select("wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode",
-                    "wo.StatusUpdateDT", "wo.QuantityOrdered", "wo.QuantityCompleted");
-    }
-
-    private Query ApplyFilters(Query query, SearchModel model)
-    {
-        if (model.TimespanFilterEnabled && model.MinimumDT.HasValue && model.MaximumDT.HasValue)
-        {
-            query = query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
-        }
-
-        if (model.WorkOrderFilterEnabled)
-        {
-            query = query.WhereIn("wo.WorkOrderNumber",
-                model.WorkOrderFilter.Select(w => w.WorkOrderNumber).ToList());
-        }
-
-        if (model.ItemNumberFilterEnabled)
-        {
-            query = query.WhereIn("wo.ItemNumber",
-                model.ItemNumberFilter.Select(i => i.ItemNumber).ToList());
-        }
-
-        return query;
-    }
-
-    private Query ApplyConditionalJoins(Query query, SearchModel model)
-    {
-        if (model.ProfitCenterFilterEnabled)
-        {
-            query = query
-                .Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
-                .Join("WorkCenter as wc", "wc.Code", "wos.WorkCenterCode")
-                .Join("OrgHierarchy as oh", "oh.WorkCenterCode", "wc.Code")
-                .WhereIn("oh.ProfitCenterCode",
-                    model.ProfitCenterFilter.Select(p => p.Code).ToList());
-        }
-
-        if (model.WorkCenterFilterEnabled)
-        {
-            query = query
-                .Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
-                .WhereIn("wos.WorkCenterCode",
-                    model.WorkCenterFilter.Select(w => w.Code).ToList());
-        }
-
-        return query;
-    }
-}
 ```
 
+This approach eliminates the need to pass filter collections from C# to SQL, simplifying the interface to a single `searchId` parameter.
+
 **Work Order Traversal (Stored Procedure):**
 
 The iterative downstream work order traversal is implemented as a stored procedure (`dbo.TraverseWorkOrders`) due to its iterative nature with temp tables and MERGE operations. The maximum iteration count is configurable via `SearchProcessingOptions.MaxTraversalIterations` (default: 20):
@@ -675,13 +600,11 @@ The system SHALL optionally extract Manufacturing Information System (MIS) data
 
 ---
 
-### Requirement: Table-Valued Parameter Creation
+### Requirement: Simplified Query Parameter Passing
 
-The system SHALL create SQL table-valued parameters for efficient filter data transmission using Microsoft.Data.SqlClient.
+The system SHALL pass only the SearchId parameter to search queries, with filter extraction handled by SQL functions.
 
-#### Migration Note: SqlClient and TVP Options
+#### Migration Note: SqlClient Package Change (Required)
-
-**SQL Client Package Change (Required)**
 
 Replace `System.Data.SqlClient` with `Microsoft.Data.SqlClient`:
 
@@ -693,92 +616,37 @@ using System.Data.SqlClient;
 using Microsoft.Data.SqlClient;
 ```
 
-This is a cross-cutting change affecting all database access code. The API is largely compatible but `Microsoft.Data.SqlClient` is the actively maintained package with security updates.
+#### Simplified Parameter Model
 
-**TVP Implementation Options**
+With extraction functions, the query builder interface is simplified:
-
-The legacy approach using `DataTable` remains valid. An alternative using `IEnumerable<SqlDataRecord>` offers better performance for large datasets:
 
 ```csharp
-// Option 1: DataTable approach (legacy pattern, still supported)
+public interface ISearchQueryBuilder
-public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter(
-    this SearchModel model)
 {
-    var dataTable = new DataTable();
+    SearchQueryResult BuildSearchQuery(int searchId);
-    dataTable.Columns.Add("WorkOrderNumber", typeof(long));
+    SearchQueryResult BuildMisQuery(int searchId);
-    foreach (var entry in model.WorkOrderFilter)
+    SearchQueryResult BuildMisNonMatchQuery(int searchId);
-    {
-        dataTable.Rows.Add(entry.WorkOrderNumber);
-    }
-    return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
 }
 
-// Option 2: SqlDataRecord approach (more efficient for large datasets)
+// Query execution only needs SearchId
-public static IEnumerable<SqlDataRecord> ToWorkOrderRecords(
+var result = queryBuilder.BuildSearchQuery(searchId);
-    this IEnumerable<WorkOrderFilterEntry> entries)
+var results = await connection.QueryAsync<SearchResult>(result.Sql, new { SearchId = searchId });
-{
-    var metadata = new SqlMetaData("WorkOrderNumber", SqlDbType.BigInt);
-    var record = new SqlDataRecord(metadata);
-
-    foreach (var entry in entries)
-    {
-        record.SetInt64(0, entry.WorkOrderNumber);
-        yield return record;
-    }
-}
 ```
 
-#### Inputs
-
-- `SearchModel` with populated filter entry collections
-- SQL Server table type definitions for each parameter type
-
-#### Outputs
-
-- `SqlMapper.ICustomQueryParameter` objects created via `AsTableValuedParameter()`:
-  - `WorkOrderFilterParameter`: WorkOrderNumber column
-  - `ItemNumberFilterParameter`: ItemNumber column
-  - `ProfitCenterFilterParameter`: Code column
-  - `WorkCenterFilterParameter`: Code column
-  - `ComponentLotFilterParameter`: ComponentLotNumber + ItemNumber columns
-  - `OperatorFilterParameter`: UserName column
-  - `ItemOperationMisFilterParameter`: ItemNumber + OperationNumber + MisNumber + MisRevision columns
-
 #### Business Rules
 
-- Each parameter type MUST match the corresponding SQL Server table type schema
+- Search queries SHALL accept only `searchId` as the filter parameter
-- DataTable column types MUST match the expected SQL types (long, string)
+- SQL extraction functions handle JSON parsing and filter population
-- Parameters are created even for empty filter collections (empty DataTable)
+- TVP infrastructure for search execution has been removed
-- Operator filter uses UserID (not AddressNumber) as the parameter value
+- TVPs remain in use for reference data lookups (LookupItemsAsync, etc.)
 - All database operations SHOULD use async methods (`QueryAsync`, `ExecuteAsync`)
 
-#### Scenario: Create work order filter parameter
+#### Scenario: Execute search with simplified parameter
 
-- **WHEN** SearchModel contains WorkOrderFilter with entries [12345, 67890]
+- **WHEN** search ID 123 is processed
-- **THEN** the system creates a DataTable with WorkOrderNumber column
+- **THEN** query builder generates SQL with `@SearchId` parameter only
-- **AND** populates two rows with the work order numbers
+- **AND** extraction functions populate temporary filter tables from Search.Criteria JSON
-- **AND** returns parameter of type "WorkOrderFilterParameter"
+- **AND** no DataTable or TVP creation is needed for search execution
-
-#### Scenario: Create component lot filter parameter
-
-- **WHEN** SearchModel contains ComponentLotFilter with lot "LOT-A" for item "ITEM-001"
-- **THEN** the system creates a DataTable with ComponentLotNumber and ItemNumber columns
-- **AND** populates one row with both values
-- **AND** returns parameter of type "ComponentLotFilterParameter"
-
-#### Scenario: Create empty filter parameter
-
-- **WHEN** SearchModel contains empty WorkOrderFilter collection
-- **THEN** the system creates a DataTable with WorkOrderNumber column
-- **AND** the DataTable has zero rows
-- **AND** returns valid parameter that produces empty result set
-
-#### Scenario: Create item operation MIS filter parameter
-
-- **WHEN** SearchModel contains ItemOperationMisFilter with one entry
-- **THEN** the system creates a DataTable with four columns
-- **AND** populates ItemNumber, OperationNumber, MisNumber, MisRevision
-- **AND** returns parameter of type "ItemOperationMisFilterParameter"
 
 ---