diff --git a/DOCUMENTATION/Architecture/DataFlow.md b/DOCUMENTATION/Architecture/DataFlow.md index 3f09d0a..a8798c9 100644 --- a/DOCUMENTATION/Architecture/DataFlow.md +++ b/DOCUMENTATION/Architecture/DataFlow.md @@ -19,6 +19,8 @@ The search flow mirrors the legacy pattern, modernized for ASP.NET Core: 4. SearchProcessorService (BackgroundService) polls └─> Finds queued searches + └─> Passes SearchId to query builder + └─> SQL extraction functions read criteria from Search.Criteria JSON └─> Executes query against local cache └─> Generates Excel via ClosedXML └─> Stores result in Search.Results (VARBINARY) diff --git a/DOCUMENTATION/Architecture/Database.md b/DOCUMENTATION/Architecture/Database.md index 2fd9005..016df49 100644 --- a/DOCUMENTATION/Architecture/Database.md +++ b/DOCUMENTATION/Architecture/Database.md @@ -121,7 +121,7 @@ The scoping tool cache database includes these primary tables: | Table | Purpose | |-------|---------| -| `Search` | User search requests, status, and results (Excel as VARBINARY) | +| `Search` | User search requests, status, criteria (JSON), and results (Excel as VARBINARY) | | `DataUpdate` | Tracks last sync timestamp per data type | | `WorkOrder_Curr` | Current work orders from JDE | | `WorkOrder_Hist` | Historical work orders from JDE | @@ -134,6 +134,38 @@ The scoping tool cache database includes these primary tables: | `ProfitCenter` | Profit center reference data | | `SchemaVersions` | DbUp tracking table (auto-created) | +## Search Criteria Extraction Functions + +The database includes SQL functions that extract filter criteria from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, with all filter extraction happening in SQL Server. + +### Scalar Functions (3) + +| Function | Returns | Extracts | +|----------|---------|----------| +| `fn_GetSearchMinimumDt` | DATETIME2 | `$.MinimumDt` | +| `fn_GetSearchMaximumDt` | DATETIME2 | `$.MaximumDt` | +| `fn_GetSearchExtractMisData` | BIT | `$.ExtractMisData` | + +### Table-Valued Functions (8) + +| Function | Returns | Extracts | +|----------|---------|----------| +| `fn_GetSearchWorkOrders` | WorkOrderNumber | `$.WorkOrderNumbers` array | +| `fn_GetSearchItemNumbers` | ItemNumber | `$.ItemNumbers` array | +| `fn_GetSearchProfitCenters` | Code | `$.ProfitCenters` array | +| `fn_GetSearchWorkCenters` | Code | `$.WorkCenters` array | +| `fn_GetSearchOperatorIDs` | OperatorID | `$.OperatorIDs` array | +| `fn_GetSearchComponentLots` | LotNumber, ItemNumber | `$.ComponentLotNumbers` array | +| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision | `$.PartOperations` array | + +### Validation Procedure + +| Procedure | Purpose | +|-----------|---------| +| `usp_ValidateSearchCriteria` | Validates search exists and has valid JSON criteria | + +Error codes: 50001 (not found), 50002 (no criteria), 50003 (invalid JSON) + ## Example Migration Scripts ### 001_CreateSearchTable.sql diff --git a/DOCUMENTATION/Architecture/Testing.md b/DOCUMENTATION/Architecture/Testing.md index 459d85b..5992aad 100644 --- a/DOCUMENTATION/Architecture/Testing.md +++ b/DOCUMENTATION/Architecture/Testing.md @@ -5,22 +5,34 @@ The test project uses xUnit for the framework, Shouldly for assertions, and NSub ## Project Structure ``` -JdeScoping.Tests/ -├── Unit/ -│ ├── Services/ -│ │ ├── SearchServiceTests.cs -│ │ ├── ExcelExportServiceTests.cs -│ │ └── DataSyncOrchestratorTests.cs -│ ├── Repositories/ -│ │ └── SearchRepositoryTests.cs -│ └── Models/ -│ └── SearchCriteriaTests.cs -└── Integration/ - ├── ApiTests/ - │ ├── SearchControllerTests.cs - │ └── LookupControllerTests.cs - └── RepositoryTests/ - └── JdeRepositoryTests.cs +tests/ +├── JdeScoping.Tests/ +│ ├── Unit/ +│ │ ├── Services/ +│ │ │ ├── SearchServiceTests.cs +│ │ │ ├── ExcelExportServiceTests.cs +│ │ │ └── DataSyncOrchestratorTests.cs +│ │ ├── Repositories/ +│ │ │ └── SearchRepositoryTests.cs +│ │ └── Models/ +│ │ └── SearchCriteriaTests.cs +│ └── Integration/ +│ ├── ApiTests/ +│ │ ├── SearchControllerTests.cs +│ │ └── LookupControllerTests.cs +│ └── RepositoryTests/ +│ └── JdeRepositoryTests.cs +├── JdeScoping.Api.IntegrationTests/ +│ └── ... (API integration tests) +└── JdeScoping.Database.Tests/ + ├── Infrastructure/ + │ └── DatabaseTestBase.cs + ├── Functions/ + │ ├── ScalarFunctionTests.cs + │ ├── SimpleTableFunctionTests.cs + │ └── ComplexTableFunctionTests.cs + └── Procedures/ + └── ValidateSearchCriteriaProcedureTests.cs ``` ## Unit Tests @@ -175,7 +187,64 @@ public class SearchRepositoryIntegrationTests : IDisposable } ``` +## Database Tests + +The `JdeScoping.Database.Tests` project tests SQL functions and stored procedures against a live SQL Server instance. + +### Test Infrastructure + +Tests inherit from `DatabaseTestBase` which provides: +- Connection management via `DatabaseTestFixture` +- Automatic test data cleanup +- Helper methods for inserting test searches + +```csharp +[Collection("DatabaseTests")] +public class ScalarFunctionTests : DatabaseTestBase +{ + [Fact] + public async Task fn_GetSearchMinimumDt_ValidSearch_ReturnsDateTime() + { + // Arrange + var criteria = new SearchCriteria { MinimumDt = new DateTime(2024, 6, 15) }; + var searchId = await InsertTestSearchAsync(criteria); + + // Act + var result = await Connection.QuerySingleOrDefaultAsync( + "SELECT dbo.fn_GetSearchMinimumDt(@SearchId)", + new { SearchId = searchId }); + + // Assert + result.Should().BeCloseTo(criteria.MinimumDt.Value, TimeSpan.FromSeconds(1)); + } +} +``` + +### Test Categories + +| Category | Tests | Description | +|----------|-------|-------------| +| Scalar Functions | 15 | Test `fn_GetSearchMinimumDt`, `fn_GetSearchMaximumDt`, `fn_GetSearchExtractMisData` | +| Simple Table Functions | 38 | Test array extraction for work orders, items, profit centers, work centers, operators | +| Complex Table Functions | 23 | Test object extraction for component lots and part operations | +| Validation Procedure | 6 | Test `usp_ValidateSearchCriteria` error handling | + +**Total: 82 database tests** + +### Running Database Tests + +```bash +# Run all database tests +dotnet test tests/JdeScoping.Database.Tests + +# Run specific test class +dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ScalarFunctionTests" +``` + +**Prerequisites**: SQL Server must be running on localhost:1434 with the ScopingTool database. + ## Related Documentation - [Core Project](./CoreProject.md) - [Dependencies](./Dependencies.md) +- [Database](./Database.md) diff --git a/DOCUMENTATION/Database/ExtractionFunctions.md b/DOCUMENTATION/Database/ExtractionFunctions.md new file mode 100644 index 0000000..35db0c0 --- /dev/null +++ b/DOCUMENTATION/Database/ExtractionFunctions.md @@ -0,0 +1,237 @@ +# Search Criteria Extraction Functions + +SQL functions that extract filter values from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, simplifying the C# to SQL interface. + +## Overview + +| Type | Count | Purpose | +|------|-------|---------| +| Scalar Functions | 3 | Extract single values (dates, booleans) | +| Table-Valued Functions | 8 | Extract arrays and object collections | +| Validation Procedure | 1 | Pre-flight validation with error codes | + +## Scalar Functions + +### fn_GetSearchMinimumDt + +Extracts the minimum date filter value. + +```sql +CREATE FUNCTION dbo.fn_GetSearchMinimumDt(@SearchId INT) +RETURNS DATETIME2(7) +``` + +**JSON Path**: `$.MinimumDt` + +**Returns**: DATETIME2 value or NULL if not found/invalid + +### fn_GetSearchMaximumDt + +Extracts the maximum date filter value. + +```sql +CREATE FUNCTION dbo.fn_GetSearchMaximumDt(@SearchId INT) +RETURNS DATETIME2(7) +``` + +**JSON Path**: `$.MaximumDt` + +**Returns**: DATETIME2 value or NULL if not found/invalid + +### fn_GetSearchExtractMisData + +Extracts the MIS data extraction flag. + +```sql +CREATE FUNCTION dbo.fn_GetSearchExtractMisData(@SearchId INT) +RETURNS BIT +``` + +**JSON Path**: `$.ExtractMisData` + +**Returns**: 1 (true), 0 (false), or NULL if not found/invalid + +## Table-Valued Functions (Simple Arrays) + +### fn_GetSearchWorkOrders + +Extracts work order number filter values. + +```sql +CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT) +RETURNS TABLE +AS RETURN ( + SELECT j.WorkOrderNumber + FROM ... OPENJSON(..., '$.WorkOrderNumbers') WITH (WorkOrderNumber BIGINT '$') j +) +``` + +**JSON Path**: `$.WorkOrderNumbers` + +**Returns**: Table with `WorkOrderNumber BIGINT` column + +### fn_GetSearchItemNumbers + +Extracts item number filter values. + +```sql +CREATE FUNCTION dbo.fn_GetSearchItemNumbers(@SearchId INT) +RETURNS TABLE +``` + +**JSON Path**: `$.ItemNumbers` + +**Returns**: Table with `ItemNumber VARCHAR(128)` column + +### fn_GetSearchProfitCenters + +Extracts profit center filter values. + +```sql +CREATE FUNCTION dbo.fn_GetSearchProfitCenters(@SearchId INT) +RETURNS TABLE +``` + +**JSON Path**: `$.ProfitCenters` + +**Returns**: Table with `Code VARCHAR(12)` column + +### fn_GetSearchWorkCenters + +Extracts work center filter values. + +```sql +CREATE FUNCTION dbo.fn_GetSearchWorkCenters(@SearchId INT) +RETURNS TABLE +``` + +**JSON Path**: `$.WorkCenters` + +**Returns**: Table with `Code VARCHAR(12)` column + +### fn_GetSearchOperatorIDs + +Extracts operator ID filter values. + +```sql +CREATE FUNCTION dbo.fn_GetSearchOperatorIDs(@SearchId INT) +RETURNS TABLE +``` + +**JSON Path**: `$.OperatorIDs` + +**Returns**: Table with `OperatorID VARCHAR(128)` column + +## Table-Valued Functions (Complex Objects) + +### fn_GetSearchComponentLots + +Extracts component lot filter values (lot/item pairs). + +```sql +CREATE FUNCTION dbo.fn_GetSearchComponentLots(@SearchId INT) +RETURNS TABLE +AS RETURN ( + SELECT j.LotNumber, j.ItemNumber + FROM ... OPENJSON(..., '$.ComponentLotNumbers') WITH ( + LotNumber VARCHAR(30) '$.LotNumber', + ItemNumber VARCHAR(128) '$.ItemNumber' + ) j +) +``` + +**JSON Path**: `$.ComponentLotNumbers` + +**Returns**: Table with `LotNumber VARCHAR(30)`, `ItemNumber VARCHAR(128)` columns + +### fn_GetSearchPartOperations + +Extracts part operation filter values (item/operation/MIS combinations). + +```sql +CREATE FUNCTION dbo.fn_GetSearchPartOperations(@SearchId INT) +RETURNS TABLE +AS RETURN ( + SELECT j.ItemNumber, j.OperationNumber, j.MisNumber, j.MisRevision + FROM ... OPENJSON(..., '$.PartOperations') WITH ( + ItemNumber VARCHAR(128) '$.ItemNumber', + OperationNumber VARCHAR(10) '$.OperationNumber', + MisNumber VARCHAR(10) '$.MisNumber', + MisRevision VARCHAR(10) '$.MisRevision' + ) j +) +``` + +**JSON Path**: `$.PartOperations` + +**Returns**: Table with `ItemNumber`, `OperationNumber`, `MisNumber`, `MisRevision` columns + +## Validation Procedure + +### usp_ValidateSearchCriteria + +Validates that a search exists and has valid JSON criteria. + +```sql +CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT) +``` + +**Error Codes**: + +| Code | Condition | Message | +|------|-----------|---------| +| 50001 | Search not found | "Search ID {id} not found" | +| 50002 | Criteria NULL or empty | "Search ID {id} has no criteria" | +| 50003 | Invalid JSON | "Search ID {id} has invalid JSON" | + +**Usage**: +```sql +-- Throws on validation failure +EXEC dbo.usp_ValidateSearchCriteria @SearchId = 123; +``` + +## Error Handling + +All extraction functions handle errors gracefully: + +- **Scalar functions**: Return NULL for missing search, NULL criteria, or invalid JSON +- **Table-valued functions**: Return empty result set for missing search, NULL criteria, or invalid JSON +- **Validation procedure**: Throws errors via `THROW` statement for calling code to handle + +## Design Pattern + +Functions use a CTE pattern to pre-filter valid JSON before calling OPENJSON: + +```sql +WITH ValidSearch AS ( + SELECT Criteria + FROM dbo.Search + WHERE ID = @SearchId + AND Criteria IS NOT NULL + AND ISJSON(Criteria) = 1 +) +SELECT ... +FROM ValidSearch s +CROSS APPLY OPENJSON(s.Criteria, '$.ArrayPath') ... +``` + +This pattern prevents OPENJSON from running on invalid JSON, which would cause runtime errors. + +## Migration Scripts + +| Script | Contents | +|--------|----------| +| `045_CreateScalarExtractionFunctions.sql` | Scalar functions | +| `046_CreateSimpleTableFunctions.sql` | Simple array TVFs | +| `047_CreateComplexTableFunctions.sql` | Complex object TVFs | +| `048_CreateValidateSearchCriteriaProcedure.sql` | Validation procedure | + +## Testing + +82 tests in `JdeScoping.Database.Tests` verify function behavior: + +```bash +dotnet test tests/JdeScoping.Database.Tests +``` + +See [Testing](../Architecture/Testing.md) for details. diff --git a/Tools/CacheConverter/Program.cs b/Tools/CacheConverter/Program.cs index bf97217..bcc9367 100644 --- a/Tools/CacheConverter/Program.cs +++ b/Tools/CacheConverter/Program.cs @@ -1,5 +1,4 @@ using System.Data; -using System.Text; using System.Text.Json; using System.Text.RegularExpressions; using ProtoBuf.Data; @@ -102,55 +101,32 @@ foreach (var jsonFile in jsonFiles) } // Stream JSON and write to protobuf in batches - using var inputFs = new FileStream(jsonFile, FileMode.Open, FileAccess.Read, FileShare.Read, 256 * 1024, FileOptions.SequentialScan); - using var decompressStream = new DecompressionStream(inputFs); - using var outputFs = new FileStream(outputFile, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024); - using var compressStream = new CompressionStream(outputFs, level: 3); + await using var inputFs = new FileStream(jsonFile, FileMode.Open, FileAccess.Read, FileShare.Read, 256 * 1024, FileOptions.SequentialScan | FileOptions.Asynchronous); + await using var decompressStream = new DecompressionStream(inputFs); + await using var outputFs = new FileStream(outputFile, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024, FileOptions.Asynchronous); + await using var compressStream = new CompressionStream(outputFs, level: 3); int rowCount = 0; int batchCount = 0; - // Stream JSON records one at a time - var buffer = new byte[4096]; - using var memoryStream = new MemoryStream(); - - int bytesRead; - while ((bytesRead = decompressStream.Read(buffer, 0, buffer.Length)) > 0) + // True streaming: DeserializeAsyncEnumerable streams each array element without loading entire JSON + var jsonOptions = new JsonSerializerOptions { PropertyNameCaseInsensitive = true }; + await foreach (var element in JsonSerializer.DeserializeAsyncEnumerable( + decompressStream, + jsonOptions)) { - memoryStream.Write(buffer, 0, bytesRead); - } + var row = dataTable.NewRow(); + ReadJsonElement(element, row, dataTable); + dataTable.Rows.Add(row); + rowCount++; - memoryStream.Position = 0; - var jsonReader = new Utf8JsonReader(memoryStream.ToArray(), new JsonReaderOptions { AllowTrailingCommas = true }); - - // Skip to start of array - while (jsonReader.Read()) - { - if (jsonReader.TokenType == JsonTokenType.StartArray) - break; - } - - // Read each object in the array - while (jsonReader.Read()) - { - if (jsonReader.TokenType == JsonTokenType.EndArray) - break; - - if (jsonReader.TokenType == JsonTokenType.StartObject) + // Write batch when we hit the batch size + if (dataTable.Rows.Count >= BatchSize) { - var row = dataTable.NewRow(); - ReadJsonObject(ref jsonReader, row, dataTable); - dataTable.Rows.Add(row); - rowCount++; - - // Write batch when we hit the batch size - if (dataTable.Rows.Count >= BatchSize) - { - using var reader = dataTable.CreateDataReader(); - DataSerializer.Serialize(compressStream, reader); - dataTable.Clear(); - batchCount++; - } + using var reader = dataTable.CreateDataReader(); + DataSerializer.Serialize(compressStream, reader); + dataTable.Clear(); + batchCount++; } } @@ -162,7 +138,7 @@ foreach (var jsonFile in jsonFiles) batchCount++; } - compressStream.Flush(); + await compressStream.FlushAsync(); var newSize = new FileInfo(outputFile).Length; totalNewSize += newSize; @@ -243,68 +219,58 @@ static Type MapSqlTypeToNet(string sqlType) => sqlType.ToUpperInvariant() switch }; /// -/// Read a JSON object into a DataRow using streaming reader. +/// Read a JsonElement (object) into a DataRow. /// -static void ReadJsonObject(ref Utf8JsonReader reader, DataRow row, DataTable table) +static void ReadJsonElement(JsonElement element, DataRow row, DataTable table) { - while (reader.Read()) + foreach (var property in element.EnumerateObject()) { - if (reader.TokenType == JsonTokenType.EndObject) - break; - - if (reader.TokenType == JsonTokenType.PropertyName) + // Find matching column (case-insensitive) + DataColumn? column = null; + foreach (DataColumn col in table.Columns) { - var propertyName = reader.GetString()!; - reader.Read(); // Move to value - - // Find matching column (case-insensitive) - DataColumn? column = null; - foreach (DataColumn col in table.Columns) + if (col.ColumnName.Equals(property.Name, StringComparison.OrdinalIgnoreCase)) { - if (col.ColumnName.Equals(propertyName, StringComparison.OrdinalIgnoreCase)) - { - column = col; - break; - } + column = col; + break; } - - if (column == null) - { - // Skip unknown property - SkipJsonValue(ref reader); - continue; - } - - row[column] = ReadJsonValue(ref reader, column.DataType); } + + if (column == null) + { + // Skip unknown property + continue; + } + + row[column] = ReadJsonElementValue(property.Value, column.DataType); } } /// -/// Read a JSON value and convert to the target .NET type. +/// Read a JSON value from JsonElement and convert to the target .NET type. /// -static object ReadJsonValue(ref Utf8JsonReader reader, Type targetType) +static object ReadJsonElementValue(JsonElement element, Type targetType) { - if (reader.TokenType == JsonTokenType.Null) + if (element.ValueKind == JsonValueKind.Null || element.ValueKind == JsonValueKind.Undefined) return DBNull.Value; if (targetType == typeof(string)) { - return reader.TokenType switch + return element.ValueKind switch { - JsonTokenType.String => reader.GetString() ?? (object)DBNull.Value, - JsonTokenType.Number => reader.GetDecimal().ToString(), - JsonTokenType.True => "true", - JsonTokenType.False => "false", + JsonValueKind.String => element.GetString() ?? (object)DBNull.Value, + JsonValueKind.Number => element.GetDecimal().ToString(), + JsonValueKind.True => "true", + JsonValueKind.False => "false", _ => DBNull.Value }; } if (targetType == typeof(DateTime)) { - if (reader.TokenType == JsonTokenType.String) + if (element.ValueKind == JsonValueKind.String) { - var str = reader.GetString(); + var str = element.GetString(); if (str != null && DateTime.TryParse(str, out var dt)) return dt; } @@ -313,71 +279,45 @@ static object ReadJsonValue(ref Utf8JsonReader reader, Type targetType) if (targetType == typeof(long)) { - return reader.TokenType switch + return element.ValueKind switch { - JsonTokenType.Number => reader.GetInt64(), - JsonTokenType.String when long.TryParse(reader.GetString(), out var val) => val, + JsonValueKind.Number => element.GetInt64(), + JsonValueKind.String when long.TryParse(element.GetString(), out var val) => val, _ => DBNull.Value }; } if (targetType == typeof(int)) { - return reader.TokenType switch + return element.ValueKind switch { - JsonTokenType.Number => reader.GetInt32(), - JsonTokenType.String when int.TryParse(reader.GetString(), out var val) => val, + JsonValueKind.Number => element.GetInt32(), + JsonValueKind.String when int.TryParse(element.GetString(), out var val) => val, _ => DBNull.Value }; } if (targetType == typeof(decimal)) { - return reader.TokenType switch + return element.ValueKind switch { - JsonTokenType.Number => reader.GetDecimal(), - JsonTokenType.String when decimal.TryParse(reader.GetString(), out var val) => val, + JsonValueKind.Number => element.GetDecimal(), + JsonValueKind.String when decimal.TryParse(element.GetString(), out var val) => val, _ => DBNull.Value }; } if (targetType == typeof(bool)) { - return reader.TokenType switch + return element.ValueKind switch { - JsonTokenType.True => true, - JsonTokenType.False => false, - JsonTokenType.Number => reader.GetInt32() != 0, - JsonTokenType.String => reader.GetString()?.Equals("true", StringComparison.OrdinalIgnoreCase) ?? false, + JsonValueKind.True => true, + JsonValueKind.False => false, + JsonValueKind.Number => element.GetInt32() != 0, + JsonValueKind.String => element.GetString()?.Equals("true", StringComparison.OrdinalIgnoreCase) ?? false, _ => DBNull.Value }; } return DBNull.Value; } - -/// -/// Skip a JSON value (used for unknown properties). -/// -static void SkipJsonValue(ref Utf8JsonReader reader) -{ - if (reader.TokenType == JsonTokenType.StartObject) - { - int depth = 1; - while (depth > 0 && reader.Read()) - { - if (reader.TokenType == JsonTokenType.StartObject) depth++; - else if (reader.TokenType == JsonTokenType.EndObject) depth--; - } - } - else if (reader.TokenType == JsonTokenType.StartArray) - { - int depth = 1; - while (depth > 0 && reader.Read()) - { - if (reader.TokenType == JsonTokenType.StartArray) depth++; - else if (reader.TokenType == JsonTokenType.EndArray) depth--; - } - } - // Simple values are already consumed by the Read() call -} diff --git a/openspec/specs/data-access/spec.md b/openspec/specs/data-access/spec.md index fc97868..6e75c88 100644 --- a/openspec/specs/data-access/spec.md +++ b/openspec/specs/data-access/spec.md @@ -467,40 +467,65 @@ catch (OracleException ex) --- -### Requirement: Table-valued parameter support +### Requirement: Search criteria extraction via SQL functions -The system SHALL use DataTable for SQL Server table-valued parameters in lookup methods. +The system SHALL use SQL extraction functions to retrieve filter criteria directly from the `Search.Criteria` JSON column. #### Implementation Pattern +Search query building now uses SearchId to invoke extraction functions rather than passing filter values from C#: + ```csharp -public async Task> LookupItemsAsync(List itemNumbers, CancellationToken ct = default) +public SearchQueryResult BuildSearchQuery(int searchId) { - await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct); + // Query builder generates SQL that calls extraction functions + // Example generated SQL fragment: + // INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId) + // INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId) - var table = new DataTable(); - table.Columns.Add("ItemNumber", typeof(string)); - foreach (var itemNumber in itemNumbers) - { - table.Rows.Add(itemNumber); - } - - var parameters = new DynamicParameters(); - parameters.Add("@itemNumbers", table.AsTableValuedParameter("dbo.ItemNumberFilterParameter")); - - return (await connection.QueryAsync( - LotFinderQueries.SQL_LOOKUP_ITEMS, - parameters, - commandTimeout: _options.Value.DefaultTimeoutSeconds)) - .AsList(); + return new SearchQueryResult(sql, new { SearchId = searchId }); } ``` #### Business Rules -- TVPs SHALL use `AsTableValuedParameter` extension with correct type name +- Query builder SHALL accept only `searchId` parameter (not full criteria object) +- SQL queries SHALL call extraction functions to populate temporary filter tables +- Extraction functions handle JSON parsing and validation in SQL Server +- Invalid JSON or missing criteria results in empty filter sets (no errors thrown) + +#### Available Extraction Functions + +| Function | Returns | +|----------|---------| +| `fn_GetSearchMinimumDt` | DATETIME2 scalar | +| `fn_GetSearchMaximumDt` | DATETIME2 scalar | +| `fn_GetSearchExtractMisData` | BIT scalar | +| `fn_GetSearchWorkOrders` | WorkOrderNumber table | +| `fn_GetSearchItemNumbers` | ItemNumber table | +| `fn_GetSearchProfitCenters` | Code table | +| `fn_GetSearchWorkCenters` | Code table | +| `fn_GetSearchOperatorIDs` | OperatorID table | +| `fn_GetSearchComponentLots` | LotNumber, ItemNumber table | +| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision table | + +#### Scenario: Build search query with extraction functions + +- **WHEN** `BuildSearchQuery(123)` is called for a search with work order filter +- **THEN** generated SQL includes `SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(123)` +- **AND** only the `@SearchId` parameter is passed to the query + +--- + +### Requirement: Table-valued parameter support for lookups + +The system SHALL use DataTable for SQL Server table-valued parameters in reference data lookup methods. + +#### Business Rules + +- TVPs are used for batch lookups (LookupItemsAsync, LookupWorkordersAsync, etc.) +- TVPs are NOT used for search query execution (replaced by extraction functions) - DataTable column names SHALL match TVP type column names -- TVPs enable efficient batch lookups with single database round-trip #### Scenario: Batch lookup with TVP diff --git a/openspec/specs/database-schema/spec.md b/openspec/specs/database-schema/spec.md index feff5d4..0ba5572 100644 --- a/openspec/specs/database-schema/spec.md +++ b/openspec/specs/database-schema/spec.md @@ -1079,6 +1079,90 @@ var results = await context.WorkOrders ## Functions +### Requirement: Search Criteria Extraction Functions + +The system SHALL provide SQL functions to extract search criteria values from the `Search.Criteria` JSON column. + +#### Scalar Functions + +| Function | Return Type | Description | +|----------|-------------|-------------| +| `fn_GetSearchMinimumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MinimumDt` value | +| `fn_GetSearchMaximumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MaximumDt` value | +| `fn_GetSearchExtractMisData(@SearchId INT)` | BIT | Extracts `$.ExtractMisData` value | + +#### Table-Valued Functions (Simple Arrays) + +| Function | Return Columns | Description | +|----------|----------------|-------------| +| `fn_GetSearchWorkOrders(@SearchId INT)` | WorkOrderNumber BIGINT | Extracts `$.WorkOrderNumbers` array | +| `fn_GetSearchItemNumbers(@SearchId INT)` | ItemNumber VARCHAR(128) | Extracts `$.ItemNumbers` array | +| `fn_GetSearchProfitCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.ProfitCenters` array | +| `fn_GetSearchWorkCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.WorkCenters` array | +| `fn_GetSearchOperatorIDs(@SearchId INT)` | OperatorID VARCHAR(128) | Extracts `$.OperatorIDs` array | + +#### Table-Valued Functions (Complex Objects) + +| Function | Return Columns | Description | +|----------|----------------|-------------| +| `fn_GetSearchComponentLots(@SearchId INT)` | LotNumber VARCHAR(30), ItemNumber VARCHAR(128) | Extracts `$.ComponentLotNumbers` array | +| `fn_GetSearchPartOperations(@SearchId INT)` | ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10) | Extracts `$.PartOperations` array | + +#### Business Rules + +- Scalar functions return NULL if search not found, criteria is NULL, or JSON is invalid +- Table-valued functions return empty result set if search not found, criteria is NULL, or JSON is invalid +- TVFs use CTE pattern to pre-filter valid JSON before OPENJSON (prevents runtime errors) +- TVFs use `OPENJSON...WITH` syntax for type-safe extraction + +#### Scenario: Extract work order filter values + +- **WHEN** Search ID 123 has Criteria containing `{"WorkOrderNumbers":[12345,67890]}` +- **THEN** `SELECT * FROM dbo.fn_GetSearchWorkOrders(123)` returns two rows with WorkOrderNumber values 12345 and 67890 + +#### Scenario: Handle invalid JSON gracefully + +- **WHEN** Search ID 456 has Criteria containing invalid JSON text +- **THEN** extraction functions return NULL (scalar) or empty result set (TVF) without throwing errors + +--- + +### Requirement: Validate Search Criteria Procedure + +The system SHALL provide a stored procedure for strict validation of search criteria with error reporting. + +#### Procedure Signature + +```sql +CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT) +``` + +#### Error Codes + +| Error Code | Condition | Message Pattern | +|------------|-----------|-----------------| +| 50001 | Search ID not found | "Search ID {id} not found" | +| 50002 | Criteria is NULL or empty | "Search ID {id} has no criteria" | +| 50003 | Criteria is not valid JSON | "Search ID {id} has invalid JSON" | + +#### Business Rules + +- Procedure throws errors using `THROW` statement for calling code to handle +- Procedure returns 0 (success) when validation passes +- Used for pre-flight validation before query execution + +#### Scenario: Validate valid search + +- **WHEN** `usp_ValidateSearchCriteria` is called for a search with valid JSON criteria +- **THEN** procedure returns 0 (success) without throwing + +#### Scenario: Validate missing search + +- **WHEN** `usp_ValidateSearchCriteria` is called for non-existent search ID 99999 +- **THEN** procedure throws error 50001 with message "Search ID 99999 not found" + +--- + ### Requirement: MatchMIS function The system SHALL provide a table-valued function to match work order steps to MIS data. diff --git a/openspec/specs/search-processing/spec.md b/openspec/specs/search-processing/spec.md index a42f1ae..a32a1f4 100644 --- a/openspec/specs/search-processing/spec.md +++ b/openspec/specs/search-processing/spec.md @@ -86,7 +86,7 @@ The legacy system uses a T4 text template (`QueryTemplate.tt`) for SQL generatio **NuGet Package:** `SqlKata` and `SqlKata.Execution` -**Primary Query Building with SqlKata:** +**Primary Query Building with SqlKata and Extraction Functions:** ```csharp using SqlKata; @@ -94,7 +94,9 @@ using SqlKata.Compilers; public interface ISearchQueryBuilder { - SearchQueryResult BuildSearchQuery(SearchModel model); + SearchQueryResult BuildSearchQuery(int searchId); + SearchQueryResult BuildMisQuery(int searchId); + SearchQueryResult BuildMisNonMatchQuery(int searchId); } public record SearchQueryResult(string Sql, IDictionary Parameters); @@ -103,127 +105,50 @@ public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder { private readonly SqlServerCompiler _compiler = new(); - public SearchQueryResult BuildSearchQuery(SearchModel model) + public SearchQueryResult BuildSearchQuery(int searchId) { + // Query uses SQL extraction functions to get criteria values + // No need to pass filter collections from C# - just the SearchId + + // Generated SQL populates temp tables from extraction functions: + // INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId) + // INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId) + // etc. + var query = new Query("WorkOrder_Curr as wo") .Select("wo.*", "wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode"); - // Conditional joins based on active filters - if (model.OperatorFilterEnabled) - { - query.Join("WorkOrderTime_Curr as wot", "wot.WorkOrderNumber", "wo.WorkOrderNumber"); - } + // Conditional joins based on temp table population (determined at runtime) + // Query builder generates SQL that checks IF EXISTS on temp tables - if (model.ProfitCenterFilterEnabled || model.WorkCenterFilterEnabled) - { - query.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber"); - } - - if (model.ComponentLotFilterEnabled) - { - query.Join("WorkOrderComponent_Curr as woc", "woc.WorkOrderNumber", "wo.WorkOrderNumber"); - } - - // Conditional WHERE clauses - if (model.TimespanFilterEnabled) - { - query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT); - } - - if (model.WorkOrderFilterEnabled) - { - query.WhereIn("wo.WorkOrderNumber", model.WorkOrderFilter.Select(w => w.WorkOrderNumber)); - } - - if (model.ItemNumberFilterEnabled) - { - query.WhereIn("wo.ItemNumber", model.ItemNumberFilter.Select(i => i.ItemNumber)); - } - - if (model.OperatorFilterEnabled) - { - query.WhereIn("wot.OperatorAN", model.OperatorFilter.Select(o => o.AddressNumber)); - } - - // Compile to SQL + parameters var compiled = _compiler.Compile(query); - return new SearchQueryResult(compiled.Sql, compiled.NamedBindings); + return new SearchQueryResult(compiled.Sql, new Dictionary { ["SearchId"] = searchId }); } } ``` -**Complex Query Composition with SqlKata:** +**Extraction Function Integration:** -```csharp -public sealed class ComposableSearchQueryBuilder -{ - private readonly SqlServerCompiler _compiler = new(); +The query builder generates SQL that populates temporary filter tables from extraction functions: - public SearchQueryResult BuildFullSearchQuery(SearchModel model) - { - // Build composable query parts - var baseQuery = BuildBaseWorkOrderQuery(model); - var filterQuery = ApplyFilters(baseQuery, model); - var joinedQuery = ApplyConditionalJoins(filterQuery, model); +```sql +-- Generated SQL fragment example: +-- Populate temp tables from extraction functions +INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId); +INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId); +INSERT INTO #P_ProfitCenters SELECT Code FROM dbo.fn_GetSearchProfitCenters(@SearchId); +INSERT INTO #P_WorkCenters SELECT Code FROM dbo.fn_GetSearchWorkCenters(@SearchId); +INSERT INTO #P_OperatorIDs SELECT OperatorID FROM dbo.fn_GetSearchOperatorIDs(@SearchId); +INSERT INTO #P_ComponentLots SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId); +INSERT INTO #P_PartOperations SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId); - var compiled = _compiler.Compile(joinedQuery); - return new SearchQueryResult(compiled.Sql, compiled.NamedBindings); - } - - private Query BuildBaseWorkOrderQuery(SearchModel model) - { - return new Query("WorkOrder_Curr as wo") - .Select("wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode", - "wo.StatusUpdateDT", "wo.QuantityOrdered", "wo.QuantityCompleted"); - } - - private Query ApplyFilters(Query query, SearchModel model) - { - if (model.TimespanFilterEnabled && model.MinimumDT.HasValue && model.MaximumDT.HasValue) - { - query = query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT); - } - - if (model.WorkOrderFilterEnabled) - { - query = query.WhereIn("wo.WorkOrderNumber", - model.WorkOrderFilter.Select(w => w.WorkOrderNumber).ToList()); - } - - if (model.ItemNumberFilterEnabled) - { - query = query.WhereIn("wo.ItemNumber", - model.ItemNumberFilter.Select(i => i.ItemNumber).ToList()); - } - - return query; - } - - private Query ApplyConditionalJoins(Query query, SearchModel model) - { - if (model.ProfitCenterFilterEnabled) - { - query = query - .Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber") - .Join("WorkCenter as wc", "wc.Code", "wos.WorkCenterCode") - .Join("OrgHierarchy as oh", "oh.WorkCenterCode", "wc.Code") - .WhereIn("oh.ProfitCenterCode", - model.ProfitCenterFilter.Select(p => p.Code).ToList()); - } - - if (model.WorkCenterFilterEnabled) - { - query = query - .Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber") - .WhereIn("wos.WorkCenterCode", - model.WorkCenterFilter.Select(w => w.Code).ToList()); - } - - return query; - } -} +-- Check which filters are active based on temp table population +DECLARE @HasWorkOrderFilter BIT = CASE WHEN EXISTS(SELECT 1 FROM #P_WorkOrders) THEN 1 ELSE 0 END; +-- etc. ``` +This approach eliminates the need to pass filter collections from C# to SQL, simplifying the interface to a single `searchId` parameter. + **Work Order Traversal (Stored Procedure):** The iterative downstream work order traversal is implemented as a stored procedure (`dbo.TraverseWorkOrders`) due to its iterative nature with temp tables and MERGE operations. The maximum iteration count is configurable via `SearchProcessingOptions.MaxTraversalIterations` (default: 20): @@ -675,13 +600,11 @@ The system SHALL optionally extract Manufacturing Information System (MIS) data --- -### Requirement: Table-Valued Parameter Creation +### Requirement: Simplified Query Parameter Passing -The system SHALL create SQL table-valued parameters for efficient filter data transmission using Microsoft.Data.SqlClient. +The system SHALL pass only the SearchId parameter to search queries, with filter extraction handled by SQL functions. -#### Migration Note: SqlClient and TVP Options - -**SQL Client Package Change (Required)** +#### Migration Note: SqlClient Package Change (Required) Replace `System.Data.SqlClient` with `Microsoft.Data.SqlClient`: @@ -693,92 +616,37 @@ using System.Data.SqlClient; using Microsoft.Data.SqlClient; ``` -This is a cross-cutting change affecting all database access code. The API is largely compatible but `Microsoft.Data.SqlClient` is the actively maintained package with security updates. +#### Simplified Parameter Model -**TVP Implementation Options** - -The legacy approach using `DataTable` remains valid. An alternative using `IEnumerable` offers better performance for large datasets: +With extraction functions, the query builder interface is simplified: ```csharp -// Option 1: DataTable approach (legacy pattern, still supported) -public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter( - this SearchModel model) +public interface ISearchQueryBuilder { - var dataTable = new DataTable(); - dataTable.Columns.Add("WorkOrderNumber", typeof(long)); - foreach (var entry in model.WorkOrderFilter) - { - dataTable.Rows.Add(entry.WorkOrderNumber); - } - return dataTable.AsTableValuedParameter("WorkOrderFilterParameter"); + SearchQueryResult BuildSearchQuery(int searchId); + SearchQueryResult BuildMisQuery(int searchId); + SearchQueryResult BuildMisNonMatchQuery(int searchId); } -// Option 2: SqlDataRecord approach (more efficient for large datasets) -public static IEnumerable ToWorkOrderRecords( - this IEnumerable entries) -{ - var metadata = new SqlMetaData("WorkOrderNumber", SqlDbType.BigInt); - var record = new SqlDataRecord(metadata); - - foreach (var entry in entries) - { - record.SetInt64(0, entry.WorkOrderNumber); - yield return record; - } -} +// Query execution only needs SearchId +var result = queryBuilder.BuildSearchQuery(searchId); +var results = await connection.QueryAsync(result.Sql, new { SearchId = searchId }); ``` -#### Inputs - -- `SearchModel` with populated filter entry collections -- SQL Server table type definitions for each parameter type - -#### Outputs - -- `SqlMapper.ICustomQueryParameter` objects created via `AsTableValuedParameter()`: - - `WorkOrderFilterParameter`: WorkOrderNumber column - - `ItemNumberFilterParameter`: ItemNumber column - - `ProfitCenterFilterParameter`: Code column - - `WorkCenterFilterParameter`: Code column - - `ComponentLotFilterParameter`: ComponentLotNumber + ItemNumber columns - - `OperatorFilterParameter`: UserName column - - `ItemOperationMisFilterParameter`: ItemNumber + OperationNumber + MisNumber + MisRevision columns - #### Business Rules -- Each parameter type MUST match the corresponding SQL Server table type schema -- DataTable column types MUST match the expected SQL types (long, string) -- Parameters are created even for empty filter collections (empty DataTable) -- Operator filter uses UserID (not AddressNumber) as the parameter value +- Search queries SHALL accept only `searchId` as the filter parameter +- SQL extraction functions handle JSON parsing and filter population +- TVP infrastructure for search execution has been removed +- TVPs remain in use for reference data lookups (LookupItemsAsync, etc.) - All database operations SHOULD use async methods (`QueryAsync`, `ExecuteAsync`) -#### Scenario: Create work order filter parameter +#### Scenario: Execute search with simplified parameter -- **WHEN** SearchModel contains WorkOrderFilter with entries [12345, 67890] -- **THEN** the system creates a DataTable with WorkOrderNumber column -- **AND** populates two rows with the work order numbers -- **AND** returns parameter of type "WorkOrderFilterParameter" - -#### Scenario: Create component lot filter parameter - -- **WHEN** SearchModel contains ComponentLotFilter with lot "LOT-A" for item "ITEM-001" -- **THEN** the system creates a DataTable with ComponentLotNumber and ItemNumber columns -- **AND** populates one row with both values -- **AND** returns parameter of type "ComponentLotFilterParameter" - -#### Scenario: Create empty filter parameter - -- **WHEN** SearchModel contains empty WorkOrderFilter collection -- **THEN** the system creates a DataTable with WorkOrderNumber column -- **AND** the DataTable has zero rows -- **AND** returns valid parameter that produces empty result set - -#### Scenario: Create item operation MIS filter parameter - -- **WHEN** SearchModel contains ItemOperationMisFilter with one entry -- **THEN** the system creates a DataTable with four columns -- **AND** populates ItemNumber, OperationNumber, MisNumber, MisRevision -- **AND** returns parameter of type "ItemOperationMisFilterParameter" +- **WHEN** search ID 123 is processed +- **THEN** query builder generates SQL with `@SearchId` parameter only +- **AND** extraction functions populate temporary filter tables from Search.Criteria JSON +- **AND** no DataTable or TVP creation is needed for search execution ---