dohertj2/jdescopingtool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update documentation for extraction functions migration

Browse Source 
- Add ExtractionFunctions.md reference document
- Update database-schema spec with 11 extraction functions
- Update data-access spec to document extraction function approach
- Update search-processing spec with new query builder interface
- Add Database.Tests to Testing.md architecture doc
- Update DataFlow.md with extraction function flow
This commit is contained in:
Joseph Doherty
2026-01-06 14:54:10 -05:00
parent 35c1e6baf0
commit c6aeb20d9c
8 changed files with 602 additions and 345 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DOCUMENTATION/Architecture/DataFlow.md
+2
View File
@@ -19,6 +19,8 @@ The search flow mirrors the legacy pattern, modernized for ASP.NET Core:
4. SearchProcessorService (BackgroundService) polls
└─> Finds queued searches
└─> Passes SearchId to query builder
└─> SQL extraction functions read criteria from Search.Criteria JSON
└─> Executes query against local cache
└─> Generates Excel via ClosedXML
└─> Stores result in Search.Results (VARBINARY)
DOCUMENTATION/Architecture/Database.md
+33 -1
View File
@@ -121,7 +121,7 @@ The scoping tool cache database includes these primary tables:
| Table | Purpose |
|-------|---------|
| `Search` | User search requests, status, and results (Excel as VARBINARY) |
| `Search` | User search requests, status, criteria (JSON), and results (Excel as VARBINARY) |
| `DataUpdate` | Tracks last sync timestamp per data type |
| `WorkOrder_Curr` | Current work orders from JDE |
| `WorkOrder_Hist` | Historical work orders from JDE |
@@ -134,6 +134,38 @@ The scoping tool cache database includes these primary tables:
| `ProfitCenter` | Profit center reference data |
| `SchemaVersions` | DbUp tracking table (auto-created) |
## Search Criteria Extraction Functions
The database includes SQL functions that extract filter criteria from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, with all filter extraction happening in SQL Server.
### Scalar Functions (3)
| Function | Returns | Extracts |
|----------|---------|----------|
| `fn_GetSearchMinimumDt` | DATETIME2 | `$.MinimumDt` |
| `fn_GetSearchMaximumDt` | DATETIME2 | `$.MaximumDt` |
| `fn_GetSearchExtractMisData` | BIT | `$.ExtractMisData` |
### Table-Valued Functions (8)
| Function | Returns | Extracts |
|----------|---------|----------|
| `fn_GetSearchWorkOrders` | WorkOrderNumber | `$.WorkOrderNumbers` array |
| `fn_GetSearchItemNumbers` | ItemNumber | `$.ItemNumbers` array |
| `fn_GetSearchProfitCenters` | Code | `$.ProfitCenters` array |
| `fn_GetSearchWorkCenters` | Code | `$.WorkCenters` array |
| `fn_GetSearchOperatorIDs` | OperatorID | `$.OperatorIDs` array |
| `fn_GetSearchComponentLots` | LotNumber, ItemNumber | `$.ComponentLotNumbers` array |
| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision | `$.PartOperations` array |
### Validation Procedure
| Procedure | Purpose |
|-----------|---------|
| `usp_ValidateSearchCriteria` | Validates search exists and has valid JSON criteria |
Error codes: 50001 (not found), 50002 (no criteria), 50003 (invalid JSON)
## Example Migration Scripts
### 001_CreateSearchTable.sql
DOCUMENTATION/Architecture/Testing.md
+85 -16
View File
@@ -5,22 +5,34 @@ The test project uses xUnit for the framework, Shouldly for assertions, and NSub
## Project Structure
```
JdeScoping.Tests/
├── Unit/
│ ├── Services/
│ │ ├── SearchServiceTests.cs
│ │ ├── ExcelExportServiceTests.cs
│ │ └── DataSyncOrchestratorTests.cs
│ ├── Repositories/
│ │ └── SearchRepositoryTests.cs
│ └── Models/
│ └── SearchCriteriaTests.cs
└── Integration/
├── ApiTests/
│ ├── SearchControllerTests.cs
│ └── LookupControllerTests.cs
└── RepositoryTests/
└── JdeRepositoryTests.cs
tests/
├── JdeScoping.Tests/
│ ├── Unit/
│ │ ├── Services/
│ │ │ ├── SearchServiceTests.cs
│ │ │ ├── ExcelExportServiceTests.cs
│ │ │ └── DataSyncOrchestratorTests.cs
│ │ ├── Repositories/
│ │ │ └── SearchRepositoryTests.cs
│ │ └── Models/
│ │ └── SearchCriteriaTests.cs
│ └── Integration/
│ ├── ApiTests/
│ │ ├── SearchControllerTests.cs
│ │ └── LookupControllerTests.cs
│ └── RepositoryTests/
│ └── JdeRepositoryTests.cs
├── JdeScoping.Api.IntegrationTests/
│ └── ... (API integration tests)
└── JdeScoping.Database.Tests/
├── Infrastructure/
│ └── DatabaseTestBase.cs
├── Functions/
│ ├── ScalarFunctionTests.cs
│ ├── SimpleTableFunctionTests.cs
│ └── ComplexTableFunctionTests.cs
└── Procedures/
└── ValidateSearchCriteriaProcedureTests.cs
```
## Unit Tests
@@ -175,7 +187,64 @@ public class SearchRepositoryIntegrationTests : IDisposable
}
```
## Database Tests
The `JdeScoping.Database.Tests` project tests SQL functions and stored procedures against a live SQL Server instance.
### Test Infrastructure
Tests inherit from `DatabaseTestBase` which provides:
- Connection management via `DatabaseTestFixture`
- Automatic test data cleanup
- Helper methods for inserting test searches
```csharp
[Collection("DatabaseTests")]
public class ScalarFunctionTests : DatabaseTestBase
{
[Fact]
public async Task fn_GetSearchMinimumDt_ValidSearch_ReturnsDateTime()
{
// Arrange
var criteria = new SearchCriteria { MinimumDt = new DateTime(2024, 6, 15) };
var searchId = await InsertTestSearchAsync(criteria);
// Act
var result = await Connection.QuerySingleOrDefaultAsync<DateTime?>(
"SELECT dbo.fn_GetSearchMinimumDt(@SearchId)",
new { SearchId = searchId });
// Assert
result.Should().BeCloseTo(criteria.MinimumDt.Value, TimeSpan.FromSeconds(1));
}
}
```
### Test Categories
| Category | Tests | Description |
|----------|-------|-------------|
| Scalar Functions | 15 | Test `fn_GetSearchMinimumDt`, `fn_GetSearchMaximumDt`, `fn_GetSearchExtractMisData` |
| Simple Table Functions | 38 | Test array extraction for work orders, items, profit centers, work centers, operators |
| Complex Table Functions | 23 | Test object extraction for component lots and part operations |
| Validation Procedure | 6 | Test `usp_ValidateSearchCriteria` error handling |
**Total: 82 database tests**
### Running Database Tests
```bash
# Run all database tests
dotnet test tests/JdeScoping.Database.Tests
# Run specific test class
dotnet test tests/JdeScoping.Database.Tests --filter "FullyQualifiedName~ScalarFunctionTests"
```
**Prerequisites**: SQL Server must be running on localhost:1434 with the ScopingTool database.
## Related Documentation
- [Core Project](./CoreProject.md)
- [Dependencies](./Dependencies.md)
- [Database](./Database.md)
DOCUMENTATION/Database/ExtractionFunctions.md
+237
View File
@@ -0,0 +1,237 @@
# Search Criteria Extraction Functions
SQL functions that extract filter values from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, simplifying the C# to SQL interface.
## Overview
| Type | Count | Purpose |
|------|-------|---------|
| Scalar Functions | 3 | Extract single values (dates, booleans) |
| Table-Valued Functions | 8 | Extract arrays and object collections |
| Validation Procedure | 1 | Pre-flight validation with error codes |
## Scalar Functions
### fn_GetSearchMinimumDt
Extracts the minimum date filter value.
```sql
CREATE FUNCTION dbo.fn_GetSearchMinimumDt(@SearchId INT)
RETURNS DATETIME2(7)
```
**JSON Path**: `$.MinimumDt`
**Returns**: DATETIME2 value or NULL if not found/invalid
### fn_GetSearchMaximumDt
Extracts the maximum date filter value.
```sql
CREATE FUNCTION dbo.fn_GetSearchMaximumDt(@SearchId INT)
RETURNS DATETIME2(7)
```
**JSON Path**: `$.MaximumDt`
**Returns**: DATETIME2 value or NULL if not found/invalid
### fn_GetSearchExtractMisData
Extracts the MIS data extraction flag.
```sql
CREATE FUNCTION dbo.fn_GetSearchExtractMisData(@SearchId INT)
RETURNS BIT
```
**JSON Path**: `$.ExtractMisData`
**Returns**: 1 (true), 0 (false), or NULL if not found/invalid
## Table-Valued Functions (Simple Arrays)
### fn_GetSearchWorkOrders
Extracts work order number filter values.
```sql
CREATE FUNCTION dbo.fn_GetSearchWorkOrders(@SearchId INT)
RETURNS TABLE
AS RETURN (
SELECT j.WorkOrderNumber
FROM ... OPENJSON(..., '$.WorkOrderNumbers') WITH (WorkOrderNumber BIGINT '$') j
)
```
**JSON Path**: `$.WorkOrderNumbers`
**Returns**: Table with `WorkOrderNumber BIGINT` column
### fn_GetSearchItemNumbers
Extracts item number filter values.
```sql
CREATE FUNCTION dbo.fn_GetSearchItemNumbers(@SearchId INT)
RETURNS TABLE
```
**JSON Path**: `$.ItemNumbers`
**Returns**: Table with `ItemNumber VARCHAR(128)` column
### fn_GetSearchProfitCenters
Extracts profit center filter values.
```sql
CREATE FUNCTION dbo.fn_GetSearchProfitCenters(@SearchId INT)
RETURNS TABLE
```
**JSON Path**: `$.ProfitCenters`
**Returns**: Table with `Code VARCHAR(12)` column
### fn_GetSearchWorkCenters
Extracts work center filter values.
```sql
CREATE FUNCTION dbo.fn_GetSearchWorkCenters(@SearchId INT)
RETURNS TABLE
```
**JSON Path**: `$.WorkCenters`
**Returns**: Table with `Code VARCHAR(12)` column
### fn_GetSearchOperatorIDs
Extracts operator ID filter values.
```sql
CREATE FUNCTION dbo.fn_GetSearchOperatorIDs(@SearchId INT)
RETURNS TABLE
```
**JSON Path**: `$.OperatorIDs`
**Returns**: Table with `OperatorID VARCHAR(128)` column
## Table-Valued Functions (Complex Objects)
### fn_GetSearchComponentLots
Extracts component lot filter values (lot/item pairs).
```sql
CREATE FUNCTION dbo.fn_GetSearchComponentLots(@SearchId INT)
RETURNS TABLE
AS RETURN (
SELECT j.LotNumber, j.ItemNumber
FROM ... OPENJSON(..., '$.ComponentLotNumbers') WITH (
LotNumber VARCHAR(30) '$.LotNumber',
ItemNumber VARCHAR(128) '$.ItemNumber'
) j
)
```
**JSON Path**: `$.ComponentLotNumbers`
**Returns**: Table with `LotNumber VARCHAR(30)`, `ItemNumber VARCHAR(128)` columns
### fn_GetSearchPartOperations
Extracts part operation filter values (item/operation/MIS combinations).
```sql
CREATE FUNCTION dbo.fn_GetSearchPartOperations(@SearchId INT)
RETURNS TABLE
AS RETURN (
SELECT j.ItemNumber, j.OperationNumber, j.MisNumber, j.MisRevision
FROM ... OPENJSON(..., '$.PartOperations') WITH (
ItemNumber VARCHAR(128) '$.ItemNumber',
OperationNumber VARCHAR(10) '$.OperationNumber',
MisNumber VARCHAR(10) '$.MisNumber',
MisRevision VARCHAR(10) '$.MisRevision'
) j
)
```
**JSON Path**: `$.PartOperations`
**Returns**: Table with `ItemNumber`, `OperationNumber`, `MisNumber`, `MisRevision` columns
## Validation Procedure
### usp_ValidateSearchCriteria
Validates that a search exists and has valid JSON criteria.
```sql
CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT)
```
**Error Codes**:
| Code | Condition | Message |
|------|-----------|---------|
| 50001 | Search not found | "Search ID {id} not found" |
| 50002 | Criteria NULL or empty | "Search ID {id} has no criteria" |
| 50003 | Invalid JSON | "Search ID {id} has invalid JSON" |
**Usage**:
```sql
-- Throws on validation failure
EXEC dbo.usp_ValidateSearchCriteria @SearchId = 123;
```
## Error Handling
All extraction functions handle errors gracefully:
- **Scalar functions**: Return NULL for missing search, NULL criteria, or invalid JSON
- **Table-valued functions**: Return empty result set for missing search, NULL criteria, or invalid JSON
- **Validation procedure**: Throws errors via `THROW` statement for calling code to handle
## Design Pattern
Functions use a CTE pattern to pre-filter valid JSON before calling OPENJSON:
```sql
WITH ValidSearch AS (
SELECT Criteria
FROM dbo.Search
WHERE ID = @SearchId
AND Criteria IS NOT NULL
AND ISJSON(Criteria) = 1
)
SELECT ...
FROM ValidSearch s
CROSS APPLY OPENJSON(s.Criteria, '$.ArrayPath') ...
```
This pattern prevents OPENJSON from running on invalid JSON, which would cause runtime errors.
## Migration Scripts
| Script | Contents |
|--------|----------|
| `045_CreateScalarExtractionFunctions.sql` | Scalar functions |
| `046_CreateSimpleTableFunctions.sql` | Simple array TVFs |
| `047_CreateComplexTableFunctions.sql` | Complex object TVFs |
| `048_CreateValidateSearchCriteriaProcedure.sql` | Validation procedure |
## Testing
82 tests in `JdeScoping.Database.Tests` verify function behavior:
```bash
dotnet test tests/JdeScoping.Database.Tests
```
See [Testing](../Architecture/Testing.md) for details.
Tools/CacheConverter/Program.cs
+61 -121
View File
@@ -1,5 +1,4 @@
using System.Data;
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
using ProtoBuf.Data;
@@ -102,55 +101,32 @@ foreach (var jsonFile in jsonFiles)
}
// Stream JSON and write to protobuf in batches
using var inputFs = new FileStream(jsonFile, FileMode.Open, FileAccess.Read, FileShare.Read, 256 * 1024, FileOptions.SequentialScan);
using var decompressStream = new DecompressionStream(inputFs);
using var outputFs = new FileStream(outputFile, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024);
using var compressStream = new CompressionStream(outputFs, level: 3);
await using var inputFs = new FileStream(jsonFile, FileMode.Open, FileAccess.Read, FileShare.Read, 256 * 1024, FileOptions.SequentialScan | FileOptions.Asynchronous);
await using var decompressStream = new DecompressionStream(inputFs);
await using var outputFs = new FileStream(outputFile, FileMode.Create, FileAccess.Write, FileShare.None, 256 * 1024, FileOptions.Asynchronous);
await using var compressStream = new CompressionStream(outputFs, level: 3);
int rowCount = 0;
int batchCount = 0;
// Stream JSON records one at a time
var buffer = new byte[4096];
using var memoryStream = new MemoryStream();
int bytesRead;
while ((bytesRead = decompressStream.Read(buffer, 0, buffer.Length)) > 0)
// True streaming: DeserializeAsyncEnumerable streams each array element without loading entire JSON
var jsonOptions = new JsonSerializerOptions { PropertyNameCaseInsensitive = true };
await foreach (var element in JsonSerializer.DeserializeAsyncEnumerable<JsonElement>(
decompressStream,
jsonOptions))
{
memoryStream.Write(buffer, 0, bytesRead);
}
var row = dataTable.NewRow();
ReadJsonElement(element, row, dataTable);
dataTable.Rows.Add(row);
rowCount++;
memoryStream.Position = 0;
var jsonReader = new Utf8JsonReader(memoryStream.ToArray(), new JsonReaderOptions { AllowTrailingCommas = true });
// Skip to start of array
while (jsonReader.Read())
{
if (jsonReader.TokenType == JsonTokenType.StartArray)
break;
}
// Read each object in the array
while (jsonReader.Read())
{
if (jsonReader.TokenType == JsonTokenType.EndArray)
break;
if (jsonReader.TokenType == JsonTokenType.StartObject)
// Write batch when we hit the batch size
if (dataTable.Rows.Count >= BatchSize)
{
var row = dataTable.NewRow();
ReadJsonObject(ref jsonReader, row, dataTable);
dataTable.Rows.Add(row);
rowCount++;
// Write batch when we hit the batch size
if (dataTable.Rows.Count >= BatchSize)
{
using var reader = dataTable.CreateDataReader();
DataSerializer.Serialize(compressStream, reader);
dataTable.Clear();
batchCount++;
}
using var reader = dataTable.CreateDataReader();
DataSerializer.Serialize(compressStream, reader);
dataTable.Clear();
batchCount++;
}
}
@@ -162,7 +138,7 @@ foreach (var jsonFile in jsonFiles)
batchCount++;
}
compressStream.Flush();
await compressStream.FlushAsync();
var newSize = new FileInfo(outputFile).Length;
totalNewSize += newSize;
@@ -243,68 +219,58 @@ static Type MapSqlTypeToNet(string sqlType) => sqlType.ToUpperInvariant() switch
};
/// <summary>
/// Read a JSON object into a DataRow using streaming reader.
/// Read a JsonElement (object) into a DataRow.
/// </summary>
static void ReadJsonObject(ref Utf8JsonReader reader, DataRow row, DataTable table)
static void ReadJsonElement(JsonElement element, DataRow row, DataTable table)
{
while (reader.Read())
foreach (var property in element.EnumerateObject())
{
if (reader.TokenType == JsonTokenType.EndObject)
break;
if (reader.TokenType == JsonTokenType.PropertyName)
// Find matching column (case-insensitive)
DataColumn? column = null;
foreach (DataColumn col in table.Columns)
{
var propertyName = reader.GetString()!;
reader.Read(); // Move to value
// Find matching column (case-insensitive)
DataColumn? column = null;
foreach (DataColumn col in table.Columns)
if (col.ColumnName.Equals(property.Name, StringComparison.OrdinalIgnoreCase))
{
if (col.ColumnName.Equals(propertyName, StringComparison.OrdinalIgnoreCase))
{
column = col;
break;
}
column = col;
break;
}
if (column == null)
{
// Skip unknown property
SkipJsonValue(ref reader);
continue;
}
row[column] = ReadJsonValue(ref reader, column.DataType);
}
if (column == null)
{
// Skip unknown property
continue;
}
row[column] = ReadJsonElementValue(property.Value, column.DataType);
}
}
/// <summary>
/// Read a JSON value and convert to the target .NET type.
/// Read a JSON value from JsonElement and convert to the target .NET type.
/// </summary>
static object ReadJsonValue(ref Utf8JsonReader reader, Type targetType)
static object ReadJsonElementValue(JsonElement element, Type targetType)
{
if (reader.TokenType == JsonTokenType.Null)
if (element.ValueKind == JsonValueKind.Null || element.ValueKind == JsonValueKind.Undefined)
return DBNull.Value;
if (targetType == typeof(string))
{
return reader.TokenType switch
return element.ValueKind switch
{
JsonTokenType.String => reader.GetString() ?? (object)DBNull.Value,
JsonTokenType.Number => reader.GetDecimal().ToString(),
JsonTokenType.True => "true",
JsonTokenType.False => "false",
JsonValueKind.String => element.GetString() ?? (object)DBNull.Value,
JsonValueKind.Number => element.GetDecimal().ToString(),
JsonValueKind.True => "true",
JsonValueKind.False => "false",
_ => DBNull.Value
};
}
if (targetType == typeof(DateTime))
{
if (reader.TokenType == JsonTokenType.String)
if (element.ValueKind == JsonValueKind.String)
{
var str = reader.GetString();
var str = element.GetString();
if (str != null && DateTime.TryParse(str, out var dt))
return dt;
}
@@ -313,71 +279,45 @@ static object ReadJsonValue(ref Utf8JsonReader reader, Type targetType)
if (targetType == typeof(long))
{
return reader.TokenType switch
return element.ValueKind switch
{
JsonTokenType.Number => reader.GetInt64(),
JsonTokenType.String when long.TryParse(reader.GetString(), out var val) => val,
JsonValueKind.Number => element.GetInt64(),
JsonValueKind.String when long.TryParse(element.GetString(), out var val) => val,
_ => DBNull.Value
};
}
if (targetType == typeof(int))
{
return reader.TokenType switch
return element.ValueKind switch
{
JsonTokenType.Number => reader.GetInt32(),
JsonTokenType.String when int.TryParse(reader.GetString(), out var val) => val,
JsonValueKind.Number => element.GetInt32(),
JsonValueKind.String when int.TryParse(element.GetString(), out var val) => val,
_ => DBNull.Value
};
}
if (targetType == typeof(decimal))
{
return reader.TokenType switch
return element.ValueKind switch
{
JsonTokenType.Number => reader.GetDecimal(),
JsonTokenType.String when decimal.TryParse(reader.GetString(), out var val) => val,
JsonValueKind.Number => element.GetDecimal(),
JsonValueKind.String when decimal.TryParse(element.GetString(), out var val) => val,
_ => DBNull.Value
};
}
if (targetType == typeof(bool))
{
return reader.TokenType switch
return element.ValueKind switch
{
JsonTokenType.True => true,
JsonTokenType.False => false,
JsonTokenType.Number => reader.GetInt32() != 0,
JsonTokenType.String => reader.GetString()?.Equals("true", StringComparison.OrdinalIgnoreCase) ?? false,
JsonValueKind.True => true,
JsonValueKind.False => false,
JsonValueKind.Number => element.GetInt32() != 0,
JsonValueKind.String => element.GetString()?.Equals("true", StringComparison.OrdinalIgnoreCase) ?? false,
_ => DBNull.Value
};
}
return DBNull.Value;
}
/// <summary>
/// Skip a JSON value (used for unknown properties).
/// </summary>
static void SkipJsonValue(ref Utf8JsonReader reader)
{
if (reader.TokenType == JsonTokenType.StartObject)
{
int depth = 1;
while (depth > 0 && reader.Read())
{
if (reader.TokenType == JsonTokenType.StartObject) depth++;
else if (reader.TokenType == JsonTokenType.EndObject) depth--;
}
}
else if (reader.TokenType == JsonTokenType.StartArray)
{
int depth = 1;
while (depth > 0 && reader.Read())
{
if (reader.TokenType == JsonTokenType.StartArray) depth++;
else if (reader.TokenType == JsonTokenType.EndArray) depth--;
}
}
// Simple values are already consumed by the Read() call
}
openspec/specs/data-access/spec.md
+46 -21
View File
@@ -467,40 +467,65 @@ catch (OracleException ex)
---
### Requirement: Table-valued parameter support
### Requirement: Search criteria extraction via SQL functions
The system SHALL use DataTable for SQL Server table-valued parameters in lookup methods.
The system SHALL use SQL extraction functions to retrieve filter criteria directly from the `Search.Criteria` JSON column.
#### Implementation Pattern
Search query building now uses SearchId to invoke extraction functions rather than passing filter values from C#:
```csharp
public async Task<List<Item>> LookupItemsAsync(List<string> itemNumbers, CancellationToken ct = default)
public SearchQueryResult BuildSearchQuery(int searchId)
{
await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
// Query builder generates SQL that calls extraction functions
// Example generated SQL fragment:
// INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
// INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
var table = new DataTable();
table.Columns.Add("ItemNumber", typeof(string));
foreach (var itemNumber in itemNumbers)
{
table.Rows.Add(itemNumber);
}
var parameters = new DynamicParameters();
parameters.Add("@itemNumbers", table.AsTableValuedParameter("dbo.ItemNumberFilterParameter"));
return (await connection.QueryAsync<Item>(
LotFinderQueries.SQL_LOOKUP_ITEMS,
parameters,
commandTimeout: _options.Value.DefaultTimeoutSeconds))
.AsList();
return new SearchQueryResult(sql, new { SearchId = searchId });
}
```
#### Business Rules
- TVPs SHALL use `AsTableValuedParameter` extension with correct type name
- Query builder SHALL accept only `searchId` parameter (not full criteria object)
- SQL queries SHALL call extraction functions to populate temporary filter tables
- Extraction functions handle JSON parsing and validation in SQL Server
- Invalid JSON or missing criteria results in empty filter sets (no errors thrown)
#### Available Extraction Functions
| Function | Returns |
|----------|---------|
| `fn_GetSearchMinimumDt` | DATETIME2 scalar |
| `fn_GetSearchMaximumDt` | DATETIME2 scalar |
| `fn_GetSearchExtractMisData` | BIT scalar |
| `fn_GetSearchWorkOrders` | WorkOrderNumber table |
| `fn_GetSearchItemNumbers` | ItemNumber table |
| `fn_GetSearchProfitCenters` | Code table |
| `fn_GetSearchWorkCenters` | Code table |
| `fn_GetSearchOperatorIDs` | OperatorID table |
| `fn_GetSearchComponentLots` | LotNumber, ItemNumber table |
| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision table |
#### Scenario: Build search query with extraction functions
- **WHEN** `BuildSearchQuery(123)` is called for a search with work order filter
- **THEN** generated SQL includes `SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(123)`
- **AND** only the `@SearchId` parameter is passed to the query
---
### Requirement: Table-valued parameter support for lookups
The system SHALL use DataTable for SQL Server table-valued parameters in reference data lookup methods.
#### Business Rules
- TVPs are used for batch lookups (LookupItemsAsync, LookupWorkordersAsync, etc.)
- TVPs are NOT used for search query execution (replaced by extraction functions)
- DataTable column names SHALL match TVP type column names
- TVPs enable efficient batch lookups with single database round-trip
#### Scenario: Batch lookup with TVP
openspec/specs/database-schema/spec.md
+84
View File
@@ -1079,6 +1079,90 @@ var results = await context.WorkOrders
## Functions
### Requirement: Search Criteria Extraction Functions
The system SHALL provide SQL functions to extract search criteria values from the `Search.Criteria` JSON column.
#### Scalar Functions
| Function | Return Type | Description |
|----------|-------------|-------------|
| `fn_GetSearchMinimumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MinimumDt` value |
| `fn_GetSearchMaximumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MaximumDt` value |
| `fn_GetSearchExtractMisData(@SearchId INT)` | BIT | Extracts `$.ExtractMisData` value |
#### Table-Valued Functions (Simple Arrays)
| Function | Return Columns | Description |
|----------|----------------|-------------|
| `fn_GetSearchWorkOrders(@SearchId INT)` | WorkOrderNumber BIGINT | Extracts `$.WorkOrderNumbers` array |
| `fn_GetSearchItemNumbers(@SearchId INT)` | ItemNumber VARCHAR(128) | Extracts `$.ItemNumbers` array |
| `fn_GetSearchProfitCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.ProfitCenters` array |
| `fn_GetSearchWorkCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.WorkCenters` array |
| `fn_GetSearchOperatorIDs(@SearchId INT)` | OperatorID VARCHAR(128) | Extracts `$.OperatorIDs` array |
#### Table-Valued Functions (Complex Objects)
| Function | Return Columns | Description |
|----------|----------------|-------------|
| `fn_GetSearchComponentLots(@SearchId INT)` | LotNumber VARCHAR(30), ItemNumber VARCHAR(128) | Extracts `$.ComponentLotNumbers` array |
| `fn_GetSearchPartOperations(@SearchId INT)` | ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10) | Extracts `$.PartOperations` array |
#### Business Rules
- Scalar functions return NULL if search not found, criteria is NULL, or JSON is invalid
- Table-valued functions return empty result set if search not found, criteria is NULL, or JSON is invalid
- TVFs use CTE pattern to pre-filter valid JSON before OPENJSON (prevents runtime errors)
- TVFs use `OPENJSON...WITH` syntax for type-safe extraction
#### Scenario: Extract work order filter values
- **WHEN** Search ID 123 has Criteria containing `{"WorkOrderNumbers":[12345,67890]}`
- **THEN** `SELECT * FROM dbo.fn_GetSearchWorkOrders(123)` returns two rows with WorkOrderNumber values 12345 and 67890
#### Scenario: Handle invalid JSON gracefully
- **WHEN** Search ID 456 has Criteria containing invalid JSON text
- **THEN** extraction functions return NULL (scalar) or empty result set (TVF) without throwing errors
---
### Requirement: Validate Search Criteria Procedure
The system SHALL provide a stored procedure for strict validation of search criteria with error reporting.
#### Procedure Signature
```sql
CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT)
```
#### Error Codes
| Error Code | Condition | Message Pattern |
|------------|-----------|-----------------|
| 50001 | Search ID not found | "Search ID {id} not found" |
| 50002 | Criteria is NULL or empty | "Search ID {id} has no criteria" |
| 50003 | Criteria is not valid JSON | "Search ID {id} has invalid JSON" |
#### Business Rules
- Procedure throws errors using `THROW` statement for calling code to handle
- Procedure returns 0 (success) when validation passes
- Used for pre-flight validation before query execution
#### Scenario: Validate valid search
- **WHEN** `usp_ValidateSearchCriteria` is called for a search with valid JSON criteria
- **THEN** procedure returns 0 (success) without throwing
#### Scenario: Validate missing search
- **WHEN** `usp_ValidateSearchCriteria` is called for non-existent search ID 99999
- **THEN** procedure throws error 50001 with message "Search ID 99999 not found"
---
### Requirement: MatchMIS function
The system SHALL provide a table-valued function to match work order steps to MIS data.
openspec/specs/search-processing/spec.md
+54 -186
View File
@@ -86,7 +86,7 @@ The legacy system uses a T4 text template (`QueryTemplate.tt`) for SQL generatio
**NuGet Package:** `SqlKata` and `SqlKata.Execution`
**Primary Query Building with SqlKata:**
**Primary Query Building with SqlKata and Extraction Functions:**
```csharp
using SqlKata;
@@ -94,7 +94,9 @@ using SqlKata.Compilers;
public interface ISearchQueryBuilder
{
SearchQueryResult BuildSearchQuery(SearchModel model);
SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);
}
public record SearchQueryResult(string Sql, IDictionary<string, object> Parameters);
@@ -103,127 +105,50 @@ public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder
{
private readonly SqlServerCompiler _compiler = new();
public SearchQueryResult BuildSearchQuery(SearchModel model)
public SearchQueryResult BuildSearchQuery(int searchId)
{
// Query uses SQL extraction functions to get criteria values
// No need to pass filter collections from C# - just the SearchId
// Generated SQL populates temp tables from extraction functions:
// INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
// INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
// etc.
var query = new Query("WorkOrder_Curr as wo")
.Select("wo.*", "wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode");
// Conditional joins based on active filters
if (model.OperatorFilterEnabled)
{
query.Join("WorkOrderTime_Curr as wot", "wot.WorkOrderNumber", "wo.WorkOrderNumber");
}
// Conditional joins based on temp table population (determined at runtime)
// Query builder generates SQL that checks IF EXISTS on temp tables
if (model.ProfitCenterFilterEnabled || model.WorkCenterFilterEnabled)
{
query.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber");
}
if (model.ComponentLotFilterEnabled)
{
query.Join("WorkOrderComponent_Curr as woc", "woc.WorkOrderNumber", "wo.WorkOrderNumber");
}
// Conditional WHERE clauses
if (model.TimespanFilterEnabled)
{
query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
}
if (model.WorkOrderFilterEnabled)
{
query.WhereIn("wo.WorkOrderNumber", model.WorkOrderFilter.Select(w => w.WorkOrderNumber));
}
if (model.ItemNumberFilterEnabled)
{
query.WhereIn("wo.ItemNumber", model.ItemNumberFilter.Select(i => i.ItemNumber));
}
if (model.OperatorFilterEnabled)
{
query.WhereIn("wot.OperatorAN", model.OperatorFilter.Select(o => o.AddressNumber));
}
// Compile to SQL + parameters
var compiled = _compiler.Compile(query);
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
return new SearchQueryResult(compiled.Sql, new Dictionary<string, object> { ["SearchId"] = searchId });
}
}
```
**Complex Query Composition with SqlKata:**
**Extraction Function Integration:**
```csharp
public sealed class ComposableSearchQueryBuilder
{
private readonly SqlServerCompiler _compiler = new();
The query builder generates SQL that populates temporary filter tables from extraction functions:
public SearchQueryResult BuildFullSearchQuery(SearchModel model)
{
// Build composable query parts
var baseQuery = BuildBaseWorkOrderQuery(model);
var filterQuery = ApplyFilters(baseQuery, model);
var joinedQuery = ApplyConditionalJoins(filterQuery, model);
```sql
-- Generated SQL fragment example:
-- Populate temp tables from extraction functions
INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId);
INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId);
INSERT INTO #P_ProfitCenters SELECT Code FROM dbo.fn_GetSearchProfitCenters(@SearchId);
INSERT INTO #P_WorkCenters SELECT Code FROM dbo.fn_GetSearchWorkCenters(@SearchId);
INSERT INTO #P_OperatorIDs SELECT OperatorID FROM dbo.fn_GetSearchOperatorIDs(@SearchId);
INSERT INTO #P_ComponentLots SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId);
INSERT INTO #P_PartOperations SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId);
var compiled = _compiler.Compile(joinedQuery);
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
}
private Query BuildBaseWorkOrderQuery(SearchModel model)
{
return new Query("WorkOrder_Curr as wo")
.Select("wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode",
"wo.StatusUpdateDT", "wo.QuantityOrdered", "wo.QuantityCompleted");
}
private Query ApplyFilters(Query query, SearchModel model)
{
if (model.TimespanFilterEnabled && model.MinimumDT.HasValue && model.MaximumDT.HasValue)
{
query = query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
}
if (model.WorkOrderFilterEnabled)
{
query = query.WhereIn("wo.WorkOrderNumber",
model.WorkOrderFilter.Select(w => w.WorkOrderNumber).ToList());
}
if (model.ItemNumberFilterEnabled)
{
query = query.WhereIn("wo.ItemNumber",
model.ItemNumberFilter.Select(i => i.ItemNumber).ToList());
}
return query;
}
private Query ApplyConditionalJoins(Query query, SearchModel model)
{
if (model.ProfitCenterFilterEnabled)
{
query = query
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
.Join("WorkCenter as wc", "wc.Code", "wos.WorkCenterCode")
.Join("OrgHierarchy as oh", "oh.WorkCenterCode", "wc.Code")
.WhereIn("oh.ProfitCenterCode",
model.ProfitCenterFilter.Select(p => p.Code).ToList());
}
if (model.WorkCenterFilterEnabled)
{
query = query
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
.WhereIn("wos.WorkCenterCode",
model.WorkCenterFilter.Select(w => w.Code).ToList());
}
return query;
}
}
-- Check which filters are active based on temp table population
DECLARE @HasWorkOrderFilter BIT = CASE WHEN EXISTS(SELECT 1 FROM #P_WorkOrders) THEN 1 ELSE 0 END;
-- etc.
```
This approach eliminates the need to pass filter collections from C# to SQL, simplifying the interface to a single `searchId` parameter.
**Work Order Traversal (Stored Procedure):**
The iterative downstream work order traversal is implemented as a stored procedure (`dbo.TraverseWorkOrders`) due to its iterative nature with temp tables and MERGE operations. The maximum iteration count is configurable via `SearchProcessingOptions.MaxTraversalIterations` (default: 20):
@@ -675,13 +600,11 @@ The system SHALL optionally extract Manufacturing Information System (MIS) data
---
### Requirement: Table-Valued Parameter Creation
### Requirement: Simplified Query Parameter Passing
The system SHALL create SQL table-valued parameters for efficient filter data transmission using Microsoft.Data.SqlClient.
The system SHALL pass only the SearchId parameter to search queries, with filter extraction handled by SQL functions.
#### Migration Note: SqlClient and TVP Options
**SQL Client Package Change (Required)**
#### Migration Note: SqlClient Package Change (Required)
Replace `System.Data.SqlClient` with `Microsoft.Data.SqlClient`:
@@ -693,92 +616,37 @@ using System.Data.SqlClient;
using Microsoft.Data.SqlClient;
```
This is a cross-cutting change affecting all database access code. The API is largely compatible but `Microsoft.Data.SqlClient` is the actively maintained package with security updates.
#### Simplified Parameter Model
**TVP Implementation Options**
The legacy approach using `DataTable` remains valid. An alternative using `IEnumerable<SqlDataRecord>` offers better performance for large datasets:
With extraction functions, the query builder interface is simplified:
```csharp
// Option 1: DataTable approach (legacy pattern, still supported)
public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter(
this SearchModel model)
public interface ISearchQueryBuilder
{
var dataTable = new DataTable();
dataTable.Columns.Add("WorkOrderNumber", typeof(long));
foreach (var entry in model.WorkOrderFilter)
{
dataTable.Rows.Add(entry.WorkOrderNumber);
}
return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);
}
// Option 2: SqlDataRecord approach (more efficient for large datasets)
public static IEnumerable<SqlDataRecord> ToWorkOrderRecords(
this IEnumerable<WorkOrderFilterEntry> entries)
{
var metadata = new SqlMetaData("WorkOrderNumber", SqlDbType.BigInt);
var record = new SqlDataRecord(metadata);
foreach (var entry in entries)
{
record.SetInt64(0, entry.WorkOrderNumber);
yield return record;
}
}
// Query execution only needs SearchId
var result = queryBuilder.BuildSearchQuery(searchId);
var results = await connection.QueryAsync<SearchResult>(result.Sql, new { SearchId = searchId });
```
#### Inputs
- `SearchModel` with populated filter entry collections
- SQL Server table type definitions for each parameter type
#### Outputs
- `SqlMapper.ICustomQueryParameter` objects created via `AsTableValuedParameter()`:
- `WorkOrderFilterParameter`: WorkOrderNumber column
- `ItemNumberFilterParameter`: ItemNumber column
- `ProfitCenterFilterParameter`: Code column
- `WorkCenterFilterParameter`: Code column
- `ComponentLotFilterParameter`: ComponentLotNumber + ItemNumber columns
- `OperatorFilterParameter`: UserName column
- `ItemOperationMisFilterParameter`: ItemNumber + OperationNumber + MisNumber + MisRevision columns
#### Business Rules
- Each parameter type MUST match the corresponding SQL Server table type schema
- DataTable column types MUST match the expected SQL types (long, string)
- Parameters are created even for empty filter collections (empty DataTable)
- Operator filter uses UserID (not AddressNumber) as the parameter value
- Search queries SHALL accept only `searchId` as the filter parameter
- SQL extraction functions handle JSON parsing and filter population
- TVP infrastructure for search execution has been removed
- TVPs remain in use for reference data lookups (LookupItemsAsync, etc.)
- All database operations SHOULD use async methods (`QueryAsync`, `ExecuteAsync`)
#### Scenario: Create work order filter parameter
#### Scenario: Execute search with simplified parameter
- **WHEN** SearchModel contains WorkOrderFilter with entries [12345, 67890]
- **THEN** the system creates a DataTable with WorkOrderNumber column
- **AND** populates two rows with the work order numbers
- **AND** returns parameter of type "WorkOrderFilterParameter"
#### Scenario: Create component lot filter parameter
- **WHEN** SearchModel contains ComponentLotFilter with lot "LOT-A" for item "ITEM-001"
- **THEN** the system creates a DataTable with ComponentLotNumber and ItemNumber columns
- **AND** populates one row with both values
- **AND** returns parameter of type "ComponentLotFilterParameter"
#### Scenario: Create empty filter parameter
- **WHEN** SearchModel contains empty WorkOrderFilter collection
- **THEN** the system creates a DataTable with WorkOrderNumber column
- **AND** the DataTable has zero rows
- **AND** returns valid parameter that produces empty result set
#### Scenario: Create item operation MIS filter parameter
- **WHEN** SearchModel contains ItemOperationMisFilter with one entry
- **THEN** the system creates a DataTable with four columns
- **AND** populates ItemNumber, OperationNumber, MisNumber, MisRevision
- **AND** returns parameter of type "ItemOperationMisFilterParameter"
- **WHEN** search ID 123 is processed
- **THEN** query builder generates SQL with `@SearchId` parameter only
- **AND** extraction functions populate temporary filter tables from Search.Criteria JSON
- **AND** no DataTable or TVP creation is needed for search execution
---