dohertj2/jdescopingtool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update documentation for extraction functions migration

Browse Source 
- Add ExtractionFunctions.md reference document
- Update database-schema spec with 11 extraction functions
- Update data-access spec to document extraction function approach
- Update search-processing spec with new query builder interface
- Add Database.Tests to Testing.md architecture doc
- Update DataFlow.md with extraction function flow
This commit is contained in:
Joseph Doherty
2026-01-06 14:54:10 -05:00
parent 35c1e6baf0
commit c6aeb20d9c
8 changed files with 602 additions and 345 deletions
Download Patch File Download Diff File Expand all files Collapse all files
openspec/specs/data-access/spec.md
+46 -21
View File
@@ -467,40 +467,65 @@ catch (OracleException ex)
---
### Requirement: Table-valued parameter support
### Requirement: Search criteria extraction via SQL functions
The system SHALL use DataTable for SQL Server table-valued parameters in lookup methods.
The system SHALL use SQL extraction functions to retrieve filter criteria directly from the `Search.Criteria` JSON column.
#### Implementation Pattern
Search query building now uses SearchId to invoke extraction functions rather than passing filter values from C#:
```csharp
public async Task<List<Item>> LookupItemsAsync(List<string> itemNumbers, CancellationToken ct = default)
public SearchQueryResult BuildSearchQuery(int searchId)
{
await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
// Query builder generates SQL that calls extraction functions
// Example generated SQL fragment:
// INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
// INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
var table = new DataTable();
table.Columns.Add("ItemNumber", typeof(string));
foreach (var itemNumber in itemNumbers)
{
table.Rows.Add(itemNumber);
}
var parameters = new DynamicParameters();
parameters.Add("@itemNumbers", table.AsTableValuedParameter("dbo.ItemNumberFilterParameter"));
return (await connection.QueryAsync<Item>(
LotFinderQueries.SQL_LOOKUP_ITEMS,
parameters,
commandTimeout: _options.Value.DefaultTimeoutSeconds))
.AsList();
return new SearchQueryResult(sql, new { SearchId = searchId });
}
```
#### Business Rules
- TVPs SHALL use `AsTableValuedParameter` extension with correct type name
- Query builder SHALL accept only `searchId` parameter (not full criteria object)
- SQL queries SHALL call extraction functions to populate temporary filter tables
- Extraction functions handle JSON parsing and validation in SQL Server
- Invalid JSON or missing criteria results in empty filter sets (no errors thrown)
#### Available Extraction Functions
| Function | Returns |
|----------|---------|
| `fn_GetSearchMinimumDt` | DATETIME2 scalar |
| `fn_GetSearchMaximumDt` | DATETIME2 scalar |
| `fn_GetSearchExtractMisData` | BIT scalar |
| `fn_GetSearchWorkOrders` | WorkOrderNumber table |
| `fn_GetSearchItemNumbers` | ItemNumber table |
| `fn_GetSearchProfitCenters` | Code table |
| `fn_GetSearchWorkCenters` | Code table |
| `fn_GetSearchOperatorIDs` | OperatorID table |
| `fn_GetSearchComponentLots` | LotNumber, ItemNumber table |
| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision table |
#### Scenario: Build search query with extraction functions
- **WHEN** `BuildSearchQuery(123)` is called for a search with work order filter
- **THEN** generated SQL includes `SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(123)`
- **AND** only the `@SearchId` parameter is passed to the query
---
### Requirement: Table-valued parameter support for lookups
The system SHALL use DataTable for SQL Server table-valued parameters in reference data lookup methods.
#### Business Rules
- TVPs are used for batch lookups (LookupItemsAsync, LookupWorkordersAsync, etc.)
- TVPs are NOT used for search query execution (replaced by extraction functions)
- DataTable column names SHALL match TVP type column names
- TVPs enable efficient batch lookups with single database round-trip
#### Scenario: Batch lookup with TVP
openspec/specs/database-schema/spec.md
+84
View File
@@ -1079,6 +1079,90 @@ var results = await context.WorkOrders
## Functions
### Requirement: Search Criteria Extraction Functions
The system SHALL provide SQL functions to extract search criteria values from the `Search.Criteria` JSON column.
#### Scalar Functions
| Function | Return Type | Description |
|----------|-------------|-------------|
| `fn_GetSearchMinimumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MinimumDt` value |
| `fn_GetSearchMaximumDt(@SearchId INT)` | DATETIME2(7) | Extracts `$.MaximumDt` value |
| `fn_GetSearchExtractMisData(@SearchId INT)` | BIT | Extracts `$.ExtractMisData` value |
#### Table-Valued Functions (Simple Arrays)
| Function | Return Columns | Description |
|----------|----------------|-------------|
| `fn_GetSearchWorkOrders(@SearchId INT)` | WorkOrderNumber BIGINT | Extracts `$.WorkOrderNumbers` array |
| `fn_GetSearchItemNumbers(@SearchId INT)` | ItemNumber VARCHAR(128) | Extracts `$.ItemNumbers` array |
| `fn_GetSearchProfitCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.ProfitCenters` array |
| `fn_GetSearchWorkCenters(@SearchId INT)` | Code VARCHAR(12) | Extracts `$.WorkCenters` array |
| `fn_GetSearchOperatorIDs(@SearchId INT)` | OperatorID VARCHAR(128) | Extracts `$.OperatorIDs` array |
#### Table-Valued Functions (Complex Objects)
| Function | Return Columns | Description |
|----------|----------------|-------------|
| `fn_GetSearchComponentLots(@SearchId INT)` | LotNumber VARCHAR(30), ItemNumber VARCHAR(128) | Extracts `$.ComponentLotNumbers` array |
| `fn_GetSearchPartOperations(@SearchId INT)` | ItemNumber VARCHAR(128), OperationNumber VARCHAR(10), MisNumber VARCHAR(10), MisRevision VARCHAR(10) | Extracts `$.PartOperations` array |
#### Business Rules
- Scalar functions return NULL if search not found, criteria is NULL, or JSON is invalid
- Table-valued functions return empty result set if search not found, criteria is NULL, or JSON is invalid
- TVFs use CTE pattern to pre-filter valid JSON before OPENJSON (prevents runtime errors)
- TVFs use `OPENJSON...WITH` syntax for type-safe extraction
#### Scenario: Extract work order filter values
- **WHEN** Search ID 123 has Criteria containing `{"WorkOrderNumbers":[12345,67890]}`
- **THEN** `SELECT * FROM dbo.fn_GetSearchWorkOrders(123)` returns two rows with WorkOrderNumber values 12345 and 67890
#### Scenario: Handle invalid JSON gracefully
- **WHEN** Search ID 456 has Criteria containing invalid JSON text
- **THEN** extraction functions return NULL (scalar) or empty result set (TVF) without throwing errors
---
### Requirement: Validate Search Criteria Procedure
The system SHALL provide a stored procedure for strict validation of search criteria with error reporting.
#### Procedure Signature
```sql
CREATE PROCEDURE dbo.usp_ValidateSearchCriteria(@SearchId INT)
```
#### Error Codes
| Error Code | Condition | Message Pattern |
|------------|-----------|-----------------|
| 50001 | Search ID not found | "Search ID {id} not found" |
| 50002 | Criteria is NULL or empty | "Search ID {id} has no criteria" |
| 50003 | Criteria is not valid JSON | "Search ID {id} has invalid JSON" |
#### Business Rules
- Procedure throws errors using `THROW` statement for calling code to handle
- Procedure returns 0 (success) when validation passes
- Used for pre-flight validation before query execution
#### Scenario: Validate valid search
- **WHEN** `usp_ValidateSearchCriteria` is called for a search with valid JSON criteria
- **THEN** procedure returns 0 (success) without throwing
#### Scenario: Validate missing search
- **WHEN** `usp_ValidateSearchCriteria` is called for non-existent search ID 99999
- **THEN** procedure throws error 50001 with message "Search ID 99999 not found"
---
### Requirement: MatchMIS function
The system SHALL provide a table-valued function to match work order steps to MIS data.
openspec/specs/search-processing/spec.md
+54 -186
View File
@@ -86,7 +86,7 @@ The legacy system uses a T4 text template (`QueryTemplate.tt`) for SQL generatio
**NuGet Package:** `SqlKata` and `SqlKata.Execution`
**Primary Query Building with SqlKata:**
**Primary Query Building with SqlKata and Extraction Functions:**
```csharp
using SqlKata;
@@ -94,7 +94,9 @@ using SqlKata.Compilers;
public interface ISearchQueryBuilder
{
SearchQueryResult BuildSearchQuery(SearchModel model);
SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);
}
public record SearchQueryResult(string Sql, IDictionary<string, object> Parameters);
@@ -103,127 +105,50 @@ public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder
{
private readonly SqlServerCompiler _compiler = new();
public SearchQueryResult BuildSearchQuery(SearchModel model)
public SearchQueryResult BuildSearchQuery(int searchId)
{
// Query uses SQL extraction functions to get criteria values
// No need to pass filter collections from C# - just the SearchId
// Generated SQL populates temp tables from extraction functions:
// INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
// INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
// etc.
var query = new Query("WorkOrder_Curr as wo")
.Select("wo.*", "wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode");
// Conditional joins based on active filters
if (model.OperatorFilterEnabled)
{
query.Join("WorkOrderTime_Curr as wot", "wot.WorkOrderNumber", "wo.WorkOrderNumber");
}
// Conditional joins based on temp table population (determined at runtime)
// Query builder generates SQL that checks IF EXISTS on temp tables
if (model.ProfitCenterFilterEnabled || model.WorkCenterFilterEnabled)
{
query.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber");
}
if (model.ComponentLotFilterEnabled)
{
query.Join("WorkOrderComponent_Curr as woc", "woc.WorkOrderNumber", "wo.WorkOrderNumber");
}
// Conditional WHERE clauses
if (model.TimespanFilterEnabled)
{
query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
}
if (model.WorkOrderFilterEnabled)
{
query.WhereIn("wo.WorkOrderNumber", model.WorkOrderFilter.Select(w => w.WorkOrderNumber));
}
if (model.ItemNumberFilterEnabled)
{
query.WhereIn("wo.ItemNumber", model.ItemNumberFilter.Select(i => i.ItemNumber));
}
if (model.OperatorFilterEnabled)
{
query.WhereIn("wot.OperatorAN", model.OperatorFilter.Select(o => o.AddressNumber));
}
// Compile to SQL + parameters
var compiled = _compiler.Compile(query);
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
return new SearchQueryResult(compiled.Sql, new Dictionary<string, object> { ["SearchId"] = searchId });
}
}
```
**Complex Query Composition with SqlKata:**
**Extraction Function Integration:**
```csharp
public sealed class ComposableSearchQueryBuilder
{
private readonly SqlServerCompiler _compiler = new();
The query builder generates SQL that populates temporary filter tables from extraction functions:
public SearchQueryResult BuildFullSearchQuery(SearchModel model)
{
// Build composable query parts
var baseQuery = BuildBaseWorkOrderQuery(model);
var filterQuery = ApplyFilters(baseQuery, model);
var joinedQuery = ApplyConditionalJoins(filterQuery, model);
```sql
-- Generated SQL fragment example:
-- Populate temp tables from extraction functions
INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId);
INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId);
INSERT INTO #P_ProfitCenters SELECT Code FROM dbo.fn_GetSearchProfitCenters(@SearchId);
INSERT INTO #P_WorkCenters SELECT Code FROM dbo.fn_GetSearchWorkCenters(@SearchId);
INSERT INTO #P_OperatorIDs SELECT OperatorID FROM dbo.fn_GetSearchOperatorIDs(@SearchId);
INSERT INTO #P_ComponentLots SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId);
INSERT INTO #P_PartOperations SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId);
var compiled = _compiler.Compile(joinedQuery);
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
}
private Query BuildBaseWorkOrderQuery(SearchModel model)
{
return new Query("WorkOrder_Curr as wo")
.Select("wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode",
"wo.StatusUpdateDT", "wo.QuantityOrdered", "wo.QuantityCompleted");
}
private Query ApplyFilters(Query query, SearchModel model)
{
if (model.TimespanFilterEnabled && model.MinimumDT.HasValue && model.MaximumDT.HasValue)
{
query = query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
}
if (model.WorkOrderFilterEnabled)
{
query = query.WhereIn("wo.WorkOrderNumber",
model.WorkOrderFilter.Select(w => w.WorkOrderNumber).ToList());
}
if (model.ItemNumberFilterEnabled)
{
query = query.WhereIn("wo.ItemNumber",
model.ItemNumberFilter.Select(i => i.ItemNumber).ToList());
}
return query;
}
private Query ApplyConditionalJoins(Query query, SearchModel model)
{
if (model.ProfitCenterFilterEnabled)
{
query = query
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
.Join("WorkCenter as wc", "wc.Code", "wos.WorkCenterCode")
.Join("OrgHierarchy as oh", "oh.WorkCenterCode", "wc.Code")
.WhereIn("oh.ProfitCenterCode",
model.ProfitCenterFilter.Select(p => p.Code).ToList());
}
if (model.WorkCenterFilterEnabled)
{
query = query
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
.WhereIn("wos.WorkCenterCode",
model.WorkCenterFilter.Select(w => w.Code).ToList());
}
return query;
}
}
-- Check which filters are active based on temp table population
DECLARE @HasWorkOrderFilter BIT = CASE WHEN EXISTS(SELECT 1 FROM #P_WorkOrders) THEN 1 ELSE 0 END;
-- etc.
```
This approach eliminates the need to pass filter collections from C# to SQL, simplifying the interface to a single `searchId` parameter.
**Work Order Traversal (Stored Procedure):**
The iterative downstream work order traversal is implemented as a stored procedure (`dbo.TraverseWorkOrders`) due to its iterative nature with temp tables and MERGE operations. The maximum iteration count is configurable via `SearchProcessingOptions.MaxTraversalIterations` (default: 20):
@@ -675,13 +600,11 @@ The system SHALL optionally extract Manufacturing Information System (MIS) data
---
### Requirement: Table-Valued Parameter Creation
### Requirement: Simplified Query Parameter Passing
The system SHALL create SQL table-valued parameters for efficient filter data transmission using Microsoft.Data.SqlClient.
The system SHALL pass only the SearchId parameter to search queries, with filter extraction handled by SQL functions.
#### Migration Note: SqlClient and TVP Options
**SQL Client Package Change (Required)**
#### Migration Note: SqlClient Package Change (Required)
Replace `System.Data.SqlClient` with `Microsoft.Data.SqlClient`:
@@ -693,92 +616,37 @@ using System.Data.SqlClient;
using Microsoft.Data.SqlClient;
```
This is a cross-cutting change affecting all database access code. The API is largely compatible but `Microsoft.Data.SqlClient` is the actively maintained package with security updates.
#### Simplified Parameter Model
**TVP Implementation Options**
The legacy approach using `DataTable` remains valid. An alternative using `IEnumerable<SqlDataRecord>` offers better performance for large datasets:
With extraction functions, the query builder interface is simplified:
```csharp
// Option 1: DataTable approach (legacy pattern, still supported)
public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter(
this SearchModel model)
public interface ISearchQueryBuilder
{
var dataTable = new DataTable();
dataTable.Columns.Add("WorkOrderNumber", typeof(long));
foreach (var entry in model.WorkOrderFilter)
{
dataTable.Rows.Add(entry.WorkOrderNumber);
}
return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);
}
// Option 2: SqlDataRecord approach (more efficient for large datasets)
public static IEnumerable<SqlDataRecord> ToWorkOrderRecords(
this IEnumerable<WorkOrderFilterEntry> entries)
{
var metadata = new SqlMetaData("WorkOrderNumber", SqlDbType.BigInt);
var record = new SqlDataRecord(metadata);
foreach (var entry in entries)
{
record.SetInt64(0, entry.WorkOrderNumber);
yield return record;
}
}
// Query execution only needs SearchId
var result = queryBuilder.BuildSearchQuery(searchId);
var results = await connection.QueryAsync<SearchResult>(result.Sql, new { SearchId = searchId });
```
#### Inputs
- `SearchModel` with populated filter entry collections
- SQL Server table type definitions for each parameter type
#### Outputs
- `SqlMapper.ICustomQueryParameter` objects created via `AsTableValuedParameter()`:
- `WorkOrderFilterParameter`: WorkOrderNumber column
- `ItemNumberFilterParameter`: ItemNumber column
- `ProfitCenterFilterParameter`: Code column
- `WorkCenterFilterParameter`: Code column
- `ComponentLotFilterParameter`: ComponentLotNumber + ItemNumber columns
- `OperatorFilterParameter`: UserName column
- `ItemOperationMisFilterParameter`: ItemNumber + OperationNumber + MisNumber + MisRevision columns
#### Business Rules
- Each parameter type MUST match the corresponding SQL Server table type schema
- DataTable column types MUST match the expected SQL types (long, string)
- Parameters are created even for empty filter collections (empty DataTable)
- Operator filter uses UserID (not AddressNumber) as the parameter value
- Search queries SHALL accept only `searchId` as the filter parameter
- SQL extraction functions handle JSON parsing and filter population
- TVP infrastructure for search execution has been removed
- TVPs remain in use for reference data lookups (LookupItemsAsync, etc.)
- All database operations SHOULD use async methods (`QueryAsync`, `ExecuteAsync`)
#### Scenario: Create work order filter parameter
#### Scenario: Execute search with simplified parameter
- **WHEN** SearchModel contains WorkOrderFilter with entries [12345, 67890]
- **THEN** the system creates a DataTable with WorkOrderNumber column
- **AND** populates two rows with the work order numbers
- **AND** returns parameter of type "WorkOrderFilterParameter"
#### Scenario: Create component lot filter parameter
- **WHEN** SearchModel contains ComponentLotFilter with lot "LOT-A" for item "ITEM-001"
- **THEN** the system creates a DataTable with ComponentLotNumber and ItemNumber columns
- **AND** populates one row with both values
- **AND** returns parameter of type "ComponentLotFilterParameter"
#### Scenario: Create empty filter parameter
- **WHEN** SearchModel contains empty WorkOrderFilter collection
- **THEN** the system creates a DataTable with WorkOrderNumber column
- **AND** the DataTable has zero rows
- **AND** returns valid parameter that produces empty result set
#### Scenario: Create item operation MIS filter parameter
- **WHEN** SearchModel contains ItemOperationMisFilter with one entry
- **THEN** the system creates a DataTable with four columns
- **AND** populates ItemNumber, OperationNumber, MisNumber, MisRevision
- **AND** returns parameter of type "ItemOperationMisFilterParameter"
- **WHEN** search ID 123 is processed
- **THEN** query builder generates SQL with `@SearchId` parameter only
- **AND** extraction functions populate temporary filter tables from Search.Criteria JSON
- **AND** no DataTable or TVP creation is needed for search execution
---