dohertj2/jdescopingtool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update documentation for extraction functions migration

Browse Source 
- Add ExtractionFunctions.md reference document
- Update database-schema spec with 11 extraction functions
- Update data-access spec to document extraction function approach
- Update search-processing spec with new query builder interface
- Add Database.Tests to Testing.md architecture doc
- Update DataFlow.md with extraction function flow
This commit is contained in:
Joseph Doherty
2026-01-06 14:54:10 -05:00
parent 35c1e6baf0
commit c6aeb20d9c
8 changed files with 602 additions and 345 deletions
Download Patch File Download Diff File Expand all files Collapse all files
openspec/specs/search-processing/spec.md
+54 -186
View File
@@ -86,7 +86,7 @@ The legacy system uses a T4 text template (`QueryTemplate.tt`) for SQL generatio
**NuGet Package:** `SqlKata` and `SqlKata.Execution`
**Primary Query Building with SqlKata:**
**Primary Query Building with SqlKata and Extraction Functions:**
```csharp
using SqlKata;
@@ -94,7 +94,9 @@ using SqlKata.Compilers;
public interface ISearchQueryBuilder
{
SearchQueryResult BuildSearchQuery(SearchModel model);
SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);
}
public record SearchQueryResult(string Sql, IDictionary<string, object> Parameters);
@@ -103,127 +105,50 @@ public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder
{
private readonly SqlServerCompiler _compiler = new();
public SearchQueryResult BuildSearchQuery(SearchModel model)
public SearchQueryResult BuildSearchQuery(int searchId)
{
// Query uses SQL extraction functions to get criteria values
// No need to pass filter collections from C# - just the SearchId
// Generated SQL populates temp tables from extraction functions:
// INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
// INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
// etc.
var query = new Query("WorkOrder_Curr as wo")
.Select("wo.*", "wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode");
// Conditional joins based on active filters
if (model.OperatorFilterEnabled)
{
query.Join("WorkOrderTime_Curr as wot", "wot.WorkOrderNumber", "wo.WorkOrderNumber");
}
// Conditional joins based on temp table population (determined at runtime)
// Query builder generates SQL that checks IF EXISTS on temp tables
if (model.ProfitCenterFilterEnabled || model.WorkCenterFilterEnabled)
{
query.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber");
}
if (model.ComponentLotFilterEnabled)
{
query.Join("WorkOrderComponent_Curr as woc", "woc.WorkOrderNumber", "wo.WorkOrderNumber");
}
// Conditional WHERE clauses
if (model.TimespanFilterEnabled)
{
query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
}
if (model.WorkOrderFilterEnabled)
{
query.WhereIn("wo.WorkOrderNumber", model.WorkOrderFilter.Select(w => w.WorkOrderNumber));
}
if (model.ItemNumberFilterEnabled)
{
query.WhereIn("wo.ItemNumber", model.ItemNumberFilter.Select(i => i.ItemNumber));
}
if (model.OperatorFilterEnabled)
{
query.WhereIn("wot.OperatorAN", model.OperatorFilter.Select(o => o.AddressNumber));
}
// Compile to SQL + parameters
var compiled = _compiler.Compile(query);
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
return new SearchQueryResult(compiled.Sql, new Dictionary<string, object> { ["SearchId"] = searchId });
}
}
```
**Complex Query Composition with SqlKata:**
**Extraction Function Integration:**
```csharp
public sealed class ComposableSearchQueryBuilder
{
private readonly SqlServerCompiler _compiler = new();
The query builder generates SQL that populates temporary filter tables from extraction functions:
public SearchQueryResult BuildFullSearchQuery(SearchModel model)
{
// Build composable query parts
var baseQuery = BuildBaseWorkOrderQuery(model);
var filterQuery = ApplyFilters(baseQuery, model);
var joinedQuery = ApplyConditionalJoins(filterQuery, model);
```sql
-- Generated SQL fragment example:
-- Populate temp tables from extraction functions
INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId);
INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId);
INSERT INTO #P_ProfitCenters SELECT Code FROM dbo.fn_GetSearchProfitCenters(@SearchId);
INSERT INTO #P_WorkCenters SELECT Code FROM dbo.fn_GetSearchWorkCenters(@SearchId);
INSERT INTO #P_OperatorIDs SELECT OperatorID FROM dbo.fn_GetSearchOperatorIDs(@SearchId);
INSERT INTO #P_ComponentLots SELECT LotNumber, ItemNumber FROM dbo.fn_GetSearchComponentLots(@SearchId);
INSERT INTO #P_PartOperations SELECT ItemNumber, OperationNumber, MisNumber, MisRevision FROM dbo.fn_GetSearchPartOperations(@SearchId);
var compiled = _compiler.Compile(joinedQuery);
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
}
private Query BuildBaseWorkOrderQuery(SearchModel model)
{
return new Query("WorkOrder_Curr as wo")
.Select("wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode",
"wo.StatusUpdateDT", "wo.QuantityOrdered", "wo.QuantityCompleted");
}
private Query ApplyFilters(Query query, SearchModel model)
{
if (model.TimespanFilterEnabled && model.MinimumDT.HasValue && model.MaximumDT.HasValue)
{
query = query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
}
if (model.WorkOrderFilterEnabled)
{
query = query.WhereIn("wo.WorkOrderNumber",
model.WorkOrderFilter.Select(w => w.WorkOrderNumber).ToList());
}
if (model.ItemNumberFilterEnabled)
{
query = query.WhereIn("wo.ItemNumber",
model.ItemNumberFilter.Select(i => i.ItemNumber).ToList());
}
return query;
}
private Query ApplyConditionalJoins(Query query, SearchModel model)
{
if (model.ProfitCenterFilterEnabled)
{
query = query
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
.Join("WorkCenter as wc", "wc.Code", "wos.WorkCenterCode")
.Join("OrgHierarchy as oh", "oh.WorkCenterCode", "wc.Code")
.WhereIn("oh.ProfitCenterCode",
model.ProfitCenterFilter.Select(p => p.Code).ToList());
}
if (model.WorkCenterFilterEnabled)
{
query = query
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
.WhereIn("wos.WorkCenterCode",
model.WorkCenterFilter.Select(w => w.Code).ToList());
}
return query;
}
}
-- Check which filters are active based on temp table population
DECLARE @HasWorkOrderFilter BIT = CASE WHEN EXISTS(SELECT 1 FROM #P_WorkOrders) THEN 1 ELSE 0 END;
-- etc.
```
This approach eliminates the need to pass filter collections from C# to SQL, simplifying the interface to a single `searchId` parameter.
**Work Order Traversal (Stored Procedure):**
The iterative downstream work order traversal is implemented as a stored procedure (`dbo.TraverseWorkOrders`) due to its iterative nature with temp tables and MERGE operations. The maximum iteration count is configurable via `SearchProcessingOptions.MaxTraversalIterations` (default: 20):
@@ -675,13 +600,11 @@ The system SHALL optionally extract Manufacturing Information System (MIS) data
---
### Requirement: Table-Valued Parameter Creation
### Requirement: Simplified Query Parameter Passing
The system SHALL create SQL table-valued parameters for efficient filter data transmission using Microsoft.Data.SqlClient.
The system SHALL pass only the SearchId parameter to search queries, with filter extraction handled by SQL functions.
#### Migration Note: SqlClient and TVP Options
**SQL Client Package Change (Required)**
#### Migration Note: SqlClient Package Change (Required)
Replace `System.Data.SqlClient` with `Microsoft.Data.SqlClient`:
@@ -693,92 +616,37 @@ using System.Data.SqlClient;
using Microsoft.Data.SqlClient;
```
This is a cross-cutting change affecting all database access code. The API is largely compatible but `Microsoft.Data.SqlClient` is the actively maintained package with security updates.
#### Simplified Parameter Model
**TVP Implementation Options**
The legacy approach using `DataTable` remains valid. An alternative using `IEnumerable<SqlDataRecord>` offers better performance for large datasets:
With extraction functions, the query builder interface is simplified:
```csharp
// Option 1: DataTable approach (legacy pattern, still supported)
public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter(
this SearchModel model)
public interface ISearchQueryBuilder
{
var dataTable = new DataTable();
dataTable.Columns.Add("WorkOrderNumber", typeof(long));
foreach (var entry in model.WorkOrderFilter)
{
dataTable.Rows.Add(entry.WorkOrderNumber);
}
return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
SearchQueryResult BuildSearchQuery(int searchId);
SearchQueryResult BuildMisQuery(int searchId);
SearchQueryResult BuildMisNonMatchQuery(int searchId);
}
// Option 2: SqlDataRecord approach (more efficient for large datasets)
public static IEnumerable<SqlDataRecord> ToWorkOrderRecords(
this IEnumerable<WorkOrderFilterEntry> entries)
{
var metadata = new SqlMetaData("WorkOrderNumber", SqlDbType.BigInt);
var record = new SqlDataRecord(metadata);
foreach (var entry in entries)
{
record.SetInt64(0, entry.WorkOrderNumber);
yield return record;
}
}
// Query execution only needs SearchId
var result = queryBuilder.BuildSearchQuery(searchId);
var results = await connection.QueryAsync<SearchResult>(result.Sql, new { SearchId = searchId });
```
#### Inputs
- `SearchModel` with populated filter entry collections
- SQL Server table type definitions for each parameter type
#### Outputs
- `SqlMapper.ICustomQueryParameter` objects created via `AsTableValuedParameter()`:
- `WorkOrderFilterParameter`: WorkOrderNumber column
- `ItemNumberFilterParameter`: ItemNumber column
- `ProfitCenterFilterParameter`: Code column
- `WorkCenterFilterParameter`: Code column
- `ComponentLotFilterParameter`: ComponentLotNumber + ItemNumber columns
- `OperatorFilterParameter`: UserName column
- `ItemOperationMisFilterParameter`: ItemNumber + OperationNumber + MisNumber + MisRevision columns
#### Business Rules
- Each parameter type MUST match the corresponding SQL Server table type schema
- DataTable column types MUST match the expected SQL types (long, string)
- Parameters are created even for empty filter collections (empty DataTable)
- Operator filter uses UserID (not AddressNumber) as the parameter value
- Search queries SHALL accept only `searchId` as the filter parameter
- SQL extraction functions handle JSON parsing and filter population
- TVP infrastructure for search execution has been removed
- TVPs remain in use for reference data lookups (LookupItemsAsync, etc.)
- All database operations SHOULD use async methods (`QueryAsync`, `ExecuteAsync`)
#### Scenario: Create work order filter parameter
#### Scenario: Execute search with simplified parameter
- **WHEN** SearchModel contains WorkOrderFilter with entries [12345, 67890]
- **THEN** the system creates a DataTable with WorkOrderNumber column
- **AND** populates two rows with the work order numbers
- **AND** returns parameter of type "WorkOrderFilterParameter"
#### Scenario: Create component lot filter parameter
- **WHEN** SearchModel contains ComponentLotFilter with lot "LOT-A" for item "ITEM-001"
- **THEN** the system creates a DataTable with ComponentLotNumber and ItemNumber columns
- **AND** populates one row with both values
- **AND** returns parameter of type "ComponentLotFilterParameter"
#### Scenario: Create empty filter parameter
- **WHEN** SearchModel contains empty WorkOrderFilter collection
- **THEN** the system creates a DataTable with WorkOrderNumber column
- **AND** the DataTable has zero rows
- **AND** returns valid parameter that produces empty result set
#### Scenario: Create item operation MIS filter parameter
- **WHEN** SearchModel contains ItemOperationMisFilter with one entry
- **THEN** the system creates a DataTable with four columns
- **AND** populates ItemNumber, OperationNumber, MisNumber, MisRevision
- **AND** returns parameter of type "ItemOperationMisFilterParameter"
- **WHEN** search ID 123 is processed
- **THEN** query builder generates SQL with `@SearchId` parameter only
- **AND** extraction functions populate temporary filter tables from Search.Criteria JSON
- **AND** no DataTable or TVP creation is needed for search execution
---