jdescopingtool/openspec/changes/archive/2026-01-01-implement-search-processing/design.md

Search Processing Design

Overview

This document describes the architecture and implementation approach for the search processing subsystem, including the SqlKata query builder pattern, filter handler architecture, and search processor service.

Architecture

High-Level Component Diagram

┌─────────────────────────────────────────────────────────────────────────┐
│                         SearchProcessor                                   │
│  - Orchestrates search execution                                         │
│  - Coordinates filter enrichment, query building, execution              │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ ILotFinder      │    │ ISearchQuery    │    │ IWorkOrder      │
│ Repository      │    │ Builder         │    │ TraversalService│
│ (enrichment)    │    │ (SQL generation)│    │ (downstream)    │
└─────────────────┘    └────────┬────────┘    └─────────────────┘
                                │
                    ┌───────────┴───────────┐
                    │                       │
                    ▼                       ▼
         ┌─────────────────┐     ┌─────────────────┐
         │ Filter Handlers │     │ SqlServerCompiler│
         │ (composable)    │     │ (SQL output)    │
         └─────────────────┘     └─────────────────┘

Project Structure

NEW/src/JdeScoping.SearchProcessing/
├── Interfaces/
│   ├── ISearchProcessor.cs
│   ├── ISearchQueryBuilder.cs
│   ├── IFilterHandler.cs
│   └── IWorkOrderTraversalService.cs
├── QueryBuilders/
│   ├── SqlKataSearchQueryBuilder.cs
│   └── MisQueryBuilder.cs
├── FilterHandlers/
│   ├── FilterHandlerBase.cs
│   ├── WorkOrderFilterHandler.cs
│   ├── ItemNumberFilterHandler.cs
│   ├── ProfitCenterFilterHandler.cs
│   ├── WorkCenterFilterHandler.cs
│   ├── OperatorFilterHandler.cs
│   ├── ComponentLotFilterHandler.cs
│   ├── ItemOperationMisFilterHandler.cs
│   └── TimespanFilterHandler.cs
├── Models/
│   ├── SearchModel.cs
│   ├── SearchQueryResult.cs
│   ├── FilterEntries/
│   │   ├── WorkOrderFilterEntry.cs
│   │   ├── ItemNumberFilterEntry.cs
│   │   ├── ProfitCenterFilterEntry.cs
│   │   ├── WorkCenterFilterEntry.cs
│   │   ├── OperatorFilterEntry.cs
│   │   ├── ComponentLotFilterEntry.cs
│   │   └── ItemOperationMisFilterEntry.cs
│   └── Results/
│       ├── SearchResult.cs
│       ├── MisSearchResult.cs
│       └── MisNonMatchSearchResult.cs
├── Services/
│   ├── SearchProcessor.cs
│   └── WorkOrderTraversalService.cs
├── Configuration/
│   └── SearchProcessingOptions.cs
├── Attributes/
│   ├── OutputColumnAttribute.cs
│   └── OutputTableAttribute.cs
├── Extensions/
│   ├── SearchModelExtensions.cs
│   └── TableValuedParameterExtensions.cs
├── ServiceCollectionExtensions.cs
└── JdeScoping.SearchProcessing.csproj

SqlKata Query Builder Architecture

ISearchQueryBuilder Interface

public interface ISearchQueryBuilder
{
    /// <summary>
    /// Builds the main search query for flagging and retrieving work orders.
    /// </summary>
    SearchQueryResult BuildSearchQuery(SearchModel model);

    /// <summary>
    /// Builds the MIS data extraction query when ExtractMisData is enabled.
    /// </summary>
    SearchQueryResult BuildMisQuery(SearchModel model);

    /// <summary>
    /// Builds the MIS non-match query for work orders without MIS records.
    /// </summary>
    SearchQueryResult BuildMisNonMatchQuery(SearchModel model);
}

public record SearchQueryResult(
    string Sql,
    IDictionary<string, object> Parameters,
    IReadOnlyList<string> TempTableSetupSql);

SqlKata Integration Pattern

The SqlKata query builder composes queries using fluent methods:

public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder
{
    private readonly SqlServerCompiler _compiler = new();
    private readonly IEnumerable<IFilterHandler> _filterHandlers;

    public SqlKataSearchQueryBuilder(IEnumerable<IFilterHandler> filterHandlers)
    {
        _filterHandlers = filterHandlers;
    }

    public SearchQueryResult BuildSearchQuery(SearchModel model)
    {
        var setupStatements = new List<string>();
        var parameters = new Dictionary<string, object>();

        // Build temp table setup SQL
        setupStatements.Add(BuildTempWoTableSql());

        // Apply filter handlers (each may add setup SQL and parameters)
        foreach (var handler in _filterHandlers.Where(h => h.IsEnabled(model)))
        {
            var filterResult = handler.Apply(model, _compiler);
            setupStatements.AddRange(filterResult.SetupSql);
            foreach (var param in filterResult.Parameters)
            {
                parameters[param.Key] = param.Value;
            }
        }

        // Build final result query
        var resultQuery = BuildResultQuery();
        var compiled = _compiler.Compile(resultQuery);

        return new SearchQueryResult(
            compiled.Sql,
            MergeParameters(parameters, compiled.NamedBindings),
            setupStatements);
    }
}

Why SqlKata Instead of T4 Templates

Aspect	T4 Template (Legacy)	SqlKata (New)
Testability	Cannot unit test	Test query structure without DB
Type Safety	String concatenation	Fluent API with IntelliSense
SQL Injection	Manual parameter handling	Parameterized by default
Maintenance	Edit .tt file, regenerate	Edit C# code directly
SDK Support	Limited in modern .NET	Full .NET 10 support
Composability	Monolithic template	Pluggable filter handlers

Filter Handler Pattern

IFilterHandler Interface

public interface IFilterHandler
{
    /// <summary>
    /// Determines if this filter is active for the given search model.
    /// </summary>
    bool IsEnabled(SearchModel model);

    /// <summary>
    /// Applies the filter, returning setup SQL and parameters.
    /// </summary>
    FilterResult Apply(SearchModel model, SqlServerCompiler compiler);

    /// <summary>
    /// Priority for handler execution order (lower = earlier).
    /// </summary>
    int Priority { get; }
}

public record FilterResult(
    IReadOnlyList<string> SetupSql,
    IDictionary<string, object> Parameters);

Filter Handler Implementations

Each filter handler encapsulates the logic for one filter type:

WorkOrderFilterHandler

public sealed class WorkOrderFilterHandler : FilterHandlerBase
{
    public override int Priority => 10;

    public override bool IsEnabled(SearchModel model)
        => model.WorkOrderFilterEnabled;

    public override FilterResult Apply(SearchModel model, SqlServerCompiler compiler)
    {
        // Generates MERGE into #Temp_WO with ManuallySpecified = 1
        // Followed by split order detection
        var sql = BuildWorkOrderMergeSql();
        var parameters = new Dictionary<string, object>
        {
            ["p_WorkOrderFilter"] = model.CreateWorkOrderFilterParameter()
        };

        return new FilterResult(new[] { sql }, parameters);
    }
}

ComponentLotFilterHandler

public sealed class ComponentLotFilterHandler : FilterHandlerBase
{
    public override int Priority => 30;

    public override bool IsEnabled(SearchModel model)
        => model.ComponentLotFilterEnabled;

    public override FilterResult Apply(SearchModel model, SqlServerCompiler compiler)
    {
        // Joins Lot -> WorkOrderComponent/LotUsage -> WorkOrder
        // Sets CARDEX = 1 flag
        var sql = BuildComponentLotMergeSql();
        var parameters = new Dictionary<string, object>
        {
            ["p_ComponentLotFilter"] = model.CreateComponentLotFilterParameter()
        };

        return new FilterResult(new[] { sql }, parameters);
    }
}

Handler Execution Order

Handlers execute in priority order to ensure dependent temp tables exist:

Priority	Handler	Creates/Uses
10	WorkOrderFilterHandler	Creates #Temp_WO entries
20	ItemNumberFilterHandler	Creates #P_ItemNumbers
30	ComponentLotFilterHandler	Uses Lot, creates #Temp_WO entries
40	ProfitCenterFilterHandler	Creates #P_WorkCenters
50	WorkCenterFilterHandler	Creates #P_WorkCenters
60	OperatorFilterHandler	Creates #P_OperatorIDs
70	ItemOperationMisFilterHandler	Creates #P_PartOperations
80	TimespanFilterHandler	Adds WHERE clause

IAsyncEnumerable Streaming Pattern

Large Result Set Handling

For searches returning thousands of work orders, streaming avoids loading all results into memory:

public interface ISearchProcessor
{
    /// <summary>
    /// Executes search and returns results as async stream.
    /// </summary>
    IAsyncEnumerable<SearchResult> ExecuteSearchAsync(
        SearchModel model,
        CancellationToken ct = default);

    /// <summary>
    /// Executes search and materializes all results into SearchModel.
    /// </summary>
    Task<SearchModel> ExecuteSearchToModelAsync(
        SearchModel model,
        CancellationToken ct = default);
}

Streaming Implementation

public sealed class SearchProcessor : ISearchProcessor
{
    private readonly IDbConnectionFactory _connectionFactory;
    private readonly ISearchQueryBuilder _queryBuilder;
    private readonly IWorkOrderTraversalService _traversalService;
    private readonly ILogger<SearchProcessor> _logger;

    public async IAsyncEnumerable<SearchResult> ExecuteSearchAsync(
        SearchModel model,
        [EnumeratorCancellation] CancellationToken ct = default)
    {
        await using var connection = await _connectionFactory
            .CreateLotFinderConnectionAsync(ct);

        // Execute setup SQL (temp tables, filter population)
        var queryResult = _queryBuilder.BuildSearchQuery(model);
        foreach (var setupSql in queryResult.TempTableSetupSql)
        {
            await connection.ExecuteAsync(setupSql, queryResult.Parameters);
        }

        // Stream results
        await foreach (var result in connection.QueryUnbufferedAsync<SearchResult>(
            queryResult.Sql,
            queryResult.Parameters)
            .WithCancellation(ct))
        {
            yield return result;
        }
    }
}

Downstream Work Order Traversal

Stored Procedure Approach

The iterative traversal logic (up to 20 iterations finding downstream work orders) is better suited to a stored procedure:

public interface IWorkOrderTraversalService
{
    /// <summary>
    /// Traverses downstream work orders via stored procedure.
    /// Called after initial filtering to find related work orders.
    /// </summary>
    Task TraverseDownstreamAsync(
        SqlConnection connection,
        CancellationToken ct = default);
}

Why Stored Procedure

• Iterative logic: WHILE loops with temp table operations are efficient in T-SQL
• Reduced round trips: Single call instead of 20+ iterations from C#
• Transaction scope: All MERGE operations in same transaction
• Legacy compatibility: Mirrors existing QueryTemplate.tt behavior

Table-Valued Parameter Helpers

Extension Methods

public static class TableValuedParameterExtensions
{
    public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter(
        this SearchModel model)
    {
        var dataTable = new DataTable();
        dataTable.Columns.Add("WorkOrderNumber", typeof(long));

        foreach (var entry in model.WorkOrderFilter)
        {
            dataTable.Rows.Add(entry.WorkOrderNumber);
        }

        return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
    }

    // Similar methods for all 7 filter types...
}

TVP Type Mapping

C# Method	SQL Server Type	Columns
CreateWorkOrderFilterParameter	WorkOrderFilterParameter	WorkOrderNumber BIGINT
CreateItemNumberFilterParameter	ItemNumberFilterParameter	ItemNumber VARCHAR(25)
CreateProfitCenterFilterParameter	ProfitCenterFilterParameter	Code VARCHAR(12)
CreateWorkCenterFilterParameter	WorkCenterFilterParameter	Code VARCHAR(12)
CreateOperatorFilterParameter	OperatorFilterParameter	UserName VARCHAR(10)
CreateComponentLotFilterParameter	ComponentLotFilterParameter	ComponentLotNumber VARCHAR(30), ItemNumber VARCHAR(25)
CreateItemOperationMisFilterParameter	ItemOperationMisFilterParameter	ItemNumber, OperationNumber, MisNumber, MisRevision

Configuration Options

SearchProcessingOptions Class

public class SearchProcessingOptions
{
    public const string SectionName = "SearchProcessing";

    /// <summary>
    /// Query timeout in seconds for search execution.
    /// </summary>
    public int QueryTimeoutSeconds { get; set; } = 600;

    /// <summary>
    /// Maximum downstream traversal iterations.
    /// </summary>
    public int MaxTraversalIterations { get; set; } = 20;

    /// <summary>
    /// Enable debug SQL logging.
    /// </summary>
    public bool EnableDebugSql { get; set; } = false;

    /// <summary>
    /// Path to write debug SQL files (when EnableDebugSql is true).
    /// </summary>
    public string? DebugSqlPath { get; set; }
}

Configuration Binding

{
  "SearchProcessing": {
    "QueryTimeoutSeconds": 600,
    "MaxTraversalIterations": 20,
    "EnableDebugSql": false,
    "DebugSqlPath": null
  }
}

Service Registration

AddSearchProcessing Extension Method

public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddSearchProcessing(
        this IServiceCollection services,
        IConfiguration configuration)
    {
        // Bind options
        services.Configure<SearchProcessingOptions>(
            configuration.GetSection(SearchProcessingOptions.SectionName));

        // Register SqlKata compiler (singleton, thread-safe)
        services.AddSingleton<SqlServerCompiler>();

        // Register filter handlers (scoped)
        services.AddScoped<IFilterHandler, WorkOrderFilterHandler>();
        services.AddScoped<IFilterHandler, ItemNumberFilterHandler>();
        services.AddScoped<IFilterHandler, ProfitCenterFilterHandler>();
        services.AddScoped<IFilterHandler, WorkCenterFilterHandler>();
        services.AddScoped<IFilterHandler, OperatorFilterHandler>();
        services.AddScoped<IFilterHandler, ComponentLotFilterHandler>();
        services.AddScoped<IFilterHandler, ItemOperationMisFilterHandler>();
        services.AddScoped<IFilterHandler, TimespanFilterHandler>();

        // Register query builders (scoped)
        services.AddScoped<ISearchQueryBuilder, SqlKataSearchQueryBuilder>();

        // Register services (scoped)
        services.AddScoped<ISearchProcessor, SearchProcessor>();
        services.AddScoped<IWorkOrderTraversalService, WorkOrderTraversalService>();

        return services;
    }
}

Testing Strategy

Unit Tests

• Query Builder Tests: Verify generated SQL structure without executing
• Filter Handler Tests: Test each handler in isolation
• Parameter Tests: Verify TVP creation for all filter types
• Mock Repository: Use NSubstitute for ILotFinderRepository

Integration Tests

• Full Search Flow: Execute search against test database
• Filter Combinations: Matrix of filter permutations
• Large Result Sets: Verify streaming behavior
• MIS Extraction: Test with and without MIS data

Test Frameworks

• xUnit: Test framework
• Shouldly: Fluent assertions
• NSubstitute: Mocking framework

NuGet Dependencies

<ItemGroup>
  <PackageReference Include="SqlKata" Version="3.0.*" />
  <PackageReference Include="SqlKata.Execution" Version="3.0.*" />
  <PackageReference Include="Dapper" Version="2.1.*" />
  <PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.*" />
  <PackageReference Include="Microsoft.Extensions.Options" Version="9.0.*" />
  <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.*" />
</ItemGroup>

<ItemGroup Condition="'$(Configuration)' == 'Test'">
  <PackageReference Include="xunit" Version="2.9.*" />
  <PackageReference Include="Shouldly" Version="4.2.*" />
  <PackageReference Include="NSubstitute" Version="5.1.*" />
</ItemGroup>