jdescopingtool/openspec/specs/search-processing/spec.md

Search Processing Specification

Purpose

The search processing subsystem enables users to query manufacturing data (work orders, lots, items, operators, and work centers) from a locally cached JDE/CMS database. It accepts multi-dimensional filter criteria, dynamically generates SQL queries using a builder pattern, executes searches against the SQL Server cache using Microsoft.Data.SqlClient, and aggregates results including downstream work order tracking and MIS (Manufacturing Information System) data extraction. This specification targets .NET 10 with modern C# language features and patterns.

Source Reference

Legacy Files	Purpose
OLD/DataModel/Models/SearchCriteria.cs	Core search criteria model storing filter parameters
OLD/DataModel/ViewModels/SearchCriteriaViewModel.cs	View model with enriched filter data for UI binding
OLD/WorkerService/Models/Reporting/SearchModel.cs	Reporting model combining criteria with filter entries and results
OLD/WorkerService/Templates/QueryTemplate.cs	T4-generated SQL query builder (to be replaced with builder pattern)
OLD/WorkerService/Templates/QueryTemplateExtension.cs	Partial class extension for QueryTemplate
OLD/WorkerService/Helpers/SearchModelHelpers.cs	Helper methods for model conversion and parameter creation
OLD/WorkerService/Models/Reporting/*.cs	Filter entry models and result models

Requirements

Requirement: Search Criteria Model

The system SHALL support a search criteria model with eight distinct filter types that can be combined in any combination.

Inputs

• Timespan Filter: Optional minimum and maximum datetime values
• Work Order Filter: Collection of work order numbers (long integers)
• Item Number Filter: Collection of item numbers (strings)
• Profit Center Filter: Collection of profit center codes (strings)
• Work Center Filter: Collection of work center codes (strings)
• Operator Filter: Collection of operator user IDs (strings)
• Component Lot Filter: Collection of lot number/item number pairs
• Item Operation MIS Filter: Collection of item/operation/MIS number/MIS revision combinations
• Extract MIS Data Flag: Boolean indicating whether to include MIS data in results

Outputs

• Serializable SearchCriteria object stored as JSON in the Search.CriteriaJSON column
• SearchModel populated with enriched filter entries (descriptions, full names, etc.)

Business Rules

• All filter collections SHALL be initialized as empty lists by default
• Filter lists MAY contain zero or more entries
• The timespan filter is considered enabled when either MinimumDT or MaximumDT has a value
• Each filter type is considered enabled when its collection contains at least one entry
• Component lot filters MUST include both lot number and item number for proper matching
• Item/Operation/MIS filters MUST include all four fields: item number, operation number, MIS number, and MIS revision

Scenario: Create search with timespan filter only

• WHEN a user specifies minimum date of 2024-01-01 and maximum date of 2024-12-31
• THEN the system creates a SearchCriteria with MinimumDT = 2024-01-01 and MaximumDT = 2024-12-31
• AND TimespanFilterEnabled returns true
• AND all other filter collections remain empty

Scenario: Create search with multiple filter types

• WHEN a user specifies work order numbers [12345, 67890] and item numbers ["PART-001", "PART-002"]
• THEN the system creates a SearchCriteria with both collections populated
• AND WorkOrderFilterEnabled and ItemNumberFilterEnabled both return true
• AND other filter collections remain empty

Scenario: Create search with component lot filter

• WHEN a user specifies component lot "LOT123" for item "ITEM-ABC"
• THEN the system creates a ComponentLotFilter entry with both LotNumber and ItemNumber
• AND the filter is used to find work orders that consumed material from that lot

Scenario: Serialize and deserialize search criteria

• WHEN a SearchCriteria object is serialized to JSON and stored
• THEN the system can deserialize it back to the original object structure
• AND all filter collections and values are preserved

---

Requirement: Query Builder Generation

The system SHALL dynamically generate SQL queries based on enabled filters using SqlKata fluent query builder (replacing the legacy T4 text template).

Migration Note: T4 Template Replacement with SqlKata

The legacy system uses a T4 text template (QueryTemplate.tt) for SQL generation. T4 templates have limited support in modern .NET SDK-style projects. The new implementation SHALL use SqlKata - a fluent SQL query builder that generates parameterized SQL and integrates well with Dapper.

NuGet Package: SqlKata and SqlKata.Execution

Primary Query Building with SqlKata:

using SqlKata;
using SqlKata.Compilers;

public interface ISearchQueryBuilder
{
    SearchQueryResult BuildSearchQuery(SearchModel model);
}

public record SearchQueryResult(string Sql, IDictionary<string, object> Parameters);

public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder
{
    private readonly SqlServerCompiler _compiler = new();

    public SearchQueryResult BuildSearchQuery(SearchModel model)
    {
        var query = new Query("WorkOrder_Curr as wo")
            .Select("wo.*", "wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode");

        // Conditional joins based on active filters
        if (model.OperatorFilterEnabled)
        {
            query.Join("WorkOrderTime_Curr as wot", "wot.WorkOrderNumber", "wo.WorkOrderNumber");
        }

        if (model.ProfitCenterFilterEnabled || model.WorkCenterFilterEnabled)
        {
            query.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber");
        }

        if (model.ComponentLotFilterEnabled)
        {
            query.Join("WorkOrderComponent_Curr as woc", "woc.WorkOrderNumber", "wo.WorkOrderNumber");
        }

        // Conditional WHERE clauses
        if (model.TimespanFilterEnabled)
        {
            query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
        }

        if (model.WorkOrderFilterEnabled)
        {
            query.WhereIn("wo.WorkOrderNumber", model.WorkOrderFilter.Select(w => w.WorkOrderNumber));
        }

        if (model.ItemNumberFilterEnabled)
        {
            query.WhereIn("wo.ItemNumber", model.ItemNumberFilter.Select(i => i.ItemNumber));
        }

        if (model.OperatorFilterEnabled)
        {
            query.WhereIn("wot.OperatorAN", model.OperatorFilter.Select(o => o.AddressNumber));
        }

        // Compile to SQL + parameters
        var compiled = _compiler.Compile(query);
        return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
    }
}

Complex Query Composition with SqlKata:

public sealed class ComposableSearchQueryBuilder
{
    private readonly SqlServerCompiler _compiler = new();

    public SearchQueryResult BuildFullSearchQuery(SearchModel model)
    {
        // Build composable query parts
        var baseQuery = BuildBaseWorkOrderQuery(model);
        var filterQuery = ApplyFilters(baseQuery, model);
        var joinedQuery = ApplyConditionalJoins(filterQuery, model);

        var compiled = _compiler.Compile(joinedQuery);
        return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
    }

    private Query BuildBaseWorkOrderQuery(SearchModel model)
    {
        return new Query("WorkOrder_Curr as wo")
            .Select("wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode",
                    "wo.StatusUpdateDT", "wo.QuantityOrdered", "wo.QuantityCompleted");
    }

    private Query ApplyFilters(Query query, SearchModel model)
    {
        if (model.TimespanFilterEnabled && model.MinimumDT.HasValue && model.MaximumDT.HasValue)
        {
            query = query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
        }

        if (model.WorkOrderFilterEnabled)
        {
            query = query.WhereIn("wo.WorkOrderNumber",
                model.WorkOrderFilter.Select(w => w.WorkOrderNumber).ToList());
        }

        if (model.ItemNumberFilterEnabled)
        {
            query = query.WhereIn("wo.ItemNumber",
                model.ItemNumberFilter.Select(i => i.ItemNumber).ToList());
        }

        return query;
    }

    private Query ApplyConditionalJoins(Query query, SearchModel model)
    {
        if (model.ProfitCenterFilterEnabled)
        {
            query = query
                .Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
                .Join("WorkCenter as wc", "wc.Code", "wos.WorkCenterCode")
                .Join("OrgHierarchy as oh", "oh.WorkCenterCode", "wc.Code")
                .WhereIn("oh.ProfitCenterCode",
                    model.ProfitCenterFilter.Select(p => p.Code).ToList());
        }

        if (model.WorkCenterFilterEnabled)
        {
            query = query
                .Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
                .WhereIn("wos.WorkCenterCode",
                    model.WorkCenterFilter.Select(w => w.Code).ToList());
        }

        return query;
    }
}

Work Order Traversal (Stored Procedure):

The iterative downstream work order traversal is implemented as a stored procedure (dbo.TraverseWorkOrders) due to its iterative nature with temp tables and MERGE operations. The maximum iteration count is configurable via SearchProcessingOptions.MaxTraversalIterations (default: 20):

public class SearchProcessingOptions
{
    public const string SectionName = "SearchProcessing";
    public int MaxTraversalIterations { get; set; } = 20;
}

TraverseWorkOrders Stored Procedure Interface:

public interface IWorkOrderTraversalService
{
    Task<List<long>> TraverseDownstreamAsync(
        IEnumerable<long> seedWorkOrders,
        CancellationToken ct = default);
}

public sealed class WorkOrderTraversalService : IWorkOrderTraversalService
{
    private readonly IDbConnectionFactory _connectionFactory;
    private readonly IOptions<SearchProcessingOptions> _options;

    public async Task<List<long>> TraverseDownstreamAsync(
        IEnumerable<long> seedWorkOrders,
        CancellationToken ct = default)
    {
        await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);

        var maxIterations = _options.Value.MaxTraversalIterations;

        return (await connection.QueryAsync<long>(
            "EXEC dbo.TraverseWorkOrders @p_SeedWorkOrders, @p_MaxIterations",
            new {
                p_SeedWorkOrders = seedWorkOrders.ToList().AsTableValuedParameter("dbo.WorkOrderList"),
                p_MaxIterations = maxIterations
            })).AsList();
    }
}

MIS Query Builder (Separate):

For MIS data extraction, a separate IMisQueryBuilder SHALL be used to generate MIS-specific queries:

public interface IMisQueryBuilder
{
    SearchQueryResult BuildMisDataQuery(SearchModel model);
    SearchQueryResult BuildMisNonMatchQuery(SearchModel model);
}

public sealed class SqlKataMisQueryBuilder : IMisQueryBuilder
{
    private readonly SqlServerCompiler _compiler = new();

    public SearchQueryResult BuildMisDataQuery(SearchModel model)
    {
        var query = new Query("MisData as m")
            .Select("m.*")
            .Join("WorkOrderStep_Curr as wos", q => q
                .On("wos.ItemNumber", "m.ItemNumber")
                .On("wos.OperationNumber", "m.OperationNumber"));
        // Additional MIS-specific filtering...
        var compiled = _compiler.Compile(query);
        return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
    }

    public SearchQueryResult BuildMisNonMatchQuery(SearchModel model)
    {
        // Investigation query for router mismatches
        var query = new Query("WorkOrderStep_Curr as wos")
            .LeftJoin("MisData as m", q => q
                .On("wos.ItemNumber", "m.ItemNumber")
                .On("wos.OperationNumber", "m.OperationNumber"))
            .WhereNull("m.ItemNumber"); // Non-matches
        var compiled = _compiler.Compile(query);
        return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
    }
}

Streaming vs Materialized Results:

The system SHALL provide two methods for result retrieval:

public interface ISearchResultReader
{
    /// <summary>Streaming - returns results as they are read (memory-efficient for large result sets)</summary>
    IAsyncEnumerable<SearchResult> StreamResultsAsync(int searchId, CancellationToken ct = default);

    /// <summary>Materialized - loads all results into memory (simpler for small result sets)</summary>
    Task<List<SearchResult>> GetResultsAsync(int searchId, CancellationToken ct = default);
}

SqlKata Benefits:

• Parameterized by default - SQL injection protection built-in; all parameters use @p_* naming convention for clarity
• Fluent API - Readable, composable query building
• Testable - Unit test query building without database
• Type-safe - Compile-time checking of method calls
• SQL Server optimized - SqlServerCompiler generates optimized T-SQL

Parameter Naming Convention:

All SQL parameters SHALL use the @p_ prefix for consistency:

• @p_SeedWorkOrders (not @SeedWorkOrders)
• @p_MaxIterations (not @MaxIterations)
• @p_MinimumDT, @p_MaximumDT, etc.

Inputs

• SearchModel containing filter criteria and enabled filter flags
• Table-valued parameters for each enabled filter type

Outputs

• Complete T-SQL query string including:
  • Temporary table creation statements
  • Filter parameter population
  • Work order flagging logic
  • Downstream work order traversal
  • Final result selection
  • Optional MIS data extraction queries

Business Rules

• The query builder MUST create a #Temp_WO temporary table to track flagged work orders
• The builder SHALL include SQL segments only for enabled filters (conditional generation)
• Work order step searching is triggered when any of these filters are enabled: timespan, profit center, work center, or operator
• The query MUST perform iterative downstream traversal (up to 20 iterations) to find work orders that received material from flagged work orders
• Split orders (orders created by splitting a parent order) SHALL be automatically included
• Query parameters MUST be passed as table-valued parameters for multi-value filters

Scenario: Generate query with work order filter only

• WHEN search criteria includes only work order numbers [12345, 67890]
• THEN the system generates SQL that creates #Temp_WO with ManuallySpecified flag
• AND includes MERGE statements to add specified work orders
• AND includes downstream traversal logic for related work orders
• AND excludes timespan, profit center, work center, and operator filter logic

Scenario: Generate query with profit center filter

• WHEN search criteria includes profit center codes ["PC01", "PC02"]
• THEN the system generates SQL that creates #P_WorkCenters temp table
• AND populates it by joining profit center codes to work centers via OrgHierarchy
• AND includes work order step search logic with profit center join
• AND sets ShouldSearchSteps() to return true

Scenario: Generate query with all filters enabled

• WHEN search criteria includes values for all eight filter types
• THEN the system generates SQL with all conditional segments included
• AND joins all filter temp tables appropriately
• AND includes MIS extraction queries when ExtractMisData is true

Scenario: Generate query with operator filter

• WHEN search criteria includes operator user IDs ["USER1", "USER2"]
• THEN the system generates SQL that creates #P_OperatorIDs temp table
• AND resolves user IDs to address numbers via JdeUser table
• AND joins work order time records to filter by operator

---

Requirement: Filter Entry Processing

The system SHALL enrich raw filter values with descriptive information from reference tables using C# record types for immutability.

Filter Entry Location

Filter entry types SHALL be located in the JdeScoping.Core.Models namespace within the Domain Models project, not in a separate processing project. This ensures proper separation of domain models from processing logic.

Migration Note: Record Types

The legacy filter entry classes should be converted to C# record types for value semantics and immutability:

// File: JdeScoping.Core/Models/FilterEntries.cs
namespace JdeScoping.Core.Models;

// Legacy class pattern
public class WorkOrderFilterEntry
{
    public long WorkOrderNumber { get; set; }
    public string ItemNumber { get; set; }
}

// Modern record pattern
public record WorkOrderFilterEntry(long WorkOrderNumber, string ItemNumber);

// With attributes for Excel output configuration
[OutputTable(TabName = "Work Order Filter", ShowHeader = true)]
public record WorkOrderFilterEntry(
    [property: OutputColumn(Order = 10, HeaderText = "Work Order Number")]
    long WorkOrderNumber,
    [property: OutputColumn(Order = 20, HeaderText = "Item Number")]
    string ItemNumber
);

Records provide:

• Immutability by default (init-only properties)
• Value-based equality
• Built-in ToString() and deconstruction
• Cleaner syntax for data transfer objects

Inputs

• Raw filter values from SearchCriteria (IDs, codes, numbers)
• Reference data from LotFinderDB lookup methods

Outputs

• Enriched filter entry records containing:
  • WorkOrderFilterEntry: WorkOrderNumber + ItemNumber
  • ItemNumberFilterEntry: ItemNumber + ItemDescription
  • ProfitCenterFilterEntry: Code + Description
  • WorkCenterFilterEntry: Code + Description
  • OperatorFilterEntry: AddressNumber + UserID + FullName
  • ComponentLotFilterEntry: LotNumber + ItemNumber
  • ItemOperationMisFilterEntry: ItemNumber + OperationNumber + MisNumber + MisRevision

Business Rules

• Filter entries SHALL be populated via LotFinderDB lookup methods
• Lookup methods MUST return empty collections for invalid or unmatched values
• Operator filter entries MUST resolve user IDs to address numbers for query execution
• Component lot entries MUST include both lot number and item number for proper traceability

Scenario: Enrich work order filter entries

• WHEN the system processes work order numbers [12345, 67890]
• THEN it calls LotFinderDB.LookupWorkorders() with the numbers
• AND creates WorkOrderFilterEntry records with work order number and item number
• AND includes only work orders that exist in the database

Scenario: Resolve operator user IDs to address numbers

• WHEN the system processes operator user IDs ["JSMITH", "MBROWN"]
• THEN it calls LotFinderDB.LookupUsers() to get JDE user records
• AND creates OperatorFilterEntry records with AddressNumber, UserID, and FullName
• AND the AddressNumber is used for joining to work order time records

Scenario: Handle invalid filter values

• WHEN the system processes item numbers containing non-existent items
• THEN the lookup returns only matching items
• AND non-existent item numbers are silently excluded from results

Scenario: Process component lot filter with item validation

• WHEN the system processes a component lot filter entry
• THEN it validates both lot number and item number exist together
• AND uses the combination to trace downstream work orders via WorkOrderComponent and LotUsage tables

---

Requirement: Work Order Flagging and Traversal

The system SHALL flag work orders for inclusion based on multiple criteria and traverse downstream relationships.

Inputs

• Filter criteria from SearchModel
• Work order relationships from database tables:
  • WorkOrder (parent/child via ParentWorkOrderNumber)
  • WorkOrderComponent (parts list relationships)
  • LotUsage (CARDEX material usage)
  • WorkOrderStep (operation details)
  • WorkOrderTime (operator time records)

Outputs

• #Temp_WO temporary table containing:
  • WorkOrderNumber: Primary key
  • LotNumber: Associated lot number
  • BranchCode: Branch code for the work order
  • ShortItemNumber: Item's short number
  • ManuallySpecified: Flag indicating direct specification
  • SplitOrder: Flag indicating split from flagged order
  • CARDEX: Flag indicating material receipt from flagged order (F4111) - Note: Also set for WorkOrderComponent matches
  • PartsList: Flag indicating parts list relationship (F3111)
  • Flagged: Flag indicating filter criteria match

Business Rules

• Work orders directly specified in the filter SHALL be marked with ManuallySpecified = 1
• Work orders split from flagged orders (matching ParentWorkOrderNumber) SHALL be marked with SplitOrder = 1
• Work orders receiving material from flagged lots via WorkOrderComponent SHALL be marked with CARDEX = 1 (not PartsList as might be expected)
• Work orders receiving material from flagged lots via LotUsage SHALL be marked with CARDEX = 1
• Work orders matching filter criteria (timespan, profit center, work center, operator) SHALL be marked with Flagged = 1
• Downstream traversal SHALL iterate up to 20 times to find all related work orders
• Traversal stops when no new work orders are found or maximum iterations reached

Scenario: Flag manually specified work orders

• WHEN work order 12345 is specified in the work order filter
• THEN the system marks it with ManuallySpecified = 1 in #Temp_WO
• AND finds any split orders from 12345 and marks them with SplitOrder = 1

Scenario: Trace downstream work orders via parts list

• WHEN work order 12345 produces lot "LOT-A" for item "ITEM-001"
• AND work order 67890 consumes "LOT-A" for "ITEM-001" via parts list
• THEN work order 67890 is added to #Temp_WO with CARDEX = 1

Scenario: Trace downstream work orders via CARDEX

• WHEN work order 12345 produces lot "LOT-A"
• AND work order 67890 consumes "LOT-A" via LotUsage (CARDEX)
• THEN work order 67890 is added to #Temp_WO with CARDEX = 1

Scenario: Multi-level downstream traversal

• WHEN work order A produces material consumed by work order B
• AND work order B produces material consumed by work order C
• THEN the iterative traversal finds A, B, and C
• AND marks each with appropriate flags based on relationship type

---

Requirement: Search Result Aggregation

The system SHALL aggregate flagged work orders into structured result objects with complete details.

Inputs

• Flagged work orders from #Temp_WO
• Work order details from WorkOrder table
• Item details from Item table
• Status details from StatusCode table
• Latest step from WorkOrderStep table
• Scrap totals from WorkOrderTotalScrap table

Outputs

• SearchResult objects containing:
  • Work order identification (number, branch code, lot number, item number)
  • Item details (planning family, stocking type)
  • Quantities (order, held, scrapped, shipped)
  • Latest operation step details
  • Status information (code, description, update timestamp)
  • Inclusion reason derived from flag combination

Business Rules

• Each result SHALL include the latest operation step (highest EndDT, then highest StepNumber)
• Inclusion reason SHALL be calculated with strict priority order:
  1. ManuallySpecified (highest) - Work order was directly specified in filter
  2. Flagged - Work order matched timespan/profit center/work center/operator criteria
  3. ComponentUsage - Work order received material from a flagged work order
  4. SplitOrder (lowest) - Work order was split from a flagged parent order
• Component usage reason SHALL distinguish between CARDEX only, Parts List only, or both
• Scrapped quantity SHALL default to 0 when no scrap records exist
• Results with null latest step are still included (work orders without operations)
• Inclusion reason MAY return "UNKNOWN" when no flags are set (edge case)

Scenario: Generate inclusion reason for manually specified order

• WHEN a work order is marked with ManuallySpecified = 1
• THEN the InclusionReason property returns "ManuallySpecified"

Scenario: Generate inclusion reason for flagged order

• WHEN a work order is marked with Flagged = 1 and ManuallySpecified = 0
• THEN the InclusionReason property returns "Flagged"

Scenario: Generate inclusion reason for component usage with both sources

• WHEN a work order is marked with CARDEX = 1 and PartsList = 1
• THEN the InclusionReason property returns "ComponentUsage (CARDEX + Parts List)"

Scenario: Retrieve latest operation step for work order

• WHEN work order 12345 has steps 10, 20, 30 with EndDT values
• THEN the result includes the step with the highest EndDT
• AND if EndDT values tie, the highest StepNumber wins

---

Requirement: MIS Data Extraction

The system SHALL optionally extract Manufacturing Information System (MIS) data for matching work orders.

Inputs

• ExtractMisData flag from search criteria
• ItemOperationMisFilter entries (when filtering by specific MIS)
• Work order step data joined with MIS matching function dbo.MatchMIS

Outputs

• MisSearchResult objects containing:
  • Item and MIS identification
  • Matching indicators (RoutingMatch, MasterMatch)
  • Test descriptions, sampling information
  • Tools/gauges and work instructions
• MisNonMatchSearchResult objects containing:
  • Work orders with steps that did not match MIS records
  • Added job step indicators
  • Matched job step suggestions based on work center and function code

Business Rules

• MIS extraction is only performed when ExtractMisData = true
• The MatchMIS table-valued function handles MIS record matching logic
• RoutingMatch indicates match to F3112Z1 (work order routing) records
• MasterMatch indicates match to F3003 (item master routing) records
• Non-match results include work orders where neither RoutingMatch nor MasterMatch is true, or MisNumber is null
• Added job step detection uses routing type and F3111 records to identify manually added steps
• Matched job step number suggests alternative step matching by work center and function code
• Note: MIS extraction does NOT join #Temp_WO - work order/component lot filters do NOT constrain MIS results
• Note: Timespan filtering in MIS extraction requires BOTH min and max values for the combined condition; separate handling exists for min-only or max-only cases

Scenario: Extract MIS data for filtered work orders

• WHEN search criteria has ExtractMisData = true
• THEN the system generates MIS extraction queries
• AND populates #TempMisData with MIS-matched step data
• AND returns MisSearchResult list with matching records

Scenario: Identify MIS non-matches for investigation

• WHEN MIS data is extracted and a step has no routing or master match
• THEN the system includes it in MisNonMatchSearchResult list
• AND indicates whether the job step was added (not in original routing)
• AND suggests matched job step number if one exists with same work center and function code

Scenario: Filter by specific MIS number and revision

• WHEN search criteria includes ItemOperationMisFilter with specific MIS entries
• THEN the query joins to #P_PartOperations temp table
• AND only returns work order steps matching the specified item/operation/MIS/revision combinations
• AND the WorkOrderTime UNION branch is skipped (only WorkOrderStep branch used)

Scenario: Handle NMR routing type

• WHEN a work order has routing type "NMR" (No Master Routing)
• THEN all job steps are considered added (WasJobStepAdded = 1)
• AND matched job step number is null (no routing to match against)

---

Requirement: Table-Valued Parameter Creation

The system SHALL create SQL table-valued parameters for efficient filter data transmission using Microsoft.Data.SqlClient.

Migration Note: SqlClient and TVP Options

SQL Client Package Change (Required)

Replace System.Data.SqlClient with Microsoft.Data.SqlClient:

// Legacy
using System.Data.SqlClient;

// Modern .NET 10
using Microsoft.Data.SqlClient;

This is a cross-cutting change affecting all database access code. The API is largely compatible but Microsoft.Data.SqlClient is the actively maintained package with security updates.

TVP Implementation Options

The legacy approach using DataTable remains valid. An alternative using IEnumerable<SqlDataRecord> offers better performance for large datasets:

// Option 1: DataTable approach (legacy pattern, still supported)
public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter(
    this SearchModel model)
{
    var dataTable = new DataTable();
    dataTable.Columns.Add("WorkOrderNumber", typeof(long));
    foreach (var entry in model.WorkOrderFilter)
    {
        dataTable.Rows.Add(entry.WorkOrderNumber);
    }
    return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
}

// Option 2: SqlDataRecord approach (more efficient for large datasets)
public static IEnumerable<SqlDataRecord> ToWorkOrderRecords(
    this IEnumerable<WorkOrderFilterEntry> entries)
{
    var metadata = new SqlMetaData("WorkOrderNumber", SqlDbType.BigInt);
    var record = new SqlDataRecord(metadata);

    foreach (var entry in entries)
    {
        record.SetInt64(0, entry.WorkOrderNumber);
        yield return record;
    }
}

Inputs

• SearchModel with populated filter entry collections
• SQL Server table type definitions for each parameter type

Outputs

• SqlMapper.ICustomQueryParameter objects created via AsTableValuedParameter():
  • WorkOrderFilterParameter: WorkOrderNumber column
  • ItemNumberFilterParameter: ItemNumber column
  • ProfitCenterFilterParameter: Code column
  • WorkCenterFilterParameter: Code column
  • ComponentLotFilterParameter: ComponentLotNumber + ItemNumber columns
  • OperatorFilterParameter: UserName column
  • ItemOperationMisFilterParameter: ItemNumber + OperationNumber + MisNumber + MisRevision columns

Business Rules

• Each parameter type MUST match the corresponding SQL Server table type schema
• DataTable column types MUST match the expected SQL types (long, string)
• Parameters are created even for empty filter collections (empty DataTable)
• Operator filter uses UserID (not AddressNumber) as the parameter value
• All database operations SHOULD use async methods (QueryAsync, ExecuteAsync)

Scenario: Create work order filter parameter

• WHEN SearchModel contains WorkOrderFilter with entries [12345, 67890]
• THEN the system creates a DataTable with WorkOrderNumber column
• AND populates two rows with the work order numbers
• AND returns parameter of type "WorkOrderFilterParameter"

Scenario: Create component lot filter parameter

• WHEN SearchModel contains ComponentLotFilter with lot "LOT-A" for item "ITEM-001"
• THEN the system creates a DataTable with ComponentLotNumber and ItemNumber columns
• AND populates one row with both values
• AND returns parameter of type "ComponentLotFilterParameter"

Scenario: Create empty filter parameter

• WHEN SearchModel contains empty WorkOrderFilter collection
• THEN the system creates a DataTable with WorkOrderNumber column
• AND the DataTable has zero rows
• AND returns valid parameter that produces empty result set

Scenario: Create item operation MIS filter parameter

• WHEN SearchModel contains ItemOperationMisFilter with one entry
• THEN the system creates a DataTable with four columns
• AND populates ItemNumber, OperationNumber, MisNumber, MisRevision
• AND returns parameter of type "ItemOperationMisFilterParameter"

---

Requirement: Output Column Configuration

The system SHALL use attribute-based configuration for Excel output column formatting.

Migration Note: Attributes and Reflection

The attribute-based pattern works well in .NET 10 with standard reflection. For high-performance scenarios, consider:

1. Source generators - Generate column metadata at compile time
2. Cached reflection - Cache PropertyInfo and attribute lookups on first access

// Attribute pattern preserved from legacy (works with records)
[OutputTable(TabName = "Search Results", TableName = "Search_Results")]
public record SearchResult(
    [property: OutputColumn(Order = 10, HeaderText = "Work Order")]
    long WorkOrderNumber,

    [property: OutputColumn(Order = 20, HeaderText = "Status Date",
              Format = OutputColumnAttribute.DATE_FORMAT)]
    DateTime? StatusUpdateDT
);

// Cached reflection helper
public static class OutputColumnCache
{
    private static readonly ConcurrentDictionary<Type, IReadOnlyList<OutputColumn>> _cache = new();

    public static IReadOnlyList<OutputColumn> GetColumns<T>() =>
        _cache.GetOrAdd(typeof(T), type => BuildColumns(type));
}

Inputs

• OutputColumnAttribute on result model properties:
  • Order: Display order (integer)
  • HeaderText: Column header text
  • Format: Excel format string
  • AutoWidth: Whether to auto-size column
  • Width: Manual width when AutoWidth is false
  • WrapText: Whether to wrap text in cells
• OutputTableAttribute on result model classes:
  • TabName: Excel worksheet tab name
  • TableName: Table identifier for styling
  • ShowHeader: Whether to display merged header

Outputs

• OutputColumn objects combining property info with attribute metadata
• Configured Excel worksheets with proper formatting

Business Rules

• Columns are ordered by the Order attribute value (ascending)
• Properties without OutputColumnAttribute are excluded from output
• Standard format "@" treats values as text
• Date format "[$-409]MM/dd/yyyy;@" applies US English date formatting
• Timestamp format "[$-409]m/d/yy h:mm AM/PM;@" includes time component
• Wrapped columns use fixed width of 65 characters by default
• Filter entry tables show headers; result tables do not

Scenario: Configure date column formatting

• WHEN a property has Format = OutputColumnAttribute.DATE_FORMAT
• THEN Excel displays values in MM/dd/yyyy format
• AND the column width auto-sizes to fit content

Scenario: Configure wrapped text column

• WHEN a property has WrapText = true and AutoWidth = false
• THEN Excel wraps long text within the cell
• AND column width is set to WRAPPED_COLUMN_WIDTH (65 characters)

Scenario: Order columns in output

• WHEN SearchResult has properties with Order values 10, 20, 30
• THEN Excel columns appear in ascending order by the Order value
• AND gaps in Order values are allowed

Scenario: Configure tab name for result type

• WHEN SearchResult has OutputTableAttribute with TabName = "Search Results"
• THEN the Excel worksheet tab is named "Search Results"
• AND the table is styled with identifier "Search_Results"

---

Migration Notes

Legacy Pattern	New Pattern	Rationale
T4 Text Template (QueryTemplate.tt)	SqlKata fluent query builder	SqlKata provides parameterized SQL by default, fluent API, composable queries, testability, and SQL Server-optimized output; T4 templates are not well-supported in modern .NET SDK projects
Iterative work order traversal in T4	Stored procedure dbo.TraverseWorkOrders	Iterative logic with temp tables and MERGE is better handled server-side; stored procedure reduces round trips and is transactionally consistent
System.Data.SqlClient	Microsoft.Data.SqlClient	Legacy package is deprecated and no longer receives security updates; Microsoft.Data.SqlClient is the supported replacement
Filter entry classes	C# record types	Records provide immutability, value equality, and cleaner syntax for DTOs
Synchronous Dapper calls	Async Dapper methods (QueryAsync, ExecuteAsync)	Async operations improve scalability and are idiomatic in modern .NET
DataTable.AsTableValuedParameter()	DataTable (unchanged) or IEnumerable<SqlDataRecord>	DataTable approach works; SqlDataRecord is more efficient for large datasets
Conditional string concatenation in T4	SqlKata fluent conditional methods (.Join(), .WhereIn(), .When())	SqlKata handles conditional query building natively with compile-time safety
SqlMapper.ICustomQueryParameter (Dapper)	Dapper table-valued parameters (unchanged)	Dapper's API remains compatible in .NET 10
Attribute-based output configuration	Attributes with optional cached reflection	Attributes work well; cached reflection improves performance
Newtonsoft.Json for criteria serialization	System.Text.Json (recommended) or Newtonsoft.Json	System.Text.Json is built-in with better performance; Newtonsoft.Json works if polymorphism needed

SqlKata Integration

NuGet Packages:

• SqlKata - Core query builder
• SqlKata.Execution - Dapper integration for query execution

DI Registration:

services.AddScoped<ISearchQueryBuilder, SqlKataSearchQueryBuilder>();
services.AddScoped<IWorkOrderTraversalService, WorkOrderTraversalService>();
services.AddSingleton<SqlServerCompiler>();

SqlKata with Dapper Execution:

// Build query with SqlKata
var (sql, parameters) = _queryBuilder.BuildSearchQuery(model);

// Execute with Dapper
var results = await connection.QueryAsync<SearchResult>(sql, parameters);

Design Decisions

Decision: 20-iteration limit for downstream traversal

The 20-iteration limit (@c_MAX_RUNS INT = 20) for downstream work order traversal is retained as a fixed value. Rationale:

• Manufacturing processes rarely exceed 20 levels of downstream consumption
• A configurable limit adds complexity without practical benefit
• The limit protects against runaway queries from circular references

Decision: Circular reference handling

Circular references in work order relationships are handled implicitly by the MERGE statement's ON clause. When a work order is already in #Temp_WO, the MERGE updates rather than inserts, preventing infinite loops. No additional detection logic is required.

Decision: MIS non-match results

MIS non-match results are always included when ExtractMisData = true. The non-match data is valuable for quality investigations and should not be optional.

Decision: MatchMIS function availability

The dbo.MatchMIS table-valued function must be recreated in the migrated database. It is a custom SQL Server function that implements MIS record matching logic based on work order routing, master routing, and operation parameters.

Decision: Filter validation location

Filter validation (e.g., requiring at least one filter) is enforced at the UI/API level, not in the model. This allows the model to represent any valid state while the API layer enforces business rules about minimum filter requirements.

Decision: Large result set handling

Large result sets are handled via streaming to Excel. The EPPlus library supports streaming writes, and the existing pattern of writing directly to the output stream is preserved. Pagination is not implemented at the search level.

---

Codex Review Findings (Spec Accuracy Issues)

The following inaccuracies were identified during review and have been addressed in this specification:

1. MIS Extraction Not Filtered: The spec states MIS extraction joins #Temp_WO or #P_PartOperations to constrain results. CORRECTED: Spec now documents that MIS extraction does NOT join #Temp_WO, so work-order/component-lot/item-operation-MIS filters do NOT constrain MIS results. See QueryTemplate.tt:173,353.

2. Timespan Filter Requires Both Bounds: The spec says either min or max boundary enables filtering. CORRECTED: Spec now documents that the LU_WO timespan filtering requires BOTH min and max values for the combined condition, with separate handling for MIS-only min/max cases. See QueryTemplate.tt:254,371, LotFinderDBExt.cs:157.

3. WorkOrderComponent Flagged as CARDEX: The spec implies PartsList flag is set for WorkOrderComponent paths. CORRECTED: Spec now explicitly states component-lot matches from WorkOrderComponent are flagged as CARDEX, not PartsList. See QueryTemplate.tt:105.

4. ItemOperationMisFilter Skips WorkOrderTime: Not documented. CORRECTED: Spec now documents that when ItemOperationMisFilterEnabled is true, the WorkOrderTime UNION branch is skipped. See QueryTemplate.tt:258.

5. InclusionReason Edge Cases: Spec mentions "Split order" but not "UNKNOWN" case. CORRECTED: Spec now documents that InclusionReason can return "UNKNOWN" when no flags are set. See SearchResult.cs:152.

6. Error Handling Not Documented: Query timeout, debug SQL/Excel writes, and marking search failed on exception are legacy behaviors addressed in the separate error-handling specification. See LotFinderDBExt.cs:136, WorkProcessor.cs:158.