Initial commit: JDE Scoping Tool migration project
Set up repository with legacy .NET Framework 4.8 source (OLD/), new .NET 10 Blazor solution (NEW/), OpenSpec specifications, documentation, and project configuration.
This commit is contained in:
Joseph Doherty
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,858 @@
|
||||
# Data Sync Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
The Data Sync subsystem maintains a local SQL Server cache of enterprise data from JDE (JD Edwards - Oracle) and CMS (Sybase) source systems. Implemented as a .NET 10 `BackgroundService`, it enables fast search operations by synchronizing data on configurable schedules (mass/daily/hourly) and uses incremental updates with MERGE operations to minimize data transfer while keeping the cache current. The service integrates with the ASP.NET Core hosting model, supporting graceful shutdown, health checks, and telemetry.
|
||||
|
||||
## Source Reference
|
||||
|
||||
| Legacy Files | Purpose |
|
||||
|--------------|---------|
|
||||
| `OLD/WorkerService/Process/UpdateProcessor.cs` | Main sync orchestration, schedule checking, update execution |
|
||||
| `OLD/WorkerService/Process/UpdateProcessor.TableManagement.cs` | Staging table creation, MERGE generation, bulk copy, index management |
|
||||
| `OLD/WorkerService/Process/UpdateProcessor.DataUpdateEntry.cs` | Update logging, history tracking, cleanup |
|
||||
| `OLD/WorkerService/dsconfig/*.json` | Per-table sync configuration files |
|
||||
| `OLD/WorkerService/Models/DataSourceConfig.cs` | Configuration model with fetch functions |
|
||||
| `OLD/WorkerService/Models/DataUpdateConfig.cs` | Schedule configuration (interval, prepurge, reindex) |
|
||||
| `OLD/WorkerService/Process/WorkProcessor.cs` | Work loop that triggers sync checks |
|
||||
| `OLD/Database/Views/LastDataUpdates.sql` | View for determining last successful sync per table/type |
|
||||
## Requirements
|
||||
### Requirement: Background Service Lifecycle
|
||||
|
||||
The system SHALL implement data synchronization as a .NET `BackgroundService` with proper lifecycle management.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `CancellationToken` from the host for graceful shutdown signals
|
||||
- `IServiceScopeFactory` for creating scoped services per sync operation
|
||||
- `IOptions<DataSyncOptions>` for configuration
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Long-running background task that processes sync schedules
|
||||
- Graceful shutdown with in-progress operation completion or cancellation
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The service MUST inherit from `BackgroundService` and implement `ExecuteAsync`
|
||||
- The service SHALL respect `CancellationToken` for graceful shutdown
|
||||
- Each sync operation MUST create a new `IServiceScope` via `IServiceScopeFactory`
|
||||
- At startup, the service MUST call `CloseOpenUpdateEntries()` to mark interrupted syncs as failed
|
||||
- The service SHALL call `PurgeUpdateEntries()` periodically to clean old history records
|
||||
- The main loop SHALL use `Task.Delay` with the cancellation token between sync checks
|
||||
|
||||
#### Scenario: Service startup initialization
|
||||
|
||||
- **WHEN** the BackgroundService starts
|
||||
- **THEN** the system SHALL invoke `CloseOpenUpdateEntries()` to mark any `DataUpdate` records with `NumberRecords = -2` as failed
|
||||
- **THEN** the system SHALL begin the main sync check loop
|
||||
|
||||
#### Scenario: Graceful shutdown during sync
|
||||
|
||||
- **WHEN** the host signals shutdown via `CancellationToken`
|
||||
- **AND** a sync operation is in progress
|
||||
- **THEN** the cancellation token SHALL propagate to all child operations
|
||||
- **THEN** the service SHALL wait for current batch completion or cancel gracefully
|
||||
- **THEN** any incomplete syncs SHALL be marked as failed with `WasSuccessful = false`
|
||||
|
||||
#### Scenario: Scoped service creation per sync
|
||||
|
||||
- **WHEN** a sync operation begins
|
||||
- **THEN** the system SHALL create a new `IServiceScope`
|
||||
- **THEN** all services for that sync operation SHALL be resolved from the scope
|
||||
- **THEN** the scope SHALL be disposed after the sync completes or fails
|
||||
|
||||
### Requirement: Strongly-Typed Configuration
|
||||
|
||||
The system SHALL use strongly-typed options classes bound from configuration instead of JSON file parsing with reflection.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `IOptions<DataSyncOptions>` injected via dependency injection
|
||||
- Configuration bound from `appsettings.json` or environment variables
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `DataSyncOptions` containing global sync settings
|
||||
- `DataSourceOptions` containing per-table configuration
|
||||
- Type-resolved `IDataFetcher<T>` implementations
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Configuration SHALL use `IOptions<DataSyncOptions>` pattern instead of JSON file loading
|
||||
- `DataSyncOptions` SHALL define: `MaxDegreeOfParallelism`, `BatchSize`, `BulkCopyBatchSize`, `LookbackMultiplier`, `PurgeRetentionDays`
|
||||
- `DataSourceOptions` SHALL define: `SourceSystem`, `TableName`, `IsEnabled`, `MassConfig`, `DailyConfig`, `HourlyConfig`, `FetcherTypeName`, `PostProcessorTypeName`
|
||||
- Each schedule config (`MassConfig`, `DailyConfig`, `HourlyConfig`) SHALL include an `Enabled` boolean flag for explicit schedule enable/disable control
|
||||
- The `FetcherTypeName` SHALL be resolved to an `IDataFetcher<T>` implementation at startup
|
||||
- The `PostProcessorTypeName` SHALL be resolved to an `IPostProcessor` implementation at startup
|
||||
- Invalid or unresolvable type names SHALL cause startup failure with descriptive error
|
||||
|
||||
#### Scenario: Configuration binding at startup
|
||||
|
||||
- **WHEN** the application starts
|
||||
- **THEN** `DataSyncOptions` SHALL be bound from the `DataSync` configuration section
|
||||
- **THEN** each `DataSourceOptions` entry SHALL be validated for required fields
|
||||
- **THEN** `FetcherTypeName` values SHALL be resolved to registered `IDataFetcher<T>` services
|
||||
|
||||
#### Scenario: Invalid fetcher type configuration
|
||||
|
||||
- **WHEN** a `DataSourceOptions.FetcherTypeName` cannot be resolved to a registered service
|
||||
- **THEN** the system SHALL throw a descriptive exception at startup
|
||||
- **THEN** the error message SHALL include the invalid type name and table name
|
||||
|
||||
### Requirement: Data Fetcher Abstraction
|
||||
|
||||
The system SHALL use `IDataFetcher<TEntity>` interfaces instead of reflection-based delegates for data retrieval.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `DateTime? minimumDT` parameter for incremental fetches
|
||||
- `CancellationToken` for cancellation support
|
||||
- Source system connection (JDE Oracle or CMS Sybase)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `IAsyncEnumerable<TEntity>` streaming data from source systems
|
||||
- Support for cancellation during long-running fetches
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each data source MUST have a corresponding `IDataFetcher<TEntity>` implementation
|
||||
- The `FetchAsync` method SHALL return `IAsyncEnumerable<TEntity>` for memory-efficient streaming
|
||||
- All fetch operations MUST accept and respect `CancellationToken`
|
||||
- JDE fetchers SHALL use `Oracle.ManagedDataAccess.Core` connections
|
||||
- CMS fetchers SHALL use `Oracle.ManagedDataAccess.Core` connections (CMS uses Oracle via legacy DDTek driver, consolidated in migration)
|
||||
- Initial implementation MAY use stub fetchers that return empty `IAsyncEnumerable<T>` streams while JDE/CMS connectivity is deferred
|
||||
- Stub fetchers SHALL implement `IDataFetcher<T>` interface with `yield break` to enable testing without external dependencies
|
||||
|
||||
#### Scenario: Streaming data fetch
|
||||
|
||||
- **WHEN** a sync operation requests data from a source system
|
||||
- **THEN** the system SHALL call `IDataFetcher<T>.FetchAsync(minimumDT, cancellationToken)`
|
||||
- **THEN** data SHALL stream via `IAsyncEnumerable<T>` without loading all records into memory
|
||||
- **THEN** cancellation SHALL stop the enumeration gracefully
|
||||
|
||||
#### Scenario: Cancellation during fetch
|
||||
|
||||
- **WHEN** the cancellation token is triggered during a fetch operation
|
||||
- **THEN** the async enumerable SHALL stop yielding records
|
||||
- **THEN** database resources SHALL be properly disposed
|
||||
- **THEN** the sync operation SHALL be marked as failed
|
||||
|
||||
### Requirement: Health Checks
|
||||
|
||||
The system SHALL expose health check endpoints for monitoring sync status.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `IHealthCheck` registration with ASP.NET Core health checks
|
||||
- Current sync state and last successful sync timestamps
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Health status: Healthy, Degraded, or Unhealthy
|
||||
- Diagnostic data including last sync times and any error messages
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The health check SHALL report `Healthy` when all enabled tables have synced within their configured intervals
|
||||
- The health check SHALL report `Degraded` when any table is overdue but syncs are progressing
|
||||
- The health check SHALL report `Unhealthy` when syncs have been failing repeatedly or the service is not running
|
||||
- Health check response SHALL include per-table sync status and timestamps
|
||||
|
||||
#### Scenario: All syncs current
|
||||
|
||||
- **WHEN** health check executes
|
||||
- **AND** all enabled tables have successful syncs within their intervals
|
||||
- **THEN** the check SHALL return `Healthy` status
|
||||
- **THEN** response SHALL include last sync timestamps per table
|
||||
|
||||
#### Scenario: Overdue syncs with progress
|
||||
|
||||
- **WHEN** health check executes
|
||||
- **AND** some tables are overdue for sync
|
||||
- **AND** sync operations are currently running or recently completed
|
||||
- **THEN** the check SHALL return `Degraded` status
|
||||
- **THEN** response SHALL identify which tables are overdue
|
||||
|
||||
#### Scenario: Repeated failures
|
||||
|
||||
- **WHEN** health check executes
|
||||
- **AND** multiple recent sync operations have failed
|
||||
- **THEN** the check SHALL return `Unhealthy` status
|
||||
- **THEN** response SHALL include error details from failed syncs
|
||||
|
||||
### Requirement: Telemetry and Metrics
|
||||
|
||||
The system SHALL emit metrics and traces for observability.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `System.Diagnostics.Metrics` meter for metrics
|
||||
- `System.Diagnostics.ActivitySource` for distributed tracing
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Counters: sync operations started, completed, failed
|
||||
- Histograms: sync duration, records processed
|
||||
- Activity spans for distributed tracing
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The service SHALL create a `Meter` named `DataSync`
|
||||
- The service SHALL emit counters for: `sync.operations.started`, `sync.operations.completed`, `sync.operations.failed`
|
||||
- The service SHALL emit histograms for: `sync.duration.seconds`, `sync.records.processed`
|
||||
- Each sync operation SHALL create an `Activity` span with tags for table name, update type, and source system
|
||||
- Activity spans SHALL include record count and duration on completion
|
||||
|
||||
#### Scenario: Sync operation telemetry
|
||||
|
||||
- **WHEN** a sync operation starts
|
||||
- **THEN** the system SHALL increment `sync.operations.started` counter
|
||||
- **THEN** the system SHALL start an `Activity` span with table and type tags
|
||||
|
||||
- **WHEN** a sync operation completes successfully
|
||||
- **THEN** the system SHALL increment `sync.operations.completed` counter
|
||||
- **THEN** the system SHALL record duration in `sync.duration.seconds` histogram
|
||||
- **THEN** the system SHALL record count in `sync.records.processed` histogram
|
||||
- **THEN** the Activity span SHALL be completed with success status
|
||||
|
||||
- **WHEN** a sync operation fails
|
||||
- **THEN** the system SHALL increment `sync.operations.failed` counter
|
||||
- **THEN** the Activity span SHALL be completed with error status and exception details
|
||||
|
||||
### Requirement: Schedule-Based Sync Triggering
|
||||
|
||||
The system SHALL support three distinct sync schedule types: Mass, Daily, and Hourly, each with independent intervals and behaviors per table.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Data source configuration via `IOptions<DataSyncOptions>`
|
||||
- `LastDataUpdates` view providing timestamps of last successful syncs
|
||||
- Current system time
|
||||
|
||||
#### Outputs
|
||||
|
||||
- List of pending `DataUpdateTask` objects requiring execution
|
||||
- Each task specifies: target table, update type, and minimum timestamp for incremental fetches
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Mass updates SHALL trigger when no prior successful mass update exists OR when the configured mass interval has elapsed since the last mass update
|
||||
- Daily updates SHALL trigger when mass is current AND daily interval has elapsed since last daily update
|
||||
- Hourly updates SHALL trigger when mass and daily are current AND hourly interval has elapsed since last hourly update
|
||||
- Schedule priority SHALL be: Mass > Daily > Hourly (mass takes precedence)
|
||||
- Incremental updates (Daily/Hourly) SHALL use a configurable lookback window (default 3x) of the interval to capture delayed records
|
||||
- Hourly incremental updates use the last Daily sync timestamp with the lookback multiplier applied to the Daily interval (not Hourly interval)
|
||||
- Only tables with `IsEnabled = true` AND the specific schedule enabled SHALL be considered for automatic sync
|
||||
|
||||
#### Scenario: Initial system startup with no prior syncs
|
||||
|
||||
- **WHEN** the system starts and no `DataUpdate` records exist for a table
|
||||
- **AND** the table has `IsEnabled = true` and `MassConfig.Enabled = true`
|
||||
- **THEN** the system SHALL queue a Mass update task for that table
|
||||
- **THEN** the `MinimumDT` parameter SHALL be null (full data fetch)
|
||||
|
||||
#### Scenario: Mass sync interval elapsed
|
||||
|
||||
- **WHEN** a table's last successful Mass update occurred more than `MassConfig.Interval` minutes ago
|
||||
- **THEN** the system SHALL queue a Mass update task for that table
|
||||
- **THEN** any pending Daily or Hourly updates for that table SHALL be superseded
|
||||
|
||||
#### Scenario: Daily sync triggers after mass is current
|
||||
|
||||
- **WHEN** a table's Mass update is current (within interval)
|
||||
- **AND** the last Daily update occurred more than `DailyConfig.Interval` minutes ago
|
||||
- **THEN** the system SHALL queue a Daily update task
|
||||
- **THEN** the `MinimumDT` SHALL be set to `LastDailyUpdateDT - (LookbackMultiplier * DailyInterval)` minutes
|
||||
|
||||
#### Scenario: Hourly sync with lookback window
|
||||
|
||||
- **WHEN** an Hourly update is triggered
|
||||
- **THEN** the system SHALL fetch records modified since `MinimumDT`
|
||||
- **AND** `MinimumDT` SHALL equal `LastDailyUpdateDT - (LookbackMultiplier * DailyInterval)` minutes (using Daily timestamp, not Hourly)
|
||||
|
||||
#### Scenario: Disabled table not scheduled
|
||||
|
||||
- **WHEN** a table has `IsEnabled = false` OR all schedule configs have `Enabled = false`
|
||||
- **THEN** the table SHALL NOT be automatically scheduled for sync
|
||||
- **THEN** syncs MAY only occur via explicit manual trigger through the admin API
|
||||
|
||||
### Requirement: Data Source Configuration
|
||||
|
||||
The system SHALL load and validate data source configurations defining sync behavior per table.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `DataSourceOptions` entries within `DataSyncOptions`
|
||||
- Each entry specifies: SourceSystem, TableName, FetcherTypeName, PostProcessorTypeName, and schedule configs
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Validated `DataSourceOptions` with resolved service types
|
||||
- Only configurations with `IsEnabled = true` are active
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each data source MUST specify a `FetcherTypeName` that resolves to an `IDataFetcher<T>` implementation
|
||||
- `PostProcessorTypeName` is optional and specifies an `IPostProcessor` implementation
|
||||
- Standard intervals SHALL be: Mass = 10080 minutes (7 days), Daily = 1440 minutes (24 hours), Hourly = 60 minutes
|
||||
- CMS data sources MAY have different intervals (e.g., MisData uses Mass = 100800 minutes / 70 days)
|
||||
- Archive tables MAY disable all schedules and require manual triggering via admin API
|
||||
|
||||
#### Scenario: Configuration validation at startup
|
||||
|
||||
- **WHEN** the service starts
|
||||
- **THEN** all `DataSourceOptions` entries SHALL be validated
|
||||
- **THEN** `FetcherTypeName` values SHALL be resolved to registered services
|
||||
- **THEN** only configurations with `IsEnabled = true` SHALL be added to the active configs list
|
||||
|
||||
#### Scenario: Disabled archive table configuration
|
||||
|
||||
- **WHEN** a configuration has `MassConfig.Enabled = false`, `DailyConfig.Enabled = false`, and `HourlyConfig.Enabled = false`
|
||||
- **THEN** the table SHALL never be automatically scheduled for sync
|
||||
- **THEN** syncs MAY only occur via explicit manual trigger through the admin API
|
||||
|
||||
#### Scenario: Post-processing action execution
|
||||
|
||||
- **WHEN** a data source specifies a `PostProcessorTypeName`
|
||||
- **AND** the data merge completes successfully
|
||||
- **THEN** the system SHALL resolve and invoke the `IPostProcessor.ProcessAsync()` method
|
||||
- **THEN** the update SHALL only be marked complete after post-processing succeeds
|
||||
|
||||
#### Scenario: CMS vs JDE source configuration
|
||||
|
||||
- **WHEN** a data source has `SourceSystem = "CMS"`
|
||||
- **THEN** the `FetcherTypeName` SHALL reference a CMS-specific `IDataFetcher<T>` implementation
|
||||
- **WHEN** a data source has `SourceSystem = "JDE"`
|
||||
- **THEN** the `FetcherTypeName` SHALL reference a JDE-specific `IDataFetcher<T>` implementation
|
||||
|
||||
### Requirement: Table Management and Merge Operations
|
||||
|
||||
The system SHALL use staging tables and SQL MERGE operations to efficiently upsert data while preserving existing records.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Source data from `IDataFetcher<T>.FetchAsync()` execution
|
||||
- Destination table schema (columns, primary key, indexes)
|
||||
- Update configuration (PrepurgeData, ReIndexData flags)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Updated destination table with merged data
|
||||
- Rebuilt indexes (if configured)
|
||||
- Staging and temp tables cleaned up
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Mass updates with `PrepurgeData = true` SHALL TRUNCATE the destination table before loading
|
||||
- Incremental updates (Daily/Hourly) SHALL use MERGE to upsert without deleting existing records
|
||||
- Data SHALL be batched in groups of 1,000,000 records for bulk copy operations
|
||||
- Bulk copy SHALL use batch size of 10,000 rows with streaming enabled
|
||||
- Staging tables SHALL be named `#Staging{TableName}_{OperationId}` (local temp tables with unique suffix for parallel isolation)
|
||||
- Temp tables SHALL be named `#{TableName}_{OperationId}` (local temp tables with unique suffix)
|
||||
- MERGE SHALL update existing records only when `LastUpdateDT` in source is greater than target (if column exists)
|
||||
- Tables without `LastUpdateDT` column SHALL update all matched rows unconditionally
|
||||
- Non-primary-key indexes SHALL be disabled during bulk load and rebuilt after
|
||||
|
||||
#### Scenario: Mass update with table truncation
|
||||
|
||||
- **WHEN** a Mass update executes with `PrepurgeData = true`
|
||||
- **THEN** the destination table SHALL be truncated before data load
|
||||
- **THEN** all records from source SHALL be inserted
|
||||
- **THEN** indexes SHALL be rebuilt if `ReIndexData = true`
|
||||
|
||||
#### Scenario: Incremental update with MERGE
|
||||
|
||||
- **WHEN** a Daily or Hourly update executes
|
||||
- **THEN** the system SHALL create a staging table matching destination schema with unique suffix
|
||||
- **THEN** source data SHALL be bulk copied to staging table
|
||||
- **THEN** data SHALL be deduplicated into temp table using `ROW_NUMBER() OVER(PARTITION BY PK ORDER BY LastUpdateDT DESC)`
|
||||
- **THEN** MERGE SHALL insert new records and update existing records where source `LastUpdateDT > target.LastUpdateDT`
|
||||
|
||||
#### Scenario: Table without LastUpdateDT column
|
||||
|
||||
- **WHEN** MERGE executes on a table without `LastUpdateDT` column
|
||||
- **THEN** all matched rows SHALL be updated unconditionally
|
||||
- **THEN** the `ReleaseDate` column (if present) SHALL only be used for ORDER BY in deduplication, not for update filtering
|
||||
|
||||
#### Scenario: Large dataset batching
|
||||
|
||||
- **WHEN** the data fetch streams more than 1,000,000 records
|
||||
- **THEN** records SHALL be processed in batches of 1,000,000
|
||||
- **THEN** each batch SHALL create fresh staging/temp tables with unique suffixes
|
||||
- **THEN** each batch SHALL execute MERGE independently
|
||||
- **THEN** total record count SHALL accumulate across all batches
|
||||
|
||||
#### Scenario: Index management during bulk load
|
||||
|
||||
- **WHEN** staging table is created
|
||||
- **THEN** an index SHALL be created on primary key columns plus `LastUpdateDT` (or `ReleaseDate`)
|
||||
- **THEN** non-PK, non-unique indexes SHALL be disabled before bulk copy
|
||||
- **THEN** indexes SHALL be rebuilt after bulk copy completes
|
||||
|
||||
### Requirement: Update Logging and Recovery
|
||||
|
||||
The system SHALL log all sync operations and support recovery from interrupted syncs.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `DataUpdate` table for recording sync history
|
||||
- `LastDataUpdates` view for querying last successful syncs
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Complete audit trail of all sync operations
|
||||
- Automatic recovery of interrupted syncs
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each sync operation MUST create a `DataUpdate` record at start with `NumberRecords = -2` (in-progress marker)
|
||||
- The sync operation MUST be wrapped in try/catch to ensure failed operations are marked properly
|
||||
- Successful completion SHALL update `EndDT`, `WasSuccessful = true`, and actual `NumberRecords`
|
||||
- Failed operations SHALL set `WasSuccessful = false` and `NumberRecords = -1`
|
||||
- Open entries (NumberRecords = -2) from prior runs SHALL be closed as failed at service startup via `CloseOpenUpdateEntries()`
|
||||
- Old `DataUpdate` records SHALL be purged periodically via `PurgeUpdateEntries()` after configurable retention period
|
||||
- All logging SHALL use `ILogger<T>` with `BeginScope()` for structured context (table name, update type, operation ID)
|
||||
|
||||
#### Scenario: Sync operation start logging
|
||||
|
||||
- **WHEN** a sync operation begins
|
||||
- **THEN** a `DataUpdate` record SHALL be inserted with `NumberRecords = -2`
|
||||
- **THEN** the record SHALL include SourceSystem, SourceData, TableName, UpdateType, StartDT
|
||||
- **THEN** the operation SHALL create a logging scope with table name and operation ID
|
||||
|
||||
#### Scenario: Successful sync completion
|
||||
|
||||
- **WHEN** a sync operation completes without errors
|
||||
- **THEN** the `DataUpdate` record SHALL be updated with `EndDT = GETDATE()`
|
||||
- **THEN** `WasSuccessful` SHALL be set to `true`
|
||||
- **THEN** `NumberRecords` SHALL reflect the total rows processed
|
||||
|
||||
#### Scenario: Failed sync handling
|
||||
|
||||
- **WHEN** a sync operation throws an exception
|
||||
- **THEN** the exception SHALL be caught in the operation wrapper
|
||||
- **THEN** the `DataUpdate` record SHALL be updated with `WasSuccessful = false`, `NumberRecords = -1`
|
||||
- **THEN** the error SHALL be logged via `ILogger<T>` with full exception details
|
||||
- **THEN** subsequent sync attempts SHALL retry the operation
|
||||
|
||||
#### Scenario: Recovery from interrupted sync at startup
|
||||
|
||||
- **WHEN** the service starts and finds `DataUpdate` records with `NumberRecords = -2`
|
||||
- **THEN** `CloseOpenUpdateEntries()` SHALL update those records to `EndDT = GETDATE()`, `WasSuccessful = false`, `NumberRecords = -1`
|
||||
- **THEN** the system SHALL treat those tables as needing fresh sync based on last successful update
|
||||
|
||||
#### Scenario: Periodic history purge
|
||||
|
||||
- **WHEN** `PurgeUpdateEntries()` executes
|
||||
- **THEN** `DataUpdate` records older than `PurgeRetentionDays` SHALL be deleted
|
||||
- **THEN** the purge SHALL run periodically (e.g., daily) independent of sync operations
|
||||
|
||||
### Requirement: Parallel Sync Execution
|
||||
|
||||
The system SHALL execute multiple table syncs in parallel to optimize throughput with proper cancellation support.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- List of pending `DataUpdateTask` objects
|
||||
- `MaxDegreeOfParallelism` from `DataSyncOptions`
|
||||
- `CancellationToken` for cancellation support
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Concurrent execution of sync operations
|
||||
- Proper isolation between parallel syncs
|
||||
- Graceful cancellation of parallel operations
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Pending updates SHALL be executed in parallel using `Parallel.ForEachAsync` or `SemaphoreSlim` with `Task.WhenAll`
|
||||
- Maximum degree of parallelism SHALL be configurable (default = 8)
|
||||
- Each sync operation MUST use its own `IServiceScope` for scoped service resolution
|
||||
- Each sync operation MUST use its own database connection
|
||||
- Staging tables MUST use unique suffixes (`_{OperationId}`) to avoid conflicts in parallel scenarios
|
||||
- `CancellationToken` MUST be passed to all parallel operations
|
||||
- Search processing is blocked while any sync operations are pending
|
||||
|
||||
#### Scenario: Multiple tables need sync
|
||||
|
||||
- **WHEN** multiple tables have pending sync tasks
|
||||
- **THEN** the system SHALL execute up to `MaxDegreeOfParallelism` sync operations concurrently
|
||||
- **THEN** each operation SHALL create its own `IServiceScope`
|
||||
- **THEN** each operation SHALL use independent SQL connections
|
||||
- **THEN** completion of one operation SHALL not affect others
|
||||
|
||||
#### Scenario: Cancellation during parallel sync
|
||||
|
||||
- **WHEN** cancellation is requested during parallel sync execution
|
||||
- **THEN** the `CancellationToken` SHALL propagate to all running operations
|
||||
- **THEN** operations SHALL check the token and exit gracefully
|
||||
- **THEN** incomplete operations SHALL be marked as failed
|
||||
|
||||
#### Scenario: Sync blocks search processing
|
||||
|
||||
- **WHEN** the work processor checks for pending sync tasks
|
||||
- **AND** pending tasks exist
|
||||
- **THEN** sync operations SHALL execute before processing any queued searches
|
||||
- **THEN** search processing SHALL only begin when no sync tasks remain pending
|
||||
|
||||
#### Scenario: Sync with isolated resources
|
||||
|
||||
- **WHEN** multiple sync operations run in parallel
|
||||
- **THEN** each operation SHALL create staging tables with unique suffixes
|
||||
- **THEN** each operation SHALL use its own scoped database connection
|
||||
- **THEN** no shared mutable state SHALL exist between parallel operations
|
||||
|
||||
### Requirement: CMS Availability and Circuit Breaker
|
||||
|
||||
The system SHALL handle CMS (Sybase) connectivity issues with circuit breaker pattern.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- CMS connection state
|
||||
- Recent CMS sync failure history
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Automatic retry with backoff for transient failures
|
||||
- Circuit breaker to prevent repeated failed connection attempts
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- CMS connections SHALL use Polly or similar circuit breaker pattern
|
||||
- The circuit SHALL open after consecutive failures (configurable, default = 3)
|
||||
- The circuit SHALL remain open for a configurable duration (default = 5 minutes)
|
||||
- Health checks SHALL report CMS circuit state
|
||||
- JDE syncs SHALL continue independently of CMS circuit state
|
||||
|
||||
#### Scenario: CMS transient failure
|
||||
|
||||
- **WHEN** a CMS sync fails with a transient error
|
||||
- **THEN** the system SHALL retry with exponential backoff
|
||||
- **THEN** the failure count SHALL increment
|
||||
|
||||
#### Scenario: Circuit breaker opens
|
||||
|
||||
- **WHEN** consecutive CMS sync failures exceed threshold
|
||||
- **THEN** the circuit breaker SHALL open
|
||||
- **THEN** subsequent CMS sync attempts SHALL fail fast without attempting connection
|
||||
- **THEN** JDE syncs SHALL continue normally
|
||||
|
||||
#### Scenario: Circuit breaker recovery
|
||||
|
||||
- **WHEN** the circuit breaker open duration elapses
|
||||
- **THEN** the circuit SHALL transition to half-open state
|
||||
- **THEN** the next CMS sync attempt SHALL be allowed
|
||||
- **THEN** success SHALL close the circuit; failure SHALL reopen it
|
||||
|
||||
### Requirement: Archive Sync Manual Trigger
|
||||
|
||||
The system SHALL support manual triggering of archive table syncs via admin API.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- HTTP request to admin API endpoint
|
||||
- Table name and optional update type parameters
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Queued sync task for the specified archive table
|
||||
- Status response indicating task queued
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Archive tables with all schedules disabled SHALL only sync via manual trigger
|
||||
- The admin API endpoint SHALL require authentication and authorization
|
||||
- Manual triggers SHALL queue a Mass update task for the specified table
|
||||
- The system SHALL return immediate acknowledgment; sync runs asynchronously
|
||||
|
||||
#### Scenario: Manual archive sync trigger
|
||||
|
||||
- **WHEN** an authenticated admin calls the manual sync API for an archive table
|
||||
- **THEN** a Mass update task SHALL be queued for that table
|
||||
- **THEN** the API SHALL return 202 Accepted with task ID
|
||||
- **THEN** the sync SHALL execute in the background service
|
||||
|
||||
### Requirement: Periodic Index Maintenance
|
||||
|
||||
The system SHALL support periodic index maintenance independent of mass syncs.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Index maintenance configuration (schedule, tables)
|
||||
- Current table statistics
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Rebuilt or reorganized indexes
|
||||
- Updated statistics
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Index maintenance MAY be configured to run on a schedule independent of mass syncs
|
||||
- Maintenance SHALL check index fragmentation before rebuilding
|
||||
- Indexes with fragmentation > 30% SHALL be rebuilt; 10-30% SHALL be reorganized
|
||||
- Statistics SHALL be updated after index maintenance
|
||||
- Maintenance operations SHALL be logged for audit
|
||||
|
||||
#### Scenario: Scheduled index maintenance
|
||||
|
||||
- **WHEN** the index maintenance schedule triggers
|
||||
- **THEN** the system SHALL check fragmentation levels for configured tables
|
||||
- **THEN** highly fragmented indexes SHALL be rebuilt
|
||||
- **THEN** moderately fragmented indexes SHALL be reorganized
|
||||
- **THEN** table statistics SHALL be updated
|
||||
|
||||
### Requirement: Background service implementation pattern
|
||||
|
||||
The system SHALL implement the data synchronization service following .NET BackgroundService best practices for hosted service lifecycle management.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `IServiceScopeFactory` for creating scoped service instances
|
||||
- `IOptions<DataSyncOptions>` for configuration access
|
||||
- `ILogger<DataSyncService>` for structured logging
|
||||
- `CancellationToken` from `ExecuteAsync` stoppingToken parameter
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Continuously running background task that checks schedules and executes syncs
|
||||
- Proper cleanup on shutdown with all resources disposed
|
||||
- Logging scope context for all operations
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The service MUST implement `BackgroundService.ExecuteAsync(CancellationToken)`
|
||||
- The main loop MUST use `Task.Delay(checkInterval, stoppingToken)` between cycles
|
||||
- Each sync cycle MUST create a new `IServiceScope` via `IServiceScopeFactory.CreateAsyncScope()`
|
||||
- All scoped services MUST be resolved from the current scope, not from root provider
|
||||
- The scope MUST be disposed using `await using` pattern after each cycle
|
||||
- Exception handling MUST catch and log errors without crashing the service
|
||||
- `OperationCanceledException` MUST be caught and result in graceful loop exit when `stoppingToken.IsCancellationRequested`
|
||||
- The service MUST NOT use static state or shared mutable collections
|
||||
|
||||
#### Scenario: Normal sync cycle execution
|
||||
|
||||
- **WHEN** the BackgroundService enters ExecuteAsync
|
||||
- **THEN** the service SHALL call CloseOpenUpdateEntriesAsync to recover from prior crashes
|
||||
- **THEN** the service SHALL enter a while loop checking `!stoppingToken.IsCancellationRequested`
|
||||
- **THEN** each iteration SHALL create a new IServiceScope
|
||||
- **THEN** the ISyncOrchestrator SHALL be resolved from the scope
|
||||
- **THEN** ExecutePendingSyncsAsync SHALL be called with the stoppingToken
|
||||
- **THEN** the scope SHALL be disposed after the call completes
|
||||
- **THEN** Task.Delay SHALL pause before the next iteration
|
||||
|
||||
#### Scenario: Exception during sync cycle
|
||||
|
||||
- **WHEN** an exception occurs during sync execution (not OperationCanceledException)
|
||||
- **THEN** the exception SHALL be caught and logged with LogError
|
||||
- **THEN** the service SHALL continue to the next iteration
|
||||
- **THEN** the current scope SHALL still be disposed properly
|
||||
- **THEN** the service SHALL NOT crash or stop unexpectedly
|
||||
|
||||
#### Scenario: Graceful shutdown request
|
||||
|
||||
- **WHEN** the host signals shutdown by canceling the stoppingToken
|
||||
- **THEN** any running Task.Delay SHALL throw OperationCanceledException
|
||||
- **THEN** the while loop SHALL exit on the IsCancellationRequested check
|
||||
- **THEN** the ExecuteAsync method SHALL complete normally
|
||||
- **THEN** any in-progress sync operations SHALL receive the cancellation and complete or cancel
|
||||
|
||||
### Requirement: Parallel fetch isolation with scoped resources
|
||||
|
||||
The system SHALL ensure complete isolation between parallel sync operations using scoped resources and unique identifiers.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- List of `DataUpdateTask` objects to execute in parallel
|
||||
- `MaxDegreeOfParallelism` configuration value
|
||||
- `CancellationToken` for coordinated cancellation
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Concurrent execution of sync operations with no resource conflicts
|
||||
- Unique staging tables per operation that do not collide
|
||||
- Independent database connections per operation
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- `Parallel.ForEachAsync` MUST be used with `ParallelOptions.CancellationToken` set
|
||||
- Each parallel task MUST create its own `IServiceScope` inside the parallel delegate
|
||||
- Database connections MUST NOT be shared across parallel operations
|
||||
- Staging table names MUST include a unique `OperationId` suffix (GUID or sequential ID)
|
||||
- Format: `#Staging{TableName}_{OperationId}` and `#{TableName}_{OperationId}`
|
||||
- Each parallel operation MUST resolve its own instances of all scoped services
|
||||
- No `ConcurrentDictionary`, shared counters, or other shared mutable state SHALL exist between operations
|
||||
- Total record counts SHALL be accumulated via return values, not shared state
|
||||
|
||||
#### Scenario: Parallel sync with isolated scopes
|
||||
|
||||
- **WHEN** multiple DataUpdateTasks are executed via Parallel.ForEachAsync
|
||||
- **THEN** each task SHALL execute the async delegate independently
|
||||
- **THEN** each delegate SHALL create a new IServiceScope using CreateAsyncScope
|
||||
- **THEN** ITableSyncOperation SHALL be resolved from each scope independently
|
||||
- **THEN** each operation SHALL use its own database connection from the scope
|
||||
- **THEN** staging tables SHALL use unique OperationId suffixes preventing name collisions
|
||||
- **THEN** completion of one operation SHALL NOT affect the execution of others
|
||||
|
||||
#### Scenario: Parallel cancellation propagation
|
||||
|
||||
- **WHEN** cancellation is requested during Parallel.ForEachAsync execution
|
||||
- **THEN** the CancellationToken SHALL propagate to all running parallel operations
|
||||
- **THEN** Parallel.ForEachAsync SHALL stop starting new operations
|
||||
- **THEN** running operations SHALL receive the token in their async methods
|
||||
- **THEN** each operation SHALL check the token and exit gracefully
|
||||
- **THEN** incomplete operations SHALL mark their DataUpdate records as failed
|
||||
|
||||
#### Scenario: Staging table uniqueness verification
|
||||
|
||||
- **WHEN** two sync operations for the same table run in parallel
|
||||
- **THEN** each operation SHALL generate a unique OperationId as GUID
|
||||
- **THEN** operation A SHALL create staging table with GuidA suffix
|
||||
- **THEN** operation B SHALL create staging table with GuidB suffix
|
||||
- **THEN** no SQL errors SHALL occur from table name conflicts
|
||||
- **THEN** each operation cleanup SHALL only drop its own staging tables
|
||||
|
||||
### Requirement: Structured logging context
|
||||
|
||||
The system SHALL use ILogger.BeginScope to attach contextual information to all log entries during sync operations.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `ILogger<T>` injected into sync operation classes
|
||||
- TableName, UpdateType, OperationId values from current operation
|
||||
|
||||
#### Outputs
|
||||
|
||||
- All log entries within the scope contain the contextual properties
|
||||
- Log aggregation systems can filter and group by table, type, or operation
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each sync operation MUST call `_logger.BeginScope(...)` at the start
|
||||
- The scope MUST include at minimum: TableName, UpdateType, OperationId
|
||||
- The scope MUST be disposed using `using` statement when operation completes
|
||||
- Nested scopes for batches SHALL preserve parent scope properties
|
||||
- LogInformation, LogWarning, LogError calls within the scope SHALL include the context automatically
|
||||
|
||||
#### Scenario: Log scope creation and usage
|
||||
|
||||
- **WHEN** a TableSyncOperation begins execution
|
||||
- **THEN** the operation SHALL create a logging scope with TableName, UpdateType, OperationId
|
||||
- **THEN** all log calls within ExecuteAsync SHALL include these properties
|
||||
- **THEN** when the operation completes the scope SHALL be disposed
|
||||
- **THEN** subsequent operations SHALL have their own independent scopes
|
||||
|
||||
## Data Source Configurations
|
||||
|
||||
### Current/Transactional Tables (Full Schedule)
|
||||
|
||||
| Table | Source | Mass Interval | Daily Interval | Hourly Interval |
|
||||
|-------|--------|---------------|----------------|-----------------|
|
||||
| WorkOrder_Curr | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| LotUsage_Curr | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| WorkOrderTime_Curr | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| WorkOrderStep_Curr | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| WorkOrderComponent_Curr | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| WorkOrderRouting | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
|
||||
### Reference Tables (Full Schedule)
|
||||
|
||||
| Table | Source | Mass Interval | Daily Interval | Hourly Interval |
|
||||
|-------|--------|---------------|----------------|-----------------|
|
||||
| Item | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| Lot | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| WorkCenter | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| ProfitCenter | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| Branch | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| JdeUser | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| StatusCode | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| FunctionCode | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| OrgHierarchy | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
| RouteMaster | JDE | 10080 min (7d) | 1440 min (24h) | 60 min |
|
||||
|
||||
### CMS Tables
|
||||
|
||||
| Table | Source | Mass Interval | Daily Interval | Hourly Interval | Notes |
|
||||
|-------|--------|---------------|----------------|-----------------|-------|
|
||||
| MisData | CMS | 100800 min (70d) | 1440 min (24h) | Disabled | Has PostProcessor |
|
||||
|
||||
### Archive Tables (Disabled - Manual Trigger via Admin API)
|
||||
|
||||
| Table | Source | Notes |
|
||||
|-------|--------|-------|
|
||||
| WorkOrder_Hist | JDE | All schedules disabled |
|
||||
| LotUsage_Hist | JDE | All schedules disabled |
|
||||
| WorkOrderStep_Hist | JDE | All schedules disabled |
|
||||
| WorkOrderTime_Hist | JDE | All schedules disabled |
|
||||
| WorkOrderComponent_Hist | JDE | All schedules disabled |
|
||||
|
||||
## Migration Notes
|
||||
|
||||
| Legacy Pattern | New Pattern | Rationale |
|
||||
|----------------|-------------|-----------|
|
||||
| `Topshelf` Windows Service | .NET `BackgroundService` | Native .NET hosting, cross-platform support |
|
||||
| `ManualResetEvent` for shutdown | `CancellationToken` | Standard .NET cancellation pattern |
|
||||
| `Thread` while loop | `BackgroundService.ExecuteAsync` with `Task.Delay` | Proper async/await, no thread blocking |
|
||||
| `Parallel.ForEach` with `MaxDegreeOfParallelism` | `Parallel.ForEachAsync` or `SemaphoreSlim` with `Task.WhenAll` | Modern async patterns, cancellation support |
|
||||
| JSON config files + `Newtonsoft.Json` | `System.Text.Json` + `IOptions<T>` pattern | Built-in JSON support, configuration binding |
|
||||
| `FunctionConverter` reflection-based delegates | `IDataFetcher<T>` interfaces | Type safety, dependency injection, testability |
|
||||
| `ActionConverter` reflection-based delegates | `IPostProcessor` interfaces | Type safety, dependency injection, testability |
|
||||
| Static `UpdateProcessor` class | Scoped/singleton services with DI | Testability, proper lifecycle management |
|
||||
| NLog | `ILogger<T>` injected + `BeginScope()` for context | Framework-integrated logging, structured context |
|
||||
| Global temp tables `##staging_*` | Local temp tables `#Staging{Table}_{OperationId}` | Better isolation in parallel scenarios |
|
||||
| `System.Data.SqlClient` | `Microsoft.Data.SqlClient` | Modern SQL Server driver with better performance |
|
||||
| Manual SQL MERGE generation | Continue with Dapper + manual MERGE | Performance critical, maintain fine control |
|
||||
| No health checks | `IHealthCheck` implementation | Kubernetes/container orchestration support |
|
||||
| No metrics/tracing | `System.Diagnostics.Metrics` + `ActivitySource` | Observability, distributed tracing |
|
||||
|
||||
## Resolved Design Decisions
|
||||
|
||||
### Archive Sync Strategy
|
||||
|
||||
**Decision**: Archive tables will be synced via manual trigger through an authenticated admin API endpoint.
|
||||
|
||||
**Rationale**: Archive data changes infrequently and full syncs are expensive. Manual triggering allows administrators to control when these resource-intensive operations occur.
|
||||
|
||||
### CMS Availability Handling
|
||||
|
||||
**Decision**: Use circuit breaker pattern (Polly) for CMS connections with configurable failure threshold and open duration.
|
||||
|
||||
**Rationale**: CMS (Sybase) may have different availability characteristics than JDE. Circuit breaker prevents cascading failures and allows JDE syncs to continue independently.
|
||||
|
||||
### Post-Processing Migration
|
||||
|
||||
**Decision**: Replace reflection-based `PostProcessingAction` with `IPostProcessor` interfaces resolved via DI.
|
||||
|
||||
**Rationale**: Type-safe interfaces enable compile-time checking, better testability, and clearer contracts. DI resolution allows for proper scoping and dependency management.
|
||||
|
||||
### Lookback Window Configuration
|
||||
|
||||
**Decision**: Make lookback multiplier configurable via `DataSyncOptions.LookbackMultiplier` (default = 3).
|
||||
|
||||
**Rationale**: Different environments may need different lookback windows based on data arrival patterns. Configuration allows tuning without code changes.
|
||||
|
||||
### Index Rebuild Strategy
|
||||
|
||||
**Decision**: Add periodic index maintenance independent of mass syncs, checking fragmentation before rebuilding.
|
||||
|
||||
**Rationale**: Mass syncs may not run frequently enough for optimal index health. Separate maintenance allows proactive optimization based on actual fragmentation levels.
|
||||
|
||||
## Codex Review Findings (Addressed)
|
||||
|
||||
The following issues were identified during code review and have been addressed in this specification:
|
||||
|
||||
1. **Hourly MinimumDT Calculation**: ADDRESSED - Spec now correctly documents that hourly updates use the daily timestamp with daily interval lookback (not hourly interval). See "Schedule-Based Sync Triggering" requirement.
|
||||
|
||||
2. **Failure Recovery**: ADDRESSED - Spec now requires `DoUpdate` wrapper with try/catch to mark failed updates. `CloseOpenUpdateEntries()` is invoked at startup. `PurgeUpdateEntries()` is invoked periodically. See "Update Logging and Recovery" and "Background Service Lifecycle" requirements.
|
||||
|
||||
3. **Disabled Schedules Can Run**: ADDRESSED - Spec now requires checking both `IsEnabled` AND specific schedule `Enabled` flags. Tables with all schedules disabled are only synced via manual trigger. See "Schedule-Based Sync Triggering" requirement.
|
||||
|
||||
4. **Temp Table Naming**: ADDRESSED - Spec now correctly documents `#Staging{Table}_{OperationId}` and `#{Table}_{OperationId}` naming with unique suffixes for parallel isolation. See "Table Management and Merge Operations" requirement.
|
||||
|
||||
5. **Archive Table Names**: ADDRESSED - Data Source Configurations table now uses correct `_Hist` suffix (LotUsage_Hist, WorkOrderStep_Hist, etc.).
|
||||
|
||||
6. **WorkOrderRouting Table**: ADDRESSED - Data Source Configurations table now correctly shows `WorkOrderRouting` (no `_Curr` suffix).
|
||||
|
||||
7. **MERGE LastUpdateDT Edge Case**: ADDRESSED - Spec now documents that tables without `LastUpdateDT` column update all matched rows unconditionally, and `ReleaseDate` is only used for ORDER BY in deduplication. See "Table without LastUpdateDT column" scenario.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,728 @@
|
||||
# Excel Export Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
The Excel Export subsystem generates multi-sheet Excel workbooks (.xlsx) containing search results and criteria documentation using the ClosedXML library on .NET 10. It provides an injectable `IExcelExportService` that transforms search data into formatted, protected spreadsheets that users can download, analyze, and share. The subsystem supports conditional sheet generation based on search options (MIS data extraction) and applies consistent styling, column definitions, and worksheet protection across all output.
|
||||
|
||||
## Source Reference
|
||||
|
||||
| Legacy File | Purpose |
|
||||
|-------------|---------|
|
||||
| OLD/WorkerService/Process/ExcelWriter.cs | Main Excel generation orchestration and sheet writing |
|
||||
| OLD/WorkerService/Helpers/ExcelHelpers.cs | Generic table loading and attribute-driven column formatting |
|
||||
| OLD/WebInterface/Helpers/ExcelTemplateGenerator.cs | Data entry template generation for bulk uploads |
|
||||
| OLD/WorkerService/Models/Reporting/OutputColumnAttribute.cs | Column metadata (order, format, width, wrap) |
|
||||
| OLD/WorkerService/Models/Reporting/OutputTableAttribute.cs | Table/tab metadata (name, header display) |
|
||||
| OLD/WorkerService/Models/Reporting/SearchResult.cs | Search results data model with column definitions |
|
||||
| OLD/WorkerService/Models/Reporting/MisSearchResult.cs | MIS data model with column definitions |
|
||||
| OLD/WorkerService/Models/Reporting/MisNonMatchSearchResult.cs | Investigation (mismatch) data model |
|
||||
| OLD/WorkerService/Models/Reporting/*.cs | Filter entry models for criteria documentation |
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Injectable Excel Export Service
|
||||
|
||||
The system SHALL provide an injectable service for Excel generation following .NET dependency injection patterns.
|
||||
|
||||
#### Service Interface
|
||||
|
||||
```csharp
|
||||
public interface IExcelExportService
|
||||
{
|
||||
Task<byte[]> GenerateAsync(SearchModel search, CancellationToken cancellationToken = default);
|
||||
Task GenerateToStreamAsync(SearchModel search, Stream outputStream, CancellationToken cancellationToken = default);
|
||||
}
|
||||
```
|
||||
|
||||
#### Test Support
|
||||
|
||||
For unit testing, the system SHALL provide mock data factories to generate sample data without database dependencies:
|
||||
|
||||
```csharp
|
||||
public static class ExcelTestDataFactory
|
||||
{
|
||||
public static SearchModel CreateSampleSearch(int resultCount = 10);
|
||||
public static List<SearchResult> CreateSampleResults(int count);
|
||||
public static List<MisSearchResult> CreateSampleMisResults(int count);
|
||||
public static List<MisNonMatchSearchResult> CreateSampleInvestigationResults(int count);
|
||||
}
|
||||
```
|
||||
|
||||
#### Configuration Class
|
||||
|
||||
```csharp
|
||||
public class ExcelExportOptions
|
||||
{
|
||||
public string CriteriaSheetPassword { get; set; } = "JDE_SCOPING_TOOL_PASS";
|
||||
public string DataSheetPassword { get; set; } = "JDESCOPINGTOOL";
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The service MUST be registered as scoped or transient in the DI container
|
||||
- The service MUST accept `ILogger<ExcelExportService>` for structured logging
|
||||
- The service MUST accept `IOptions<ExcelExportOptions>` for configuration
|
||||
- Logging MUST use `BeginScope()` to include search context (SearchId, SearchName)
|
||||
- The service MUST use ClosedXML (`XLWorkbook`) for workbook generation
|
||||
- Temporary files SHALL use `Path.GetTempPath()` for cross-platform temp directory access
|
||||
- Debug file output (`DebugWriteToFile`) SHALL be disabled by default; enable via configuration for troubleshooting
|
||||
|
||||
#### Scenario: Register service in DI container
|
||||
|
||||
- **WHEN** the application starts
|
||||
- **THEN** `IExcelExportService` is registered with `ExcelExportService` implementation as scoped
|
||||
|
||||
#### Scenario: Log export operations with context
|
||||
|
||||
- **WHEN** generating an export for a search
|
||||
- **THEN** log entries include SearchId and SearchName via `ILogger.BeginScope()`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Multi-Sheet Workbook Generation
|
||||
|
||||
The system SHALL generate Excel workbooks with multiple worksheets based on search configuration and results.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `SearchModel` containing:
|
||||
- Search metadata (name, username, timestamps)
|
||||
- Filter criteria (timespan, work orders, items, profit centers, work centers, operators, component lots, item/operation/MIS)
|
||||
- Search results (`List<SearchResult>`)
|
||||
- MIS results (`List<MisSearchResult>`) - when ExtractMisData is enabled
|
||||
- MIS non-match results (`List<MisNonMatchSearchResult>`) - when ExtractMisData is enabled
|
||||
- `ExtractMisData` boolean flag
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Excel workbook as `byte[]` containing:
|
||||
- "Search Criteria" sheet (always present)
|
||||
- "Search Results" sheet (always present)
|
||||
- "MIS Info" sheet (conditional - only when ExtractMisData is true and results not null)
|
||||
- "Investigation" sheet (conditional - only when ExtractMisData is true and results not null)
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The "Search Criteria" sheet MUST always be the first sheet in the workbook
|
||||
- The "Search Results" sheet MUST always be the second sheet
|
||||
- MIS-related sheets SHALL only be included when `ExtractMisData` is true AND respective result collections are not null
|
||||
- The workbook MUST be returned as a byte array for storage in the database
|
||||
|
||||
#### Scenario: Generate standard search export
|
||||
|
||||
- **WHEN** a search completes with ExtractMisData set to false
|
||||
- **THEN** the system creates a workbook with exactly two sheets: "Search Criteria" and "Search Results"
|
||||
|
||||
#### Scenario: Generate full export with MIS data
|
||||
|
||||
- **WHEN** a search completes with ExtractMisData set to true and both MIS collections populated
|
||||
- **THEN** the system creates a workbook with four sheets: "Search Criteria", "Search Results", "MIS Info", and "Investigation"
|
||||
|
||||
#### Scenario: Generate export with empty MIS results
|
||||
|
||||
- **WHEN** a search completes with ExtractMisData true but MisResults is empty (not null)
|
||||
- **THEN** the system creates the "MIS Info" sheet with empty data table
|
||||
|
||||
#### Scenario: Generate export with null MIS results
|
||||
|
||||
- **WHEN** a search completes with ExtractMisData true but MisResults is null
|
||||
- **THEN** the system skips the "MIS Info" sheet entirely
|
||||
|
||||
#### Scenario: Generate export with null investigation results
|
||||
|
||||
- **WHEN** a search completes with ExtractMisData true but MisNonMatchResults is null
|
||||
- **THEN** the system skips the "Investigation" sheet entirely
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Search Criteria Documentation Sheet
|
||||
|
||||
The system SHALL generate a "Search Criteria" sheet documenting all search parameters and execution metadata.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Search metadata: Name, UserName, SubmitDT, StartDT, EndDT
|
||||
- Timespan filter: MinimumDT, MaximumDT
|
||||
- Filter collections: WorkOrderFilter, ItemNumberFilter, ProfitCenterFilter, WorkCenterFilter, OperatorFilter, ComponentLotFilter, ItemOperationMisFilter
|
||||
- ExtractMisData flag
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Worksheet named "Search Criteria" containing:
|
||||
- Search name and username header rows
|
||||
- Timestamp section (submit, start, completed)
|
||||
- Timespan filter table
|
||||
- Multiple filter tables (one per filter type)
|
||||
- Extract MIS data indicator
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Header cells MUST use bold, centered text with Gainsboro (light gray) background via `XLColor.Gainsboro`
|
||||
- Timestamps MUST be formatted as "MMM dd, yyyy hh:mm:ss tt EST"
|
||||
- Filter tables MUST be separated by 2 blank rows (current row + 3 for next table start)
|
||||
- Columns MUST auto-fit with 15% additional padding (width * 1.15)
|
||||
- The sheet MUST be password-protected using password from `ExcelExportOptions.CriteriaSheetPassword`
|
||||
- Filter tables MUST use the Light18 table style (`XLTableTheme.TableStyleLight18`)
|
||||
|
||||
#### Scenario: Document search with all filters populated
|
||||
|
||||
- **WHEN** generating criteria sheet for a search with all filter types populated
|
||||
- **THEN** the system creates tables for each filter type in order: Timespan, Work Order, Item Number, Profit Center, Work Center, Component Lot, Operator, Item/Operation/MIS
|
||||
|
||||
#### Scenario: Document search with empty filters
|
||||
|
||||
- **WHEN** generating criteria sheet for a search with empty filter collections
|
||||
- **THEN** the system still creates empty tables with headers for each filter type
|
||||
|
||||
#### Scenario: Format ExtractMisData indicator
|
||||
|
||||
- **WHEN** ExtractMisData is true
|
||||
- **THEN** the system displays "YES" in the Extract MIS data row
|
||||
|
||||
#### Scenario: Format ExtractMisData indicator negative
|
||||
|
||||
- **WHEN** ExtractMisData is false
|
||||
- **THEN** the system displays "NO" in the Extract MIS data row
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Search Results Sheet Generation
|
||||
|
||||
The system SHALL generate a "Search Results" sheet containing work order search results with standardized columns.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `List<SearchResult>` containing work order data with:
|
||||
- Work order identifiers (number, branch code, lot number)
|
||||
- Item information (number, planning family, stocking type)
|
||||
- Quantities (order, held, scrapped, shipped)
|
||||
- Operation details (step branch, number, description, function description)
|
||||
- Timestamps (step update, status update)
|
||||
- Status information (code, description)
|
||||
- Inclusion reason (computed from flags)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Worksheet named "Search Results" with:
|
||||
- 19 columns in defined order
|
||||
- Data formatted as Excel table named "Search_Results"
|
||||
- Light18 table style applied
|
||||
- No worksheet protection (attribute-driven path)
|
||||
|
||||
#### Column Definitions
|
||||
|
||||
| Order | Header | Data Type | Format |
|
||||
|-------|--------|-----------|--------|
|
||||
| 10 | Work Order Number | long | Standard |
|
||||
| 20 | Work Order Branch Code | string | Standard |
|
||||
| 30 | Lot Number | string | Standard |
|
||||
| 40 | Item Number | string | Standard |
|
||||
| 50 | Planning Family | string | Standard |
|
||||
| 55 | Stocking Type | string | Standard |
|
||||
| 60 | Order Quantity | decimal | Standard |
|
||||
| 70 | Held Quantity | decimal | Standard |
|
||||
| 80 | Scrapped Quantity | decimal | Standard |
|
||||
| 90 | Shipped Quantity | decimal | Standard |
|
||||
| 100 | Operation Step Branch Code | string | Standard |
|
||||
| 110 | Operation Step | decimal | Standard |
|
||||
| 120 | Operation Step Description | string | Standard |
|
||||
| 130 | Function Operation Description | string | Standard |
|
||||
| 140 | Operation Step Update Timestamp | DateTime | `[$-409]m/d/yy h:mm AM/PM;@` |
|
||||
| 150 | Status Code | string | Standard |
|
||||
| 160 | Status Description | string | Standard |
|
||||
| 170 | Status Update Timestamp | DateTime? | `[$-409]MM/dd/yyyy;@` |
|
||||
| 180 | Inclusion Reason | string (computed) | Standard |
|
||||
| 190 | (Additional column per legacy code) | - | Standard |
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Columns MUST auto-fit with 30% additional padding (width * 1.3)
|
||||
- The table MUST NOT show totals row
|
||||
- Timestamp columns MUST use the defined Excel number formats for proper date/time display
|
||||
- The Inclusion Reason column MUST compute values from boolean flags (ManuallySpecified, Flagged, CARDEX, PartsList, SplitOrder)
|
||||
|
||||
#### Scenario: Format inclusion reason for manually specified
|
||||
|
||||
- **WHEN** a result has ManuallySpecified = true
|
||||
- **THEN** the Inclusion Reason displays "ManuallySpecified"
|
||||
|
||||
#### Scenario: Format inclusion reason for flagged
|
||||
|
||||
- **WHEN** a result has Flagged = true (and ManuallySpecified = false)
|
||||
- **THEN** the Inclusion Reason displays "Flagged"
|
||||
|
||||
#### Scenario: Format inclusion reason for CARDEX only
|
||||
|
||||
- **WHEN** a result has CARDEX = true and PartsList = false
|
||||
- **THEN** the Inclusion Reason displays "ComponentUsage (CARDEX)"
|
||||
|
||||
#### Scenario: Format inclusion reason for PartsList only
|
||||
|
||||
- **WHEN** a result has PartsList = true and CARDEX = false
|
||||
- **THEN** the Inclusion Reason displays "ComponentUsage (Parts List)"
|
||||
|
||||
#### Scenario: Format inclusion reason for CARDEX and parts list
|
||||
|
||||
- **WHEN** a result has both CARDEX = true and PartsList = true
|
||||
- **THEN** the Inclusion Reason displays "ComponentUsage (CARDEX + Parts List)"
|
||||
|
||||
#### Scenario: Format inclusion reason for split order
|
||||
|
||||
- **WHEN** a result has only SplitOrder = true
|
||||
- **THEN** the Inclusion Reason displays "Split order"
|
||||
|
||||
#### Scenario: Format inclusion reason unknown
|
||||
|
||||
- **WHEN** a result has no matching boolean flags
|
||||
- **THEN** the Inclusion Reason displays "UNKNOWN"
|
||||
|
||||
---
|
||||
|
||||
### Requirement: MIS Info Sheet Generation
|
||||
|
||||
The system SHALL generate a "MIS Info" sheet containing Manufacturing Instruction Sheet data when enabled.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `List<MisSearchResult>` containing:
|
||||
- Item identification (number, description)
|
||||
- MIS metadata (number, revision, status, release date)
|
||||
- Sequence/step numbers (MIS job step, job step, matched)
|
||||
- Match indicators (RoutingMatch, MasterMatch)
|
||||
- Description fields (function operation, test description)
|
||||
- Sampling information (type, value)
|
||||
- Long text fields (tools/gauges, work instructions)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Worksheet named "MIS Info" with:
|
||||
- 19 columns in defined order
|
||||
- Data formatted as Excel table named "MIS_Info"
|
||||
- Light18 table style applied
|
||||
- Specific columns with text wrapping and fixed width
|
||||
|
||||
#### Column Definitions
|
||||
|
||||
| Order | Header | Format | Width |
|
||||
|-------|--------|--------|-------|
|
||||
| 10 | Item Number | Standard | Auto |
|
||||
| 20 | MIS Job Step Sequence Number | Standard | Auto |
|
||||
| 30 | MIS Number | Standard | Auto |
|
||||
| 40 | MIS Revision | Standard | Auto |
|
||||
| 50 | Item Description | Standard | Auto |
|
||||
| 60 | MIS Release Status | Standard | Auto |
|
||||
| 70 | MIS Release Date | Timestamp | Auto |
|
||||
| 80 | Branch Code | Standard | Auto |
|
||||
| 90 | Job Step Sequence Number | Standard | Auto |
|
||||
| 100 | Matched Sequence Number | Standard | Auto |
|
||||
| 110 | Matched to F3112Z1? | Standard | Auto |
|
||||
| 120 | Matched to F3003? | Standard | Auto |
|
||||
| 130 | Function Operation Description | Standard | Auto |
|
||||
| 140 | Char Number | Standard | Auto |
|
||||
| 150 | Test Description | Standard | 65 (wrapped) |
|
||||
| 160 | Sampling Type | Standard | Auto |
|
||||
| 170 | Sampling Value | Standard | Auto |
|
||||
| 180 | Tools & Gauges | Standard | 65 (wrapped) |
|
||||
| 190 | Work Instructions | Standard | 65 (wrapped) |
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Columns with long text (Test Description, Tools & Gauges, Work Instructions) MUST have WrapText enabled and fixed width of 65
|
||||
- Auto-width columns MUST apply 30% additional padding
|
||||
- Wrapped columns MUST NOT be auto-fitted
|
||||
|
||||
#### Scenario: Generate MIS sheet with wrapped columns
|
||||
|
||||
- **WHEN** generating MIS Info sheet with data
|
||||
- **THEN** Test Description, Tools & Gauges, and Work Instructions columns have fixed 65-character width with text wrapping enabled
|
||||
|
||||
#### Scenario: Handle null MIS results
|
||||
|
||||
- **WHEN** MisResults is null
|
||||
- **THEN** the system skips MIS Info sheet generation entirely (returns early)
|
||||
|
||||
#### Scenario: Format boolean match indicators
|
||||
|
||||
- **WHEN** RoutingMatch or MasterMatch values are written
|
||||
- **THEN** they display as "True" or "False" text values
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Investigation Sheet Generation
|
||||
|
||||
The system SHALL generate an "Investigation" sheet containing router mismatch data for analysis.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `List<MisNonMatchSearchResult>` containing:
|
||||
- Work center and order identification
|
||||
- Job step details (number, description, dates)
|
||||
- Function and routing information
|
||||
- Item details (number, description)
|
||||
- Match indicators (WasJobStepAdded, MatchedJobStepNumber)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Worksheet named "Investigation" with:
|
||||
- 12 columns in defined order
|
||||
- Data formatted as Excel table named "Investigation"
|
||||
- Light18 table style applied
|
||||
|
||||
#### Column Definitions
|
||||
|
||||
| Order | Header | Format |
|
||||
|-------|--------|--------|
|
||||
| 10 | Work Center Code | Standard |
|
||||
| 20 | Work Order Number | Standard |
|
||||
| 30 | Work Order Start Date | `[$-409]MM/dd/yyyy;@` |
|
||||
| 40 | Job Step Number | Standard |
|
||||
| 50 | Function Operation Description | Standard |
|
||||
| 60 | Job Step End Date | `[$-409]MM/dd/yyyy;@` |
|
||||
| 70 | Function Code | Standard |
|
||||
| 75 | Was Job Step Added? | Standard (boolean) |
|
||||
| 76 | Matched Job Step Number | Standard |
|
||||
| 80 | Item Number | Standard |
|
||||
| 90 | Item Description | Standard |
|
||||
| 100 | Routing Type | Standard |
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Date columns MUST use `DATE_FORMAT` (`[$-409]MM/dd/yyyy;@`)
|
||||
- All columns MUST auto-fit with 30% additional padding
|
||||
|
||||
#### Scenario: Handle null mismatch results
|
||||
|
||||
- **WHEN** MisNonMatchResults is null
|
||||
- **THEN** the system skips Investigation sheet generation entirely
|
||||
|
||||
#### Scenario: Format date columns
|
||||
|
||||
- **WHEN** generating Investigation sheet
|
||||
- **THEN** Work Order Start Date and Job Step End Date columns use `[$-409]MM/dd/yyyy;@` number format
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Worksheet Protection
|
||||
|
||||
The system SHALL apply password-based protection to worksheets with configurable allowed operations.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Worksheet to protect
|
||||
- Protection password from `IOptions<ExcelExportOptions>`
|
||||
- Editable range definition (cells beyond data area)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Protected worksheet with:
|
||||
- Locked data cells
|
||||
- Unlocked extension area for user additions
|
||||
- Specific operations allowed/disallowed
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Protection passwords MUST be loaded from `ExcelExportOptions` configuration
|
||||
- Protected ranges MUST allow the following operations via `IXLSheetProtection`:
|
||||
- Delete columns: YES
|
||||
- Delete rows: NO
|
||||
- Auto filter: YES
|
||||
- Format cells: YES
|
||||
- Format columns: YES
|
||||
- Format rows: YES
|
||||
- Select locked cells: YES
|
||||
- Select unlocked cells: YES
|
||||
- Edit objects: YES
|
||||
- Sort: YES
|
||||
- Unprotected area MUST extend 1000 rows and columns beyond data range
|
||||
|
||||
#### ClosedXML Protection Example
|
||||
|
||||
```csharp
|
||||
var protection = worksheet.Protect(options.Value.DataSheetPassword);
|
||||
protection.AllowElement(XLSheetProtectionElements.DeleteColumns);
|
||||
protection.AllowElement(XLSheetProtectionElements.AutoFilter);
|
||||
protection.AllowElement(XLSheetProtectionElements.FormatCells);
|
||||
protection.AllowElement(XLSheetProtectionElements.FormatColumns);
|
||||
protection.AllowElement(XLSheetProtectionElements.FormatRows);
|
||||
protection.AllowElement(XLSheetProtectionElements.SelectLockedCells);
|
||||
protection.AllowElement(XLSheetProtectionElements.SelectUnlockedCells);
|
||||
protection.AllowElement(XLSheetProtectionElements.EditObjects);
|
||||
protection.AllowElement(XLSheetProtectionElements.Sort);
|
||||
```
|
||||
|
||||
#### Scenario: Protect search criteria sheet
|
||||
|
||||
- **WHEN** generating Search Criteria sheet
|
||||
- **THEN** sheet is protected using `ExcelExportOptions.CriteriaSheetPassword`
|
||||
|
||||
#### Scenario: Apply consistent protection settings
|
||||
|
||||
- **WHEN** protecting any data worksheet
|
||||
- **THEN** all boolean protection flags match the defined allowed operations
|
||||
|
||||
#### Scenario: Allow filtering and sorting
|
||||
|
||||
- **WHEN** user opens protected worksheet
|
||||
- **THEN** they can filter and sort data without entering password
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Attribute-Driven Column Configuration
|
||||
|
||||
The system SHALL use C# attributes to define column metadata for automatic table generation.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Data model classes decorated with `OutputTableAttribute` and `OutputColumnAttribute`
|
||||
- Properties marked with `OutputColumnAttribute` defining column order, header, format, width, and wrap settings
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Dynamically generated Excel tables based on attribute configuration
|
||||
|
||||
#### Property Access Strategy
|
||||
|
||||
- The system SHALL use native .NET reflection (`PropertyInfo.GetValue()`) for property access
|
||||
- Source generators MAY be used as future optimization for compile-time property mapping
|
||||
|
||||
#### OutputColumnAttribute Properties
|
||||
|
||||
| Property | Type | Default | Purpose |
|
||||
|----------|------|---------|---------|
|
||||
| Order | int | - | Column sort order |
|
||||
| HeaderText | string | - | Column header display text |
|
||||
| Format | string | "@" (text) | Excel number format |
|
||||
| AutoWidth | bool | true | Enable auto-fit |
|
||||
| Width | double | - | Fixed width (when AutoWidth=false) |
|
||||
| WrapText | bool | false | Enable text wrapping |
|
||||
|
||||
#### OutputTableAttribute Properties
|
||||
|
||||
| Property | Type | Purpose |
|
||||
|----------|------|---------|
|
||||
| TabName | string | Worksheet tab name |
|
||||
| TableName | string | Excel table name |
|
||||
| ShowHeader | bool | Show merged header row above table |
|
||||
|
||||
#### Standard Formats
|
||||
|
||||
| Constant | Value | Usage |
|
||||
|----------|-------|-------|
|
||||
| STD_FORMAT | "@" | Text format (default) |
|
||||
| DATE_FORMAT | "[$-409]MM/dd/yyyy;@" | Date only |
|
||||
| TIMESTAMP_FORMAT | "[$-409]m/d/yy h:mm AM/PM;@" | Date and time |
|
||||
| WRAPPED_COLUMN_WIDTH | 65 | Fixed width for wrapped columns |
|
||||
|
||||
#### Scenario: Generate table from decorated model
|
||||
|
||||
- **WHEN** LoadTab is called with a model type having OutputTableAttribute
|
||||
- **THEN** worksheet name and table name are derived from attribute values
|
||||
|
||||
#### Scenario: Order columns by attribute
|
||||
|
||||
- **WHEN** generating table from model with OutputColumnAttribute
|
||||
- **THEN** columns appear in Order property sequence, with ties broken alphabetically by property name
|
||||
|
||||
#### Scenario: Apply custom format to column
|
||||
|
||||
- **WHEN** a property has OutputColumnAttribute with Format specified
|
||||
- **THEN** that format is applied to the entire column's number format
|
||||
|
||||
#### Scenario: Apply wrapped text configuration
|
||||
|
||||
- **WHEN** a property has WrapText=true and AutoWidth=false
|
||||
- **THEN** column has text wrapping enabled and uses fixed Width value
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Data Entry Template Generation
|
||||
|
||||
The system SHALL generate simple Excel templates for bulk data entry of filter values.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Source data collection (optional, for pre-population)
|
||||
- Header text (single column) or header array (multi-column)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Single-sheet workbook named "Data Entry Template"
|
||||
- Header row with standard formatting
|
||||
- Optional pre-populated data rows
|
||||
- All columns formatted as text ("@")
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Header row MUST be bold, centered, with Gainsboro background via `XLColor.Gainsboro`
|
||||
- Single-column templates MUST use 45-character width
|
||||
- Multi-column templates MUST use 65-character width per column
|
||||
- All cells MUST use text format to preserve leading zeros in item/lot numbers
|
||||
|
||||
#### Scenario: Generate empty single-column template
|
||||
|
||||
- **WHEN** Generate is called with null sourceData
|
||||
- **THEN** template contains only header row with no data
|
||||
|
||||
#### Scenario: Generate pre-populated template
|
||||
|
||||
- **WHEN** Generate is called with sourceData containing values
|
||||
- **THEN** data rows are populated starting at row 2
|
||||
|
||||
#### Scenario: Generate multi-column template
|
||||
|
||||
- **WHEN** Generate is called with object[][] sourceData and string[] headers
|
||||
- **THEN** multiple columns are created with respective headers
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Header Cell Formatting
|
||||
|
||||
The system SHALL apply consistent header formatting across all worksheets.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Cell range to format
|
||||
- Optional text value
|
||||
- Optional merge flag
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Formatted header cell(s) with:
|
||||
- Horizontal center alignment
|
||||
- Bold font
|
||||
- Solid Gainsboro (light gray) background fill via `XLColor.Gainsboro`
|
||||
|
||||
#### ClosedXML Formatting Example
|
||||
|
||||
```csharp
|
||||
var cell = worksheet.Cell(row, column);
|
||||
cell.Value = headerText;
|
||||
cell.Style.Font.Bold = true;
|
||||
cell.Style.Alignment.Horizontal = XLAlignmentHorizontalValues.Center;
|
||||
cell.Style.Fill.BackgroundColor = XLColor.Gainsboro;
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Header format MUST be applied to:
|
||||
- All table header rows
|
||||
- Label cells in criteria sheet (column 1)
|
||||
- Filter table section headers
|
||||
- Merge flag MUST only be set for multi-cell ranges using `IXLRange.Merge()`
|
||||
- Text value MUST be written when provided
|
||||
|
||||
#### Scenario: Format single header cell
|
||||
|
||||
- **WHEN** ApplyHeaderFormat is called on single cell with text
|
||||
- **THEN** cell has bold font, center alignment, Gainsboro background, and displays provided text
|
||||
|
||||
#### Scenario: Format merged header range
|
||||
|
||||
- **WHEN** ApplyHeaderFormat is called on multi-cell range with merge=true
|
||||
- **THEN** cells are merged and formatted as single header
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Async Generation Pattern
|
||||
|
||||
The system SHALL support async/await patterns for export generation.
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The `GenerateAsync` method MUST return `Task<byte[]>`
|
||||
- Since ClosedXML's `SaveAs` to `MemoryStream` is synchronous, the implementation MAY wrap the CPU-bound work in `Task.Run()` to avoid blocking
|
||||
- The method MUST accept `CancellationToken` for cancellation support
|
||||
- For very large exports, future versions MAY implement `Stream`-based output
|
||||
|
||||
#### Implementation Example
|
||||
|
||||
```csharp
|
||||
public async Task<byte[]> GenerateAsync(SearchModel search, CancellationToken cancellationToken = default)
|
||||
{
|
||||
using var scope = _logger.BeginScope(new Dictionary<string, object>
|
||||
{
|
||||
["SearchId"] = search.Id,
|
||||
["SearchName"] = search.Name
|
||||
});
|
||||
|
||||
_logger.LogInformation("Starting Excel export generation");
|
||||
|
||||
// ClosedXML operations are synchronous, wrap in Task.Run for non-blocking
|
||||
return await Task.Run(() =>
|
||||
{
|
||||
using var workbook = new XLWorkbook();
|
||||
|
||||
// ... build sheets ...
|
||||
|
||||
using var stream = new MemoryStream();
|
||||
workbook.SaveAs(stream);
|
||||
return stream.ToArray();
|
||||
}, cancellationToken);
|
||||
}
|
||||
```
|
||||
|
||||
#### Scenario: Generate export asynchronously
|
||||
|
||||
- **WHEN** `GenerateAsync` is called
|
||||
- **THEN** the method returns a `Task<byte[]>` that completes when workbook generation finishes
|
||||
|
||||
#### Scenario: Support cancellation
|
||||
|
||||
- **WHEN** `GenerateAsync` is called with a `CancellationToken` that is cancelled
|
||||
- **THEN** the operation throws `OperationCanceledException`
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
| Legacy Pattern | New Pattern | Rationale |
|
||||
|----------------|-------------|-----------|
|
||||
| EPPlus 4.x (LGPL) | ClosedXML (MIT license) | EPPlus 7+ requires commercial license; ClosedXML is fully free and MIT licensed |
|
||||
| `System.Drawing.Color` | `XLColor` | ClosedXML uses its own color type (`XLColor.Gainsboro`, etc.) |
|
||||
| `ExcelPackage` | `XLWorkbook` | ClosedXML workbook class |
|
||||
| `ExcelWorksheet` | `IXLWorksheet` | ClosedXML worksheet interface |
|
||||
| `ExcelRange` | `IXLRange` or `IXLCell` | ClosedXML range/cell interfaces |
|
||||
| Fasterflect reflection | Native reflection or source generators | Reduce dependencies; native `PropertyInfo.GetValue()` is sufficient; source generators available for future optimization |
|
||||
| Extension methods on EPPlus types | Service class with `IExcelExportService` interface | Enable testing and alternative implementations |
|
||||
| Static `ExcelWriter.Generate()` | Injectable `IExcelExportService` | Dependency injection for testability and configuration |
|
||||
| Hard-coded passwords | `IOptions<ExcelExportOptions>` configuration | Move protection passwords to configuration for security and flexibility |
|
||||
| Byte array return | `byte[]` and `Stream` | Support both: `GenerateAsync` returns byte[], `GenerateToStreamAsync` writes to Stream for large exports |
|
||||
| Synchronous generation | Async wrapping via `Task.Run()` | Support async patterns; ClosedXML SaveAs is sync, wrap for non-blocking |
|
||||
| TableStyles enum | `XLTableTheme` | ClosedXML table themes (e.g., `XLTableTheme.TableStyleLight18`) |
|
||||
| NLog static logging | `ILogger<T>` injected + `BeginScope()` | Modern structured logging with contextual scopes |
|
||||
|
||||
## Resolved Design Decisions
|
||||
|
||||
1. **Library Selection**: Use ClosedXML (MIT license) - fully free for commercial use, similar API to EPPlus, active maintenance.
|
||||
|
||||
2. **Password Protection**: Move to `IOptions<ExcelExportOptions>` configuration. Default values preserved for backward compatibility but can be overridden via appsettings.json.
|
||||
|
||||
3. **Large Export Handling**: Implement streaming architecture for memory-efficient large exports.
|
||||
- The system SHALL support `Stream`-based output for large workbooks to avoid memory pressure
|
||||
- The system SHALL provide both `GenerateAsync` (returns `byte[]`) and `GenerateToStreamAsync` (writes to `Stream`) methods
|
||||
- For exports exceeding a configurable row threshold, the streaming approach SHALL be preferred
|
||||
|
||||
4. **Async Support**: `GenerateAsync` method wraps synchronous ClosedXML operations in `Task.Run()` to avoid blocking.
|
||||
|
||||
5. **Format Compatibility**: Maintain locale ID 409 (US English) for timestamp formats. International configuration deferred to future version.
|
||||
|
||||
6. **Template Generator**: Retain `ExcelTemplateGenerator` functionality for bulk data entry via the Blazor UI.
|
||||
|
||||
## Codex Review Findings (Addressed)
|
||||
|
||||
The following inaccuracies were identified during review and have been addressed in this specification:
|
||||
|
||||
1. **Table Style and Protection**: CORRECTED - Spec now states data sheets use Light18 style (via `XLTableTheme.TableStyleLight18`) and protection is applied only to criteria sheet by default. Data sheets generated via attribute-driven `LoadTab` do not apply protection.
|
||||
|
||||
2. **Column Counts Corrected**:
|
||||
- Search Results: CORRECTED to 19 columns (was incorrectly 18)
|
||||
- Investigation: CORRECTED to 12 columns (was incorrectly 11)
|
||||
|
||||
3. **Investigation Date Format**: CORRECTED - Spec now states `DATE_FORMAT` (`[$-409]MM/dd/yyyy;@`) is used, not "m/d/yyyy".
|
||||
|
||||
4. **Inclusion Reason Scenarios Complete**: ADDED - Scenarios now include CARDEX-only, PartsList-only, and UNKNOWN cases.
|
||||
|
||||
5. **Null List Handling**: CLARIFIED - Spec now explicitly states null checks are required before calling sheet generation methods. Implementation MUST check for null before generating MIS Info and Investigation sheets.
|
||||
|
||||
6. **Criteria Table Spacing**: CORRECTED - Spec now states "2 blank rows" between filter tables (current row + 3 for next table start).
|
||||
@@ -0,0 +1,120 @@
|
||||
# infrastructure Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change setup-solution-foundation. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: Service registration pattern
|
||||
|
||||
The system SHALL use extension methods on IServiceCollection to register module-specific services.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- IServiceCollection services
|
||||
- IConfiguration configuration
|
||||
|
||||
#### Outputs
|
||||
|
||||
- IServiceCollection (fluent return for chaining)
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each module SHALL have one extension method (AddDataAccess, AddDataSync, AddAuth, AddExcelExport, AddSearchProcessing)
|
||||
- Extension methods SHALL bind their module's Options class from configuration
|
||||
- Extension methods SHALL register services with appropriate lifetimes:
|
||||
- Scoped: Database connections, repositories, unit-of-work
|
||||
- Singleton: Configuration options, HTTP clients, caching services
|
||||
- Transient: Stateless processors, validators
|
||||
- Extension methods SHALL return IServiceCollection for fluent chaining
|
||||
|
||||
#### Scenario: Module service registration
|
||||
|
||||
- **WHEN** Program.cs calls builder.Services.AddDataAccess(configuration)
|
||||
- **THEN** DataAccessOptions is bound from the "DataAccess" configuration section
|
||||
- **AND** ILotFinderRepository is registered with Scoped lifetime
|
||||
- **AND** the method returns IServiceCollection for further chaining
|
||||
|
||||
#### Scenario: Service lifetime correctness
|
||||
|
||||
- **WHEN** a Scoped service is requested multiple times within the same HTTP request
|
||||
- **THEN** the same instance is returned each time
|
||||
- **AND** a new instance is created for the next HTTP request
|
||||
|
||||
#### Scenario: Chained registration
|
||||
|
||||
- **WHEN** Program.cs chains multiple extension methods
|
||||
- **THEN** all modules are registered in the order called
|
||||
- **AND** the final IServiceCollection contains all registered services
|
||||
|
||||
### Requirement: Configuration binding pattern
|
||||
|
||||
The system SHALL use IOptions<T> pattern to bind strongly-typed configuration from appsettings.json.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- appsettings.json with named sections
|
||||
- Options class with matching property names
|
||||
|
||||
#### Outputs
|
||||
|
||||
- IOptions<T> resolved from DI with bound values
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each Options class SHALL define a static SectionName constant matching the JSON section
|
||||
- Options classes SHALL use C# naming conventions (PascalCase properties)
|
||||
- Configuration sections SHALL use matching PascalCase names
|
||||
- Default values SHALL be defined in Options class properties
|
||||
- Options classes SHALL be registered using services.Configure<T>(section)
|
||||
|
||||
#### Scenario: Configuration binding at startup
|
||||
|
||||
- **WHEN** the application starts with valid appsettings.json
|
||||
- **THEN** IOptions<DataAccessOptions> resolves with values from the DataAccess section
|
||||
- **AND** properties not specified in JSON use their default values
|
||||
|
||||
#### Scenario: Missing configuration section
|
||||
|
||||
- **WHEN** the application starts without a required configuration section
|
||||
- **THEN** IOptions<T> resolves with all default property values
|
||||
- **AND** no exception is thrown at startup
|
||||
|
||||
#### Scenario: Development override
|
||||
|
||||
- **WHEN** the application runs in Development environment
|
||||
- **THEN** appsettings.Development.json values override appsettings.json values
|
||||
- **AND** IOptions<DataAccessOptions>.Value.EnableDetailedLogging is true
|
||||
|
||||
### Requirement: Extension method organization
|
||||
|
||||
The system SHALL organize extension methods in the JdeScoping.Core project under an Extensions namespace.
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Extension methods SHALL be in namespace JdeScoping.Core.Extensions
|
||||
- Each module SHALL have a dedicated static class: {Module}ServiceExtensions
|
||||
- Extension method SHALL be named Add{Module}
|
||||
- Files SHALL be located at: JdeScoping.Core/Extensions/{Module}ServiceExtensions.cs
|
||||
|
||||
#### Scenario: Extension method discovery
|
||||
|
||||
- **WHEN** a developer adds using JdeScoping.Core.Extensions
|
||||
- **THEN** all AddXxx extension methods are available on IServiceCollection
|
||||
- **AND** IntelliSense shows method documentation
|
||||
|
||||
### Requirement: Options class organization
|
||||
|
||||
The system SHALL organize Options classes in the JdeScoping.Core project under an Options namespace.
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Options classes SHALL be in namespace JdeScoping.Core.Options
|
||||
- Class names SHALL follow pattern: {Module}Options
|
||||
- SectionName constant SHALL match the JSON section name exactly
|
||||
- Files SHALL be located at: JdeScoping.Core/Options/{Module}Options.cs
|
||||
|
||||
#### Scenario: Options class consistency
|
||||
|
||||
- **WHEN** DataAccessOptions is defined with SectionName = "DataAccess"
|
||||
- **THEN** configuration.GetSection("DataAccess") returns the matching section
|
||||
- **AND** services.Configure<DataAccessOptions>(section) binds all properties
|
||||
|
||||
@@ -0,0 +1,955 @@
|
||||
# Search Processing Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
The search processing subsystem enables users to query manufacturing data (work orders, lots, items, operators, and work centers) from a locally cached JDE/CMS database. It accepts multi-dimensional filter criteria, dynamically generates SQL queries using a builder pattern, executes searches against the SQL Server cache using Microsoft.Data.SqlClient, and aggregates results including downstream work order tracking and MIS (Manufacturing Information System) data extraction. This specification targets .NET 10 with modern C# language features and patterns.
|
||||
|
||||
## Source Reference
|
||||
|
||||
| Legacy Files | Purpose |
|
||||
|--------------|---------|
|
||||
| `OLD/DataModel/Models/SearchCriteria.cs` | Core search criteria model storing filter parameters |
|
||||
| `OLD/DataModel/ViewModels/SearchCriteriaViewModel.cs` | View model with enriched filter data for UI binding |
|
||||
| `OLD/WorkerService/Models/Reporting/SearchModel.cs` | Reporting model combining criteria with filter entries and results |
|
||||
| `OLD/WorkerService/Templates/QueryTemplate.cs` | T4-generated SQL query builder (to be replaced with builder pattern) |
|
||||
| `OLD/WorkerService/Templates/QueryTemplateExtension.cs` | Partial class extension for QueryTemplate |
|
||||
| `OLD/WorkerService/Helpers/SearchModelHelpers.cs` | Helper methods for model conversion and parameter creation |
|
||||
| `OLD/WorkerService/Models/Reporting/*.cs` | Filter entry models and result models |
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Search Criteria Model
|
||||
|
||||
The system SHALL support a search criteria model with eight distinct filter types that can be combined in any combination.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- **Timespan Filter**: Optional minimum and maximum datetime values
|
||||
- **Work Order Filter**: Collection of work order numbers (long integers)
|
||||
- **Item Number Filter**: Collection of item numbers (strings)
|
||||
- **Profit Center Filter**: Collection of profit center codes (strings)
|
||||
- **Work Center Filter**: Collection of work center codes (strings)
|
||||
- **Operator Filter**: Collection of operator user IDs (strings)
|
||||
- **Component Lot Filter**: Collection of lot number/item number pairs
|
||||
- **Item Operation MIS Filter**: Collection of item/operation/MIS number/MIS revision combinations
|
||||
- **Extract MIS Data Flag**: Boolean indicating whether to include MIS data in results
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Serializable `SearchCriteria` object stored as JSON in the `Search.CriteriaJSON` column
|
||||
- `SearchModel` populated with enriched filter entries (descriptions, full names, etc.)
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- All filter collections SHALL be initialized as empty lists by default
|
||||
- Filter lists MAY contain zero or more entries
|
||||
- The timespan filter is considered enabled when either `MinimumDT` or `MaximumDT` has a value
|
||||
- Each filter type is considered enabled when its collection contains at least one entry
|
||||
- Component lot filters MUST include both lot number and item number for proper matching
|
||||
- Item/Operation/MIS filters MUST include all four fields: item number, operation number, MIS number, and MIS revision
|
||||
|
||||
#### Scenario: Create search with timespan filter only
|
||||
|
||||
- **WHEN** a user specifies minimum date of 2024-01-01 and maximum date of 2024-12-31
|
||||
- **THEN** the system creates a SearchCriteria with `MinimumDT` = 2024-01-01 and `MaximumDT` = 2024-12-31
|
||||
- **AND** `TimespanFilterEnabled` returns true
|
||||
- **AND** all other filter collections remain empty
|
||||
|
||||
#### Scenario: Create search with multiple filter types
|
||||
|
||||
- **WHEN** a user specifies work order numbers [12345, 67890] and item numbers ["PART-001", "PART-002"]
|
||||
- **THEN** the system creates a SearchCriteria with both collections populated
|
||||
- **AND** `WorkOrderFilterEnabled` and `ItemNumberFilterEnabled` both return true
|
||||
- **AND** other filter collections remain empty
|
||||
|
||||
#### Scenario: Create search with component lot filter
|
||||
|
||||
- **WHEN** a user specifies component lot "LOT123" for item "ITEM-ABC"
|
||||
- **THEN** the system creates a ComponentLotFilter entry with both LotNumber and ItemNumber
|
||||
- **AND** the filter is used to find work orders that consumed material from that lot
|
||||
|
||||
#### Scenario: Serialize and deserialize search criteria
|
||||
|
||||
- **WHEN** a SearchCriteria object is serialized to JSON and stored
|
||||
- **THEN** the system can deserialize it back to the original object structure
|
||||
- **AND** all filter collections and values are preserved
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Query Builder Generation
|
||||
|
||||
The system SHALL dynamically generate SQL queries based on enabled filters using SqlKata fluent query builder (replacing the legacy T4 text template).
|
||||
|
||||
#### Migration Note: T4 Template Replacement with SqlKata
|
||||
|
||||
The legacy system uses a T4 text template (`QueryTemplate.tt`) for SQL generation. T4 templates have limited support in modern .NET SDK-style projects. The new implementation SHALL use **SqlKata** - a fluent SQL query builder that generates parameterized SQL and integrates well with Dapper.
|
||||
|
||||
**NuGet Package:** `SqlKata` and `SqlKata.Execution`
|
||||
|
||||
**Primary Query Building with SqlKata:**
|
||||
|
||||
```csharp
|
||||
using SqlKata;
|
||||
using SqlKata.Compilers;
|
||||
|
||||
public interface ISearchQueryBuilder
|
||||
{
|
||||
SearchQueryResult BuildSearchQuery(SearchModel model);
|
||||
}
|
||||
|
||||
public record SearchQueryResult(string Sql, IDictionary<string, object> Parameters);
|
||||
|
||||
public sealed class SqlKataSearchQueryBuilder : ISearchQueryBuilder
|
||||
{
|
||||
private readonly SqlServerCompiler _compiler = new();
|
||||
|
||||
public SearchQueryResult BuildSearchQuery(SearchModel model)
|
||||
{
|
||||
var query = new Query("WorkOrder_Curr as wo")
|
||||
.Select("wo.*", "wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode");
|
||||
|
||||
// Conditional joins based on active filters
|
||||
if (model.OperatorFilterEnabled)
|
||||
{
|
||||
query.Join("WorkOrderTime_Curr as wot", "wot.WorkOrderNumber", "wo.WorkOrderNumber");
|
||||
}
|
||||
|
||||
if (model.ProfitCenterFilterEnabled || model.WorkCenterFilterEnabled)
|
||||
{
|
||||
query.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber");
|
||||
}
|
||||
|
||||
if (model.ComponentLotFilterEnabled)
|
||||
{
|
||||
query.Join("WorkOrderComponent_Curr as woc", "woc.WorkOrderNumber", "wo.WorkOrderNumber");
|
||||
}
|
||||
|
||||
// Conditional WHERE clauses
|
||||
if (model.TimespanFilterEnabled)
|
||||
{
|
||||
query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
|
||||
}
|
||||
|
||||
if (model.WorkOrderFilterEnabled)
|
||||
{
|
||||
query.WhereIn("wo.WorkOrderNumber", model.WorkOrderFilter.Select(w => w.WorkOrderNumber));
|
||||
}
|
||||
|
||||
if (model.ItemNumberFilterEnabled)
|
||||
{
|
||||
query.WhereIn("wo.ItemNumber", model.ItemNumberFilter.Select(i => i.ItemNumber));
|
||||
}
|
||||
|
||||
if (model.OperatorFilterEnabled)
|
||||
{
|
||||
query.WhereIn("wot.OperatorAN", model.OperatorFilter.Select(o => o.AddressNumber));
|
||||
}
|
||||
|
||||
// Compile to SQL + parameters
|
||||
var compiled = _compiler.Compile(query);
|
||||
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Complex Query Composition with SqlKata:**
|
||||
|
||||
```csharp
|
||||
public sealed class ComposableSearchQueryBuilder
|
||||
{
|
||||
private readonly SqlServerCompiler _compiler = new();
|
||||
|
||||
public SearchQueryResult BuildFullSearchQuery(SearchModel model)
|
||||
{
|
||||
// Build composable query parts
|
||||
var baseQuery = BuildBaseWorkOrderQuery(model);
|
||||
var filterQuery = ApplyFilters(baseQuery, model);
|
||||
var joinedQuery = ApplyConditionalJoins(filterQuery, model);
|
||||
|
||||
var compiled = _compiler.Compile(joinedQuery);
|
||||
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
|
||||
}
|
||||
|
||||
private Query BuildBaseWorkOrderQuery(SearchModel model)
|
||||
{
|
||||
return new Query("WorkOrder_Curr as wo")
|
||||
.Select("wo.WorkOrderNumber", "wo.ItemNumber", "wo.StatusCode",
|
||||
"wo.StatusUpdateDT", "wo.QuantityOrdered", "wo.QuantityCompleted");
|
||||
}
|
||||
|
||||
private Query ApplyFilters(Query query, SearchModel model)
|
||||
{
|
||||
if (model.TimespanFilterEnabled && model.MinimumDT.HasValue && model.MaximumDT.HasValue)
|
||||
{
|
||||
query = query.WhereBetween("wo.StatusUpdateDT", model.MinimumDT, model.MaximumDT);
|
||||
}
|
||||
|
||||
if (model.WorkOrderFilterEnabled)
|
||||
{
|
||||
query = query.WhereIn("wo.WorkOrderNumber",
|
||||
model.WorkOrderFilter.Select(w => w.WorkOrderNumber).ToList());
|
||||
}
|
||||
|
||||
if (model.ItemNumberFilterEnabled)
|
||||
{
|
||||
query = query.WhereIn("wo.ItemNumber",
|
||||
model.ItemNumberFilter.Select(i => i.ItemNumber).ToList());
|
||||
}
|
||||
|
||||
return query;
|
||||
}
|
||||
|
||||
private Query ApplyConditionalJoins(Query query, SearchModel model)
|
||||
{
|
||||
if (model.ProfitCenterFilterEnabled)
|
||||
{
|
||||
query = query
|
||||
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
|
||||
.Join("WorkCenter as wc", "wc.Code", "wos.WorkCenterCode")
|
||||
.Join("OrgHierarchy as oh", "oh.WorkCenterCode", "wc.Code")
|
||||
.WhereIn("oh.ProfitCenterCode",
|
||||
model.ProfitCenterFilter.Select(p => p.Code).ToList());
|
||||
}
|
||||
|
||||
if (model.WorkCenterFilterEnabled)
|
||||
{
|
||||
query = query
|
||||
.Join("WorkOrderStep_Curr as wos", "wos.WorkOrderNumber", "wo.WorkOrderNumber")
|
||||
.WhereIn("wos.WorkCenterCode",
|
||||
model.WorkCenterFilter.Select(w => w.Code).ToList());
|
||||
}
|
||||
|
||||
return query;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Work Order Traversal (Stored Procedure):**
|
||||
|
||||
The iterative downstream work order traversal is implemented as a stored procedure (`dbo.TraverseWorkOrders`) due to its iterative nature with temp tables and MERGE operations. The maximum iteration count is configurable via `SearchProcessingOptions.MaxTraversalIterations` (default: 20):
|
||||
|
||||
```csharp
|
||||
public class SearchProcessingOptions
|
||||
{
|
||||
public const string SectionName = "SearchProcessing";
|
||||
public int MaxTraversalIterations { get; set; } = 20;
|
||||
}
|
||||
```
|
||||
|
||||
**TraverseWorkOrders Stored Procedure Interface:**
|
||||
|
||||
```csharp
|
||||
public interface IWorkOrderTraversalService
|
||||
{
|
||||
Task<List<long>> TraverseDownstreamAsync(
|
||||
IEnumerable<long> seedWorkOrders,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public sealed class WorkOrderTraversalService : IWorkOrderTraversalService
|
||||
{
|
||||
private readonly IDbConnectionFactory _connectionFactory;
|
||||
private readonly IOptions<SearchProcessingOptions> _options;
|
||||
|
||||
public async Task<List<long>> TraverseDownstreamAsync(
|
||||
IEnumerable<long> seedWorkOrders,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
|
||||
|
||||
var maxIterations = _options.Value.MaxTraversalIterations;
|
||||
|
||||
return (await connection.QueryAsync<long>(
|
||||
"EXEC dbo.TraverseWorkOrders @p_SeedWorkOrders, @p_MaxIterations",
|
||||
new {
|
||||
p_SeedWorkOrders = seedWorkOrders.ToList().AsTableValuedParameter("dbo.WorkOrderList"),
|
||||
p_MaxIterations = maxIterations
|
||||
})).AsList();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**MIS Query Builder (Separate):**
|
||||
|
||||
For MIS data extraction, a separate `IMisQueryBuilder` SHALL be used to generate MIS-specific queries:
|
||||
|
||||
```csharp
|
||||
public interface IMisQueryBuilder
|
||||
{
|
||||
SearchQueryResult BuildMisDataQuery(SearchModel model);
|
||||
SearchQueryResult BuildMisNonMatchQuery(SearchModel model);
|
||||
}
|
||||
|
||||
public sealed class SqlKataMisQueryBuilder : IMisQueryBuilder
|
||||
{
|
||||
private readonly SqlServerCompiler _compiler = new();
|
||||
|
||||
public SearchQueryResult BuildMisDataQuery(SearchModel model)
|
||||
{
|
||||
var query = new Query("MisData as m")
|
||||
.Select("m.*")
|
||||
.Join("WorkOrderStep_Curr as wos", q => q
|
||||
.On("wos.ItemNumber", "m.ItemNumber")
|
||||
.On("wos.OperationNumber", "m.OperationNumber"));
|
||||
// Additional MIS-specific filtering...
|
||||
var compiled = _compiler.Compile(query);
|
||||
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
|
||||
}
|
||||
|
||||
public SearchQueryResult BuildMisNonMatchQuery(SearchModel model)
|
||||
{
|
||||
// Investigation query for router mismatches
|
||||
var query = new Query("WorkOrderStep_Curr as wos")
|
||||
.LeftJoin("MisData as m", q => q
|
||||
.On("wos.ItemNumber", "m.ItemNumber")
|
||||
.On("wos.OperationNumber", "m.OperationNumber"))
|
||||
.WhereNull("m.ItemNumber"); // Non-matches
|
||||
var compiled = _compiler.Compile(query);
|
||||
return new SearchQueryResult(compiled.Sql, compiled.NamedBindings);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Streaming vs Materialized Results:**
|
||||
|
||||
The system SHALL provide two methods for result retrieval:
|
||||
|
||||
```csharp
|
||||
public interface ISearchResultReader
|
||||
{
|
||||
/// <summary>Streaming - returns results as they are read (memory-efficient for large result sets)</summary>
|
||||
IAsyncEnumerable<SearchResult> StreamResultsAsync(int searchId, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Materialized - loads all results into memory (simpler for small result sets)</summary>
|
||||
Task<List<SearchResult>> GetResultsAsync(int searchId, CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
**SqlKata Benefits:**
|
||||
|
||||
- **Parameterized by default** - SQL injection protection built-in; all parameters use `@p_*` naming convention for clarity
|
||||
- **Fluent API** - Readable, composable query building
|
||||
- **Testable** - Unit test query building without database
|
||||
- **Type-safe** - Compile-time checking of method calls
|
||||
- **SQL Server optimized** - SqlServerCompiler generates optimized T-SQL
|
||||
|
||||
**Parameter Naming Convention:**
|
||||
|
||||
All SQL parameters SHALL use the `@p_` prefix for consistency:
|
||||
- `@p_SeedWorkOrders` (not `@SeedWorkOrders`)
|
||||
- `@p_MaxIterations` (not `@MaxIterations`)
|
||||
- `@p_MinimumDT`, `@p_MaximumDT`, etc.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `SearchModel` containing filter criteria and enabled filter flags
|
||||
- Table-valued parameters for each enabled filter type
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Complete T-SQL query string including:
|
||||
- Temporary table creation statements
|
||||
- Filter parameter population
|
||||
- Work order flagging logic
|
||||
- Downstream work order traversal
|
||||
- Final result selection
|
||||
- Optional MIS data extraction queries
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The query builder MUST create a `#Temp_WO` temporary table to track flagged work orders
|
||||
- The builder SHALL include SQL segments only for enabled filters (conditional generation)
|
||||
- Work order step searching is triggered when any of these filters are enabled: timespan, profit center, work center, or operator
|
||||
- The query MUST perform iterative downstream traversal (up to 20 iterations) to find work orders that received material from flagged work orders
|
||||
- Split orders (orders created by splitting a parent order) SHALL be automatically included
|
||||
- Query parameters MUST be passed as table-valued parameters for multi-value filters
|
||||
|
||||
#### Scenario: Generate query with work order filter only
|
||||
|
||||
- **WHEN** search criteria includes only work order numbers [12345, 67890]
|
||||
- **THEN** the system generates SQL that creates `#Temp_WO` with ManuallySpecified flag
|
||||
- **AND** includes MERGE statements to add specified work orders
|
||||
- **AND** includes downstream traversal logic for related work orders
|
||||
- **AND** excludes timespan, profit center, work center, and operator filter logic
|
||||
|
||||
#### Scenario: Generate query with profit center filter
|
||||
|
||||
- **WHEN** search criteria includes profit center codes ["PC01", "PC02"]
|
||||
- **THEN** the system generates SQL that creates `#P_WorkCenters` temp table
|
||||
- **AND** populates it by joining profit center codes to work centers via `OrgHierarchy`
|
||||
- **AND** includes work order step search logic with profit center join
|
||||
- **AND** sets `ShouldSearchSteps()` to return true
|
||||
|
||||
#### Scenario: Generate query with all filters enabled
|
||||
|
||||
- **WHEN** search criteria includes values for all eight filter types
|
||||
- **THEN** the system generates SQL with all conditional segments included
|
||||
- **AND** joins all filter temp tables appropriately
|
||||
- **AND** includes MIS extraction queries when `ExtractMisData` is true
|
||||
|
||||
#### Scenario: Generate query with operator filter
|
||||
|
||||
- **WHEN** search criteria includes operator user IDs ["USER1", "USER2"]
|
||||
- **THEN** the system generates SQL that creates `#P_OperatorIDs` temp table
|
||||
- **AND** resolves user IDs to address numbers via `JdeUser` table
|
||||
- **AND** joins work order time records to filter by operator
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Filter Entry Processing
|
||||
|
||||
The system SHALL enrich raw filter values with descriptive information from reference tables using C# record types for immutability.
|
||||
|
||||
#### Filter Entry Location
|
||||
|
||||
Filter entry types SHALL be located in the `JdeScoping.Core.Models` namespace within the Domain Models project, not in a separate processing project. This ensures proper separation of domain models from processing logic.
|
||||
|
||||
#### Migration Note: Record Types
|
||||
|
||||
The legacy filter entry classes should be converted to C# record types for value semantics and immutability:
|
||||
|
||||
```csharp
|
||||
// File: JdeScoping.Core/Models/FilterEntries.cs
|
||||
namespace JdeScoping.Core.Models;
|
||||
|
||||
// Legacy class pattern
|
||||
public class WorkOrderFilterEntry
|
||||
{
|
||||
public long WorkOrderNumber { get; set; }
|
||||
public string ItemNumber { get; set; }
|
||||
}
|
||||
|
||||
// Modern record pattern
|
||||
public record WorkOrderFilterEntry(long WorkOrderNumber, string ItemNumber);
|
||||
|
||||
// With attributes for Excel output configuration
|
||||
[OutputTable(TabName = "Work Order Filter", ShowHeader = true)]
|
||||
public record WorkOrderFilterEntry(
|
||||
[property: OutputColumn(Order = 10, HeaderText = "Work Order Number")]
|
||||
long WorkOrderNumber,
|
||||
[property: OutputColumn(Order = 20, HeaderText = "Item Number")]
|
||||
string ItemNumber
|
||||
);
|
||||
```
|
||||
|
||||
Records provide:
|
||||
- Immutability by default (init-only properties)
|
||||
- Value-based equality
|
||||
- Built-in `ToString()` and deconstruction
|
||||
- Cleaner syntax for data transfer objects
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Raw filter values from `SearchCriteria` (IDs, codes, numbers)
|
||||
- Reference data from `LotFinderDB` lookup methods
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Enriched filter entry records containing:
|
||||
- `WorkOrderFilterEntry`: WorkOrderNumber + ItemNumber
|
||||
- `ItemNumberFilterEntry`: ItemNumber + ItemDescription
|
||||
- `ProfitCenterFilterEntry`: Code + Description
|
||||
- `WorkCenterFilterEntry`: Code + Description
|
||||
- `OperatorFilterEntry`: AddressNumber + UserID + FullName
|
||||
- `ComponentLotFilterEntry`: LotNumber + ItemNumber
|
||||
- `ItemOperationMisFilterEntry`: ItemNumber + OperationNumber + MisNumber + MisRevision
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Filter entries SHALL be populated via `LotFinderDB` lookup methods
|
||||
- Lookup methods MUST return empty collections for invalid or unmatched values
|
||||
- Operator filter entries MUST resolve user IDs to address numbers for query execution
|
||||
- Component lot entries MUST include both lot number and item number for proper traceability
|
||||
|
||||
#### Scenario: Enrich work order filter entries
|
||||
|
||||
- **WHEN** the system processes work order numbers [12345, 67890]
|
||||
- **THEN** it calls `LotFinderDB.LookupWorkorders()` with the numbers
|
||||
- **AND** creates `WorkOrderFilterEntry` records with work order number and item number
|
||||
- **AND** includes only work orders that exist in the database
|
||||
|
||||
#### Scenario: Resolve operator user IDs to address numbers
|
||||
|
||||
- **WHEN** the system processes operator user IDs ["JSMITH", "MBROWN"]
|
||||
- **THEN** it calls `LotFinderDB.LookupUsers()` to get JDE user records
|
||||
- **AND** creates `OperatorFilterEntry` records with AddressNumber, UserID, and FullName
|
||||
- **AND** the AddressNumber is used for joining to work order time records
|
||||
|
||||
#### Scenario: Handle invalid filter values
|
||||
|
||||
- **WHEN** the system processes item numbers containing non-existent items
|
||||
- **THEN** the lookup returns only matching items
|
||||
- **AND** non-existent item numbers are silently excluded from results
|
||||
|
||||
#### Scenario: Process component lot filter with item validation
|
||||
|
||||
- **WHEN** the system processes a component lot filter entry
|
||||
- **THEN** it validates both lot number and item number exist together
|
||||
- **AND** uses the combination to trace downstream work orders via `WorkOrderComponent` and `LotUsage` tables
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Work Order Flagging and Traversal
|
||||
|
||||
The system SHALL flag work orders for inclusion based on multiple criteria and traverse downstream relationships.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Filter criteria from `SearchModel`
|
||||
- Work order relationships from database tables:
|
||||
- `WorkOrder` (parent/child via ParentWorkOrderNumber)
|
||||
- `WorkOrderComponent` (parts list relationships)
|
||||
- `LotUsage` (CARDEX material usage)
|
||||
- `WorkOrderStep` (operation details)
|
||||
- `WorkOrderTime` (operator time records)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `#Temp_WO` temporary table containing:
|
||||
- `WorkOrderNumber`: Primary key
|
||||
- `LotNumber`: Associated lot number
|
||||
- `BranchCode`: Branch code for the work order
|
||||
- `ShortItemNumber`: Item's short number
|
||||
- `ManuallySpecified`: Flag indicating direct specification
|
||||
- `SplitOrder`: Flag indicating split from flagged order
|
||||
- `CARDEX`: Flag indicating material receipt from flagged order (F4111) - **Note: Also set for WorkOrderComponent matches**
|
||||
- `PartsList`: Flag indicating parts list relationship (F3111)
|
||||
- `Flagged`: Flag indicating filter criteria match
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Work orders directly specified in the filter SHALL be marked with `ManuallySpecified = 1`
|
||||
- Work orders split from flagged orders (matching `ParentWorkOrderNumber`) SHALL be marked with `SplitOrder = 1`
|
||||
- Work orders receiving material from flagged lots via `WorkOrderComponent` SHALL be marked with `CARDEX = 1` (not PartsList as might be expected)
|
||||
- Work orders receiving material from flagged lots via `LotUsage` SHALL be marked with `CARDEX = 1`
|
||||
- Work orders matching filter criteria (timespan, profit center, work center, operator) SHALL be marked with `Flagged = 1`
|
||||
- Downstream traversal SHALL iterate up to 20 times to find all related work orders
|
||||
- Traversal stops when no new work orders are found or maximum iterations reached
|
||||
|
||||
#### Scenario: Flag manually specified work orders
|
||||
|
||||
- **WHEN** work order 12345 is specified in the work order filter
|
||||
- **THEN** the system marks it with `ManuallySpecified = 1` in `#Temp_WO`
|
||||
- **AND** finds any split orders from 12345 and marks them with `SplitOrder = 1`
|
||||
|
||||
#### Scenario: Trace downstream work orders via parts list
|
||||
|
||||
- **WHEN** work order 12345 produces lot "LOT-A" for item "ITEM-001"
|
||||
- **AND** work order 67890 consumes "LOT-A" for "ITEM-001" via parts list
|
||||
- **THEN** work order 67890 is added to `#Temp_WO` with `CARDEX = 1`
|
||||
|
||||
#### Scenario: Trace downstream work orders via CARDEX
|
||||
|
||||
- **WHEN** work order 12345 produces lot "LOT-A"
|
||||
- **AND** work order 67890 consumes "LOT-A" via LotUsage (CARDEX)
|
||||
- **THEN** work order 67890 is added to `#Temp_WO` with `CARDEX = 1`
|
||||
|
||||
#### Scenario: Multi-level downstream traversal
|
||||
|
||||
- **WHEN** work order A produces material consumed by work order B
|
||||
- **AND** work order B produces material consumed by work order C
|
||||
- **THEN** the iterative traversal finds A, B, and C
|
||||
- **AND** marks each with appropriate flags based on relationship type
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Search Result Aggregation
|
||||
|
||||
The system SHALL aggregate flagged work orders into structured result objects with complete details.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Flagged work orders from `#Temp_WO`
|
||||
- Work order details from `WorkOrder` table
|
||||
- Item details from `Item` table
|
||||
- Status details from `StatusCode` table
|
||||
- Latest step from `WorkOrderStep` table
|
||||
- Scrap totals from `WorkOrderTotalScrap` table
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `SearchResult` objects containing:
|
||||
- Work order identification (number, branch code, lot number, item number)
|
||||
- Item details (planning family, stocking type)
|
||||
- Quantities (order, held, scrapped, shipped)
|
||||
- Latest operation step details
|
||||
- Status information (code, description, update timestamp)
|
||||
- Inclusion reason derived from flag combination
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each result SHALL include the latest operation step (highest EndDT, then highest StepNumber)
|
||||
- Inclusion reason SHALL be calculated with strict priority order:
|
||||
1. **ManuallySpecified** (highest) - Work order was directly specified in filter
|
||||
2. **Flagged** - Work order matched timespan/profit center/work center/operator criteria
|
||||
3. **ComponentUsage** - Work order received material from a flagged work order
|
||||
4. **SplitOrder** (lowest) - Work order was split from a flagged parent order
|
||||
- Component usage reason SHALL distinguish between CARDEX only, Parts List only, or both
|
||||
- Scrapped quantity SHALL default to 0 when no scrap records exist
|
||||
- Results with null latest step are still included (work orders without operations)
|
||||
- Inclusion reason MAY return "UNKNOWN" when no flags are set (edge case)
|
||||
|
||||
#### Scenario: Generate inclusion reason for manually specified order
|
||||
|
||||
- **WHEN** a work order is marked with `ManuallySpecified = 1`
|
||||
- **THEN** the `InclusionReason` property returns "ManuallySpecified"
|
||||
|
||||
#### Scenario: Generate inclusion reason for flagged order
|
||||
|
||||
- **WHEN** a work order is marked with `Flagged = 1` and `ManuallySpecified = 0`
|
||||
- **THEN** the `InclusionReason` property returns "Flagged"
|
||||
|
||||
#### Scenario: Generate inclusion reason for component usage with both sources
|
||||
|
||||
- **WHEN** a work order is marked with `CARDEX = 1` and `PartsList = 1`
|
||||
- **THEN** the `InclusionReason` property returns "ComponentUsage (CARDEX + Parts List)"
|
||||
|
||||
#### Scenario: Retrieve latest operation step for work order
|
||||
|
||||
- **WHEN** work order 12345 has steps 10, 20, 30 with EndDT values
|
||||
- **THEN** the result includes the step with the highest EndDT
|
||||
- **AND** if EndDT values tie, the highest StepNumber wins
|
||||
|
||||
---
|
||||
|
||||
### Requirement: MIS Data Extraction
|
||||
|
||||
The system SHALL optionally extract Manufacturing Information System (MIS) data for matching work orders.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `ExtractMisData` flag from search criteria
|
||||
- `ItemOperationMisFilter` entries (when filtering by specific MIS)
|
||||
- Work order step data joined with MIS matching function `dbo.MatchMIS`
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `MisSearchResult` objects containing:
|
||||
- Item and MIS identification
|
||||
- Matching indicators (RoutingMatch, MasterMatch)
|
||||
- Test descriptions, sampling information
|
||||
- Tools/gauges and work instructions
|
||||
- `MisNonMatchSearchResult` objects containing:
|
||||
- Work orders with steps that did not match MIS records
|
||||
- Added job step indicators
|
||||
- Matched job step suggestions based on work center and function code
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- MIS extraction is only performed when `ExtractMisData = true`
|
||||
- The `MatchMIS` table-valued function handles MIS record matching logic
|
||||
- RoutingMatch indicates match to F3112Z1 (work order routing) records
|
||||
- MasterMatch indicates match to F3003 (item master routing) records
|
||||
- Non-match results include work orders where neither RoutingMatch nor MasterMatch is true, or MisNumber is null
|
||||
- Added job step detection uses routing type and F3111 records to identify manually added steps
|
||||
- Matched job step number suggests alternative step matching by work center and function code
|
||||
- **Note**: MIS extraction does NOT join `#Temp_WO` - work order/component lot filters do NOT constrain MIS results
|
||||
- **Note**: Timespan filtering in MIS extraction requires BOTH min and max values for the combined condition; separate handling exists for min-only or max-only cases
|
||||
|
||||
#### Scenario: Extract MIS data for filtered work orders
|
||||
|
||||
- **WHEN** search criteria has `ExtractMisData = true`
|
||||
- **THEN** the system generates MIS extraction queries
|
||||
- **AND** populates `#TempMisData` with MIS-matched step data
|
||||
- **AND** returns `MisSearchResult` list with matching records
|
||||
|
||||
#### Scenario: Identify MIS non-matches for investigation
|
||||
|
||||
- **WHEN** MIS data is extracted and a step has no routing or master match
|
||||
- **THEN** the system includes it in `MisNonMatchSearchResult` list
|
||||
- **AND** indicates whether the job step was added (not in original routing)
|
||||
- **AND** suggests matched job step number if one exists with same work center and function code
|
||||
|
||||
#### Scenario: Filter by specific MIS number and revision
|
||||
|
||||
- **WHEN** search criteria includes ItemOperationMisFilter with specific MIS entries
|
||||
- **THEN** the query joins to `#P_PartOperations` temp table
|
||||
- **AND** only returns work order steps matching the specified item/operation/MIS/revision combinations
|
||||
- **AND** the WorkOrderTime UNION branch is skipped (only WorkOrderStep branch used)
|
||||
|
||||
#### Scenario: Handle NMR routing type
|
||||
|
||||
- **WHEN** a work order has routing type "NMR" (No Master Routing)
|
||||
- **THEN** all job steps are considered added (`WasJobStepAdded = 1`)
|
||||
- **AND** matched job step number is null (no routing to match against)
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Table-Valued Parameter Creation
|
||||
|
||||
The system SHALL create SQL table-valued parameters for efficient filter data transmission using Microsoft.Data.SqlClient.
|
||||
|
||||
#### Migration Note: SqlClient and TVP Options
|
||||
|
||||
**SQL Client Package Change (Required)**
|
||||
|
||||
Replace `System.Data.SqlClient` with `Microsoft.Data.SqlClient`:
|
||||
|
||||
```csharp
|
||||
// Legacy
|
||||
using System.Data.SqlClient;
|
||||
|
||||
// Modern .NET 10
|
||||
using Microsoft.Data.SqlClient;
|
||||
```
|
||||
|
||||
This is a cross-cutting change affecting all database access code. The API is largely compatible but `Microsoft.Data.SqlClient` is the actively maintained package with security updates.
|
||||
|
||||
**TVP Implementation Options**
|
||||
|
||||
The legacy approach using `DataTable` remains valid. An alternative using `IEnumerable<SqlDataRecord>` offers better performance for large datasets:
|
||||
|
||||
```csharp
|
||||
// Option 1: DataTable approach (legacy pattern, still supported)
|
||||
public static SqlMapper.ICustomQueryParameter CreateWorkOrderFilterParameter(
|
||||
this SearchModel model)
|
||||
{
|
||||
var dataTable = new DataTable();
|
||||
dataTable.Columns.Add("WorkOrderNumber", typeof(long));
|
||||
foreach (var entry in model.WorkOrderFilter)
|
||||
{
|
||||
dataTable.Rows.Add(entry.WorkOrderNumber);
|
||||
}
|
||||
return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
|
||||
}
|
||||
|
||||
// Option 2: SqlDataRecord approach (more efficient for large datasets)
|
||||
public static IEnumerable<SqlDataRecord> ToWorkOrderRecords(
|
||||
this IEnumerable<WorkOrderFilterEntry> entries)
|
||||
{
|
||||
var metadata = new SqlMetaData("WorkOrderNumber", SqlDbType.BigInt);
|
||||
var record = new SqlDataRecord(metadata);
|
||||
|
||||
foreach (var entry in entries)
|
||||
{
|
||||
record.SetInt64(0, entry.WorkOrderNumber);
|
||||
yield return record;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `SearchModel` with populated filter entry collections
|
||||
- SQL Server table type definitions for each parameter type
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `SqlMapper.ICustomQueryParameter` objects created via `AsTableValuedParameter()`:
|
||||
- `WorkOrderFilterParameter`: WorkOrderNumber column
|
||||
- `ItemNumberFilterParameter`: ItemNumber column
|
||||
- `ProfitCenterFilterParameter`: Code column
|
||||
- `WorkCenterFilterParameter`: Code column
|
||||
- `ComponentLotFilterParameter`: ComponentLotNumber + ItemNumber columns
|
||||
- `OperatorFilterParameter`: UserName column
|
||||
- `ItemOperationMisFilterParameter`: ItemNumber + OperationNumber + MisNumber + MisRevision columns
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Each parameter type MUST match the corresponding SQL Server table type schema
|
||||
- DataTable column types MUST match the expected SQL types (long, string)
|
||||
- Parameters are created even for empty filter collections (empty DataTable)
|
||||
- Operator filter uses UserID (not AddressNumber) as the parameter value
|
||||
- All database operations SHOULD use async methods (`QueryAsync`, `ExecuteAsync`)
|
||||
|
||||
#### Scenario: Create work order filter parameter
|
||||
|
||||
- **WHEN** SearchModel contains WorkOrderFilter with entries [12345, 67890]
|
||||
- **THEN** the system creates a DataTable with WorkOrderNumber column
|
||||
- **AND** populates two rows with the work order numbers
|
||||
- **AND** returns parameter of type "WorkOrderFilterParameter"
|
||||
|
||||
#### Scenario: Create component lot filter parameter
|
||||
|
||||
- **WHEN** SearchModel contains ComponentLotFilter with lot "LOT-A" for item "ITEM-001"
|
||||
- **THEN** the system creates a DataTable with ComponentLotNumber and ItemNumber columns
|
||||
- **AND** populates one row with both values
|
||||
- **AND** returns parameter of type "ComponentLotFilterParameter"
|
||||
|
||||
#### Scenario: Create empty filter parameter
|
||||
|
||||
- **WHEN** SearchModel contains empty WorkOrderFilter collection
|
||||
- **THEN** the system creates a DataTable with WorkOrderNumber column
|
||||
- **AND** the DataTable has zero rows
|
||||
- **AND** returns valid parameter that produces empty result set
|
||||
|
||||
#### Scenario: Create item operation MIS filter parameter
|
||||
|
||||
- **WHEN** SearchModel contains ItemOperationMisFilter with one entry
|
||||
- **THEN** the system creates a DataTable with four columns
|
||||
- **AND** populates ItemNumber, OperationNumber, MisNumber, MisRevision
|
||||
- **AND** returns parameter of type "ItemOperationMisFilterParameter"
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Output Column Configuration
|
||||
|
||||
The system SHALL use attribute-based configuration for Excel output column formatting.
|
||||
|
||||
#### Migration Note: Attributes and Reflection
|
||||
|
||||
The attribute-based pattern works well in .NET 10 with standard reflection. For high-performance scenarios, consider:
|
||||
|
||||
1. **Source generators** - Generate column metadata at compile time
|
||||
2. **Cached reflection** - Cache `PropertyInfo` and attribute lookups on first access
|
||||
|
||||
```csharp
|
||||
// Attribute pattern preserved from legacy (works with records)
|
||||
[OutputTable(TabName = "Search Results", TableName = "Search_Results")]
|
||||
public record SearchResult(
|
||||
[property: OutputColumn(Order = 10, HeaderText = "Work Order")]
|
||||
long WorkOrderNumber,
|
||||
|
||||
[property: OutputColumn(Order = 20, HeaderText = "Status Date",
|
||||
Format = OutputColumnAttribute.DATE_FORMAT)]
|
||||
DateTime? StatusUpdateDT
|
||||
);
|
||||
|
||||
// Cached reflection helper
|
||||
public static class OutputColumnCache
|
||||
{
|
||||
private static readonly ConcurrentDictionary<Type, IReadOnlyList<OutputColumn>> _cache = new();
|
||||
|
||||
public static IReadOnlyList<OutputColumn> GetColumns<T>() =>
|
||||
_cache.GetOrAdd(typeof(T), type => BuildColumns(type));
|
||||
}
|
||||
```
|
||||
|
||||
#### Inputs
|
||||
|
||||
- `OutputColumnAttribute` on result model properties:
|
||||
- `Order`: Display order (integer)
|
||||
- `HeaderText`: Column header text
|
||||
- `Format`: Excel format string
|
||||
- `AutoWidth`: Whether to auto-size column
|
||||
- `Width`: Manual width when AutoWidth is false
|
||||
- `WrapText`: Whether to wrap text in cells
|
||||
- `OutputTableAttribute` on result model classes:
|
||||
- `TabName`: Excel worksheet tab name
|
||||
- `TableName`: Table identifier for styling
|
||||
- `ShowHeader`: Whether to display merged header
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `OutputColumn` objects combining property info with attribute metadata
|
||||
- Configured Excel worksheets with proper formatting
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Columns are ordered by the `Order` attribute value (ascending)
|
||||
- Properties without `OutputColumnAttribute` are excluded from output
|
||||
- Standard format `"@"` treats values as text
|
||||
- Date format `"[$-409]MM/dd/yyyy;@"` applies US English date formatting
|
||||
- Timestamp format `"[$-409]m/d/yy h:mm AM/PM;@"` includes time component
|
||||
- Wrapped columns use fixed width of 65 characters by default
|
||||
- Filter entry tables show headers; result tables do not
|
||||
|
||||
#### Scenario: Configure date column formatting
|
||||
|
||||
- **WHEN** a property has `Format = OutputColumnAttribute.DATE_FORMAT`
|
||||
- **THEN** Excel displays values in MM/dd/yyyy format
|
||||
- **AND** the column width auto-sizes to fit content
|
||||
|
||||
#### Scenario: Configure wrapped text column
|
||||
|
||||
- **WHEN** a property has `WrapText = true` and `AutoWidth = false`
|
||||
- **THEN** Excel wraps long text within the cell
|
||||
- **AND** column width is set to `WRAPPED_COLUMN_WIDTH` (65 characters)
|
||||
|
||||
#### Scenario: Order columns in output
|
||||
|
||||
- **WHEN** SearchResult has properties with Order values 10, 20, 30
|
||||
- **THEN** Excel columns appear in ascending order by the Order value
|
||||
- **AND** gaps in Order values are allowed
|
||||
|
||||
#### Scenario: Configure tab name for result type
|
||||
|
||||
- **WHEN** SearchResult has `OutputTableAttribute` with `TabName = "Search Results"`
|
||||
- **THEN** the Excel worksheet tab is named "Search Results"
|
||||
- **AND** the table is styled with identifier "Search_Results"
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
| Legacy Pattern | New Pattern | Rationale |
|
||||
|----------------|-------------|-----------|
|
||||
| T4 Text Template (`QueryTemplate.tt`) | **SqlKata** fluent query builder | SqlKata provides parameterized SQL by default, fluent API, composable queries, testability, and SQL Server-optimized output; T4 templates are not well-supported in modern .NET SDK projects |
|
||||
| Iterative work order traversal in T4 | Stored procedure `dbo.TraverseWorkOrders` | Iterative logic with temp tables and MERGE is better handled server-side; stored procedure reduces round trips and is transactionally consistent |
|
||||
| `System.Data.SqlClient` | `Microsoft.Data.SqlClient` | Legacy package is deprecated and no longer receives security updates; Microsoft.Data.SqlClient is the supported replacement |
|
||||
| Filter entry classes | C# record types | Records provide immutability, value equality, and cleaner syntax for DTOs |
|
||||
| Synchronous Dapper calls | Async Dapper methods (`QueryAsync`, `ExecuteAsync`) | Async operations improve scalability and are idiomatic in modern .NET |
|
||||
| `DataTable.AsTableValuedParameter()` | DataTable (unchanged) or `IEnumerable<SqlDataRecord>` | DataTable approach works; SqlDataRecord is more efficient for large datasets |
|
||||
| Conditional string concatenation in T4 | SqlKata fluent conditional methods (`.Join()`, `.WhereIn()`, `.When()`) | SqlKata handles conditional query building natively with compile-time safety |
|
||||
| `SqlMapper.ICustomQueryParameter` (Dapper) | Dapper table-valued parameters (unchanged) | Dapper's API remains compatible in .NET 10 |
|
||||
| Attribute-based output configuration | Attributes with optional cached reflection | Attributes work well; cached reflection improves performance |
|
||||
| Newtonsoft.Json for criteria serialization | System.Text.Json (recommended) or Newtonsoft.Json | System.Text.Json is built-in with better performance; Newtonsoft.Json works if polymorphism needed |
|
||||
|
||||
### SqlKata Integration
|
||||
|
||||
**NuGet Packages:**
|
||||
- `SqlKata` - Core query builder
|
||||
- `SqlKata.Execution` - Dapper integration for query execution
|
||||
|
||||
**DI Registration:**
|
||||
```csharp
|
||||
services.AddScoped<ISearchQueryBuilder, SqlKataSearchQueryBuilder>();
|
||||
services.AddScoped<IWorkOrderTraversalService, WorkOrderTraversalService>();
|
||||
services.AddSingleton<SqlServerCompiler>();
|
||||
```
|
||||
|
||||
**SqlKata with Dapper Execution:**
|
||||
```csharp
|
||||
// Build query with SqlKata
|
||||
var (sql, parameters) = _queryBuilder.BuildSearchQuery(model);
|
||||
|
||||
// Execute with Dapper
|
||||
var results = await connection.QueryAsync<SearchResult>(sql, parameters);
|
||||
```
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision: 20-iteration limit for downstream traversal
|
||||
|
||||
The 20-iteration limit (`@c_MAX_RUNS INT = 20`) for downstream work order traversal is **retained as a fixed value**. Rationale:
|
||||
- Manufacturing processes rarely exceed 20 levels of downstream consumption
|
||||
- A configurable limit adds complexity without practical benefit
|
||||
- The limit protects against runaway queries from circular references
|
||||
|
||||
### Decision: Circular reference handling
|
||||
|
||||
Circular references in work order relationships are **handled implicitly** by the MERGE statement's `ON` clause. When a work order is already in `#Temp_WO`, the MERGE updates rather than inserts, preventing infinite loops. No additional detection logic is required.
|
||||
|
||||
### Decision: MIS non-match results
|
||||
|
||||
MIS non-match results are **always included** when `ExtractMisData = true`. The non-match data is valuable for quality investigations and should not be optional.
|
||||
|
||||
### Decision: MatchMIS function availability
|
||||
|
||||
The `dbo.MatchMIS` table-valued function **must be recreated** in the migrated database. It is a custom SQL Server function that implements MIS record matching logic based on work order routing, master routing, and operation parameters.
|
||||
|
||||
### Decision: Filter validation location
|
||||
|
||||
Filter validation (e.g., requiring at least one filter) is enforced **at the UI/API level**, not in the model. This allows the model to represent any valid state while the API layer enforces business rules about minimum filter requirements.
|
||||
|
||||
### Decision: Large result set handling
|
||||
|
||||
Large result sets are handled via **streaming to Excel**. The EPPlus library supports streaming writes, and the existing pattern of writing directly to the output stream is preserved. Pagination is not implemented at the search level.
|
||||
|
||||
---
|
||||
|
||||
## Codex Review Findings (Spec Accuracy Issues)
|
||||
|
||||
The following inaccuracies were identified during review and have been addressed in this specification:
|
||||
|
||||
1. **MIS Extraction Not Filtered**: ~~The spec states MIS extraction joins `#Temp_WO` or `#P_PartOperations` to constrain results.~~ **CORRECTED**: Spec now documents that MIS extraction does NOT join `#Temp_WO`, so work-order/component-lot/item-operation-MIS filters do NOT constrain MIS results. See `QueryTemplate.tt:173,353`.
|
||||
|
||||
2. **Timespan Filter Requires Both Bounds**: ~~The spec says either min or max boundary enables filtering.~~ **CORRECTED**: Spec now documents that the `LU_WO` timespan filtering requires BOTH min and max values for the combined condition, with separate handling for MIS-only min/max cases. See `QueryTemplate.tt:254,371`, `LotFinderDBExt.cs:157`.
|
||||
|
||||
3. **WorkOrderComponent Flagged as CARDEX**: ~~The spec implies `PartsList` flag is set for WorkOrderComponent paths.~~ **CORRECTED**: Spec now explicitly states component-lot matches from `WorkOrderComponent` are flagged as CARDEX, not PartsList. See `QueryTemplate.tt:105`.
|
||||
|
||||
4. **ItemOperationMisFilter Skips WorkOrderTime**: ~~Not documented.~~ **CORRECTED**: Spec now documents that when `ItemOperationMisFilterEnabled` is true, the WorkOrderTime UNION branch is skipped. See `QueryTemplate.tt:258`.
|
||||
|
||||
5. **InclusionReason Edge Cases**: ~~Spec mentions "Split order" but not "UNKNOWN" case.~~ **CORRECTED**: Spec now documents that `InclusionReason` can return `"UNKNOWN"` when no flags are set. See `SearchResult.cs:152`.
|
||||
|
||||
6. **Error Handling Not Documented**: Query timeout, debug SQL/Excel writes, and marking search failed on exception are legacy behaviors addressed in the separate error-handling specification. See `LotFinderDBExt.cs:136`, `WorkProcessor.cs:158`.
|
||||
@@ -0,0 +1,302 @@
|
||||
# SQL Business Logic Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
This specification documents stored procedures and functions that implement database-level business logic for the search processing workflow. These SQL objects manage the lifecycle of search requests (submission, execution, completion) and provide complex matching logic for Manufacturing Information System (MIS) data.
|
||||
|
||||
## Source Reference
|
||||
|
||||
| Legacy File | Purpose |
|
||||
|-------------|---------|
|
||||
| OLD/Database/StoredProcedures/SubmitSearch.sql | Create new search record with Queued status |
|
||||
| OLD/Database/StoredProcedures/StartSearch.sql | Transition search to Processing status |
|
||||
| OLD/Database/StoredProcedures/CompleteSearch.sql | Finalize search with results or failure |
|
||||
| OLD/Database/StoredProcedures/ResetPartialSearches.sql | Recovery procedure for stuck searches |
|
||||
| OLD/Database/Functions/MatchMis.sql | MIS data matching based on routing/master |
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Submit search procedure
|
||||
|
||||
The system SHALL provide a SubmitSearch stored procedure that creates new search records in the database.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- @p_UserName (VARCHAR(128)) - Username of the person submitting the search
|
||||
- @p_Name (VARCHAR(128)) - User-provided name/description for the search
|
||||
- @p_Criteria (VARCHAR(MAX)) - JSON-serialized search criteria
|
||||
|
||||
#### Outputs
|
||||
|
||||
- @o_SearchID (INT OUTPUT) - The auto-generated ID of the newly created search record
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The procedure SHALL insert a new record into the Search table
|
||||
- Status SHALL be set to 1 (Queued) for all new searches
|
||||
- SubmitDT SHALL be set to the current date/time via GETDATE()
|
||||
- The new SearchId SHALL be retrieved using SCOPE_IDENTITY() to ensure the correct ID in concurrent scenarios
|
||||
|
||||
#### Scenario: Submit new search
|
||||
|
||||
- **WHEN** SubmitSearch is called with valid username, search name, and criteria
|
||||
- **THEN** a new Search record is inserted with Status = 1 (Queued)
|
||||
- **AND** SubmitDT is set to the current timestamp
|
||||
- **AND** the new SearchId is returned via the @o_SearchID OUTPUT parameter
|
||||
|
||||
#### Scenario: Concurrent search submissions
|
||||
|
||||
- **WHEN** multiple users submit searches simultaneously
|
||||
- **THEN** each call returns the correct SearchId via SCOPE_IDENTITY()
|
||||
- **AND** no cross-session ID contamination occurs
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Start search procedure
|
||||
|
||||
The system SHALL provide a StartSearch stored procedure that marks a search as being actively processed.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- @p_SearchID (INT) - The ID of the search to start processing
|
||||
|
||||
#### Outputs
|
||||
|
||||
- None (procedure updates record in place)
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The procedure SHALL update the Status to 2 (Processing)
|
||||
- The procedure SHALL set StartDT to the current date/time via GETDATE()
|
||||
- Only the search record matching the provided ID SHALL be updated
|
||||
|
||||
#### Scenario: Start processing a queued search
|
||||
|
||||
- **WHEN** StartSearch is called with a valid SearchID
|
||||
- **THEN** the search Status is updated to 2 (Processing)
|
||||
- **AND** StartDT is set to the current timestamp
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Complete search procedure
|
||||
|
||||
The system SHALL provide a CompleteSearch stored procedure that finalizes a search with results or failure status.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- @p_SearchID (INT) - The ID of the search to complete
|
||||
- @p_WasSuccessful (BIT) - Flag indicating success (1) or failure (0)
|
||||
- @p_Results (VARBINARY(MAX)) - Binary Excel file data (may be NULL on failure)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- None (procedure updates record in place)
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- When @p_WasSuccessful = 1, Status SHALL be set to 3 (Complete)
|
||||
- When @p_WasSuccessful = 0, Status SHALL be set to 4 (Failed)
|
||||
- The Results column SHALL be updated with the provided binary data
|
||||
- EndDT SHALL be set to the current date/time via GETDATE()
|
||||
|
||||
#### Scenario: Complete successful search
|
||||
|
||||
- **WHEN** CompleteSearch is called with WasSuccessful = 1 and Excel binary data
|
||||
- **THEN** the search Status is updated to 3 (Complete)
|
||||
- **AND** the Results column contains the Excel binary
|
||||
- **AND** EndDT is set to the current timestamp
|
||||
|
||||
#### Scenario: Complete failed search
|
||||
|
||||
- **WHEN** CompleteSearch is called with WasSuccessful = 0
|
||||
- **THEN** the search Status is updated to 4 (Failed)
|
||||
- **AND** EndDT is set to the current timestamp
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Reset partial searches procedure
|
||||
|
||||
The system SHALL provide a ResetPartialSearches stored procedure that recovers searches stuck in a processing state.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- None (parameterless procedure)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- None (procedure updates records in place)
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The procedure SHALL identify searches where StartDT IS NOT NULL AND EndDT IS NULL
|
||||
- These searches represent work that was started but never completed (e.g., service crash)
|
||||
- Status SHALL be reset to 1 (Queued) for all identified searches
|
||||
- StartDT SHALL be set to NULL to allow re-processing
|
||||
|
||||
#### Scenario: Service restart recovery
|
||||
|
||||
- **WHEN** the worker service starts and calls ResetPartialSearches
|
||||
- **THEN** all searches with StartDT set but EndDT NULL are reset
|
||||
- **AND** Status is changed back to 1 (Queued)
|
||||
- **AND** StartDT is cleared to NULL
|
||||
- **AND** these searches become eligible for re-processing
|
||||
|
||||
#### Scenario: No stuck searches
|
||||
|
||||
- **WHEN** ResetPartialSearches is called with no searches in partial state
|
||||
- **THEN** no records are modified
|
||||
|
||||
---
|
||||
|
||||
### Requirement: MIS matching function
|
||||
|
||||
The system SHALL provide a MatchMIS table-valued function that correlates work order operations with Manufacturing Information System (MIS) quality documents.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- @workOrderNumber (BIGINT) - Work order number to match
|
||||
- @itemNumber (VARCHAR(25)) - Item/part number
|
||||
- @branchCode (VARCHAR(12)) - Branch/plant code
|
||||
- @routingType (VARCHAR(3)) - Routing type identifier
|
||||
- @issueDate (DATETIME) - Issue date for routing validity
|
||||
- @workCenterCode (VARCHAR(12)) - Work center identifier
|
||||
- @sequenceNumber (DECIMAL(7,2)) - Operation sequence number
|
||||
- @steptimestamp (DATETIME) - Timestamp of the operation step
|
||||
- @functionCode (VARCHAR(15)) - Function/operation code
|
||||
- @functionOperationDescription (VARCHAR(80)) - Description of the operation
|
||||
|
||||
#### Outputs
|
||||
|
||||
Returns a table with the following columns:
|
||||
- WorkOrderNumber, ItemNumber, ItemDescription, BranchCode, WorkCenterCode
|
||||
- StepTimestamp, SequenceNumber, FunctionCode, FunctionOperationDescription
|
||||
- MatchedSequenceNumber (DECIMAL(7,2)) - Sequence number used for MIS lookup
|
||||
- RoutingMatch (BIT) - Whether match came from work order routing
|
||||
- MasterMatch (BIT) - Whether match came from route master
|
||||
- MisNumber, RevID, CharNumber, MisSequenceNumber (VARCHAR(32)) - MIS document identifiers
|
||||
- TestDescription, ToolsGauges, WorkInstructions (VARCHAR(2000)) - MIS content
|
||||
- SamplingType, SamplingValue, Status (VARCHAR(32)) - MIS metadata
|
||||
- ReleaseDate (DATETIME) - MIS document release date
|
||||
|
||||
#### Business Rules
|
||||
|
||||
1. **Parent Work Order Resolution**: The function SHALL resolve to the parent work order number if available (using TRY_CONVERT to handle non-numeric parent references)
|
||||
|
||||
2. **Sequence Number Alias Discovery**: The function SHALL find alias sequence numbers using a two-tier approach:
|
||||
- First, check WorkOrderRouting (F3112Z1) for matches on work center and function code using the earliest transaction date
|
||||
- If no routing matches exist, check RouteMaster (F3003) for matches within the issue date validity range
|
||||
|
||||
3. **MIS Matching Priority**: The function SHALL attempt MIS matching in priority order:
|
||||
- First priority: Match MisData records with Status = 'Current' where steptimestamp falls within ReleaseDate to ObsoleteDate
|
||||
- Second priority: Match MisData records with Status = 'BackLevel' where ReleaseDate falls between issueDate and steptimestamp
|
||||
- Third priority: Return alias information without MIS data if aliases exist but no MIS match
|
||||
- Fourth priority: Return input parameters with NULL matched sequence and both match flags = 0
|
||||
|
||||
4. **Item Description Lookup**: The function SHALL lookup and include the item description from the Item table
|
||||
|
||||
5. **Date Range Handling**: NULL ReleaseDate SHALL be treated as '1970-01-01', NULL ObsoleteDate SHALL be treated as '2029-01-01'
|
||||
|
||||
#### Scenario: Match via work order routing with current MIS
|
||||
|
||||
- **WHEN** MatchMIS is called for a work order with routing data
|
||||
- **AND** a current MIS document exists for the matched sequence
|
||||
- **THEN** results include RoutingMatch = 1 and full MIS details
|
||||
- **AND** MasterMatch indicates whether route master also matched
|
||||
|
||||
#### Scenario: Match via route master when no routing exists
|
||||
|
||||
- **WHEN** MatchMIS is called for a work order without specific routing
|
||||
- **AND** a route master entry exists for the item/branch/work center
|
||||
- **THEN** results include MasterMatch = 1 and RoutingMatch = 0
|
||||
- **AND** MIS details are included if available
|
||||
|
||||
#### Scenario: BackLevel MIS fallback
|
||||
|
||||
- **WHEN** no current MIS document matches
|
||||
- **AND** a BackLevel MIS document exists with ReleaseDate in the valid range
|
||||
- **THEN** results include the BackLevel MIS details with Status = 'BackLevel'
|
||||
|
||||
#### Scenario: No MIS match found
|
||||
|
||||
- **WHEN** no MIS documents match the routing or master sequence
|
||||
- **THEN** results include the input parameters with alias information
|
||||
- **AND** MIS-specific columns are NULL
|
||||
|
||||
#### Scenario: No routing or master match
|
||||
|
||||
- **WHEN** no work order routing or route master entries match
|
||||
- **THEN** results include input parameters only
|
||||
- **AND** MatchedSequenceNumber is NULL
|
||||
- **AND** RoutingMatch = 0 and MasterMatch = 0
|
||||
|
||||
## Search Status Codes
|
||||
|
||||
| Code | Name | Description |
|
||||
|------|------|-------------|
|
||||
| 1 | Queued | Search submitted, awaiting processing |
|
||||
| 2 | Processing | Search actively being executed |
|
||||
| 3 | Complete | Search finished successfully with results |
|
||||
| 4 | Failed | Search encountered an error |
|
||||
|
||||
## State Transition Diagram
|
||||
|
||||
```
|
||||
[New Search]
|
||||
|
|
||||
v
|
||||
SubmitSearch
|
||||
|
|
||||
v
|
||||
+-------+
|
||||
|Queued |<---------+
|
||||
| (1) | |
|
||||
+---+---+ |
|
||||
| |
|
||||
v |
|
||||
StartSearch ResetPartialSearches
|
||||
| |
|
||||
v |
|
||||
+----------+ |
|
||||
|Processing|-------+
|
||||
| (2) |
|
||||
+----+-----+
|
||||
|
|
||||
v
|
||||
CompleteSearch
|
||||
|
|
||||
+---------------+
|
||||
| |
|
||||
v v
|
||||
+--------+ +------+
|
||||
|Complete| |Failed|
|
||||
| (3) | | (4) |
|
||||
+--------+ +------+
|
||||
```
|
||||
|
||||
## Migration Notes
|
||||
|
||||
| Legacy Pattern | New Pattern | Rationale |
|
||||
|----------------|-------------|-----------|
|
||||
| Status as INT codes (1,2,3,4) | Consider enum type or constants | Improves readability and type safety |
|
||||
| SCOPE_IDENTITY() for new ID | EF Core auto-populates Id on SaveChanges | Framework handles identity retrieval |
|
||||
| Stored procedures for state transitions | Entity state management with domain events | Enables better testability and event sourcing |
|
||||
| VARBINARY(MAX) for Excel storage | Consider cloud blob storage reference | Reduces database size, enables streaming |
|
||||
| Table-valued function for MIS matching | LINQ query or stored procedure | Evaluate performance tradeoffs |
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- **MatchMIS migration**: Keep as SQL function - proven logic, better performance for large datasets
|
||||
- **Domain events**: Yes, emit events on status transitions - enables audit trail and future event sourcing
|
||||
- **Excel storage**: Keep in database VARBINARY - match legacy, simpler deployment
|
||||
- **Reset timeout**: No timeout - match legacy behavior, reset any search with StartDT but no EndDT
|
||||
|
||||
## Codex Review Findings
|
||||
|
||||
### Medium Priority
|
||||
1. **State transitions lack validation**: Legacy procedures update by ID without checking current Status. ResetPartialSearches uses StartDT/EndDT presence, not Status value. The spec's state diagram implies stricter transitions than implemented.
|
||||
2. **MatchMIS alias discovery incomplete**: RouteMaster is still merged when routing aliases exist (to set MasterMatch on existing sequences). Routing uses earliest transaction date for entire work order which may skip later matches.
|
||||
3. **MatchMIS truncation**: Alias sequence numbers are cast to INT before string compare, which is a lossy conversion not documented in the spec.
|
||||
|
||||
### Low Priority
|
||||
1. **CompleteSearch NULL behavior**: When @Success is NULL, SQL treats it as failure (Status=4) and still overwrites Results. This edge case is undocumented.
|
||||
2. **Output column order**: MatchMIS columns in spec differ from SQL (SamplingType/SamplingValue vs ToolsGauges/WorkInstructions position) - may matter for ordinal access.
|
||||
@@ -0,0 +1,404 @@
|
||||
# SQL Views and Types Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
This specification defines the SQL views that unify current and historical data across partitioned tables, the table-valued parameter (TVP) types used for efficient bulk filtering in search queries, and an aggregation view for work order scrap totals. These database objects provide the foundation for search query execution and data synchronization status tracking.
|
||||
|
||||
## Source Reference
|
||||
|
||||
| Legacy File | Purpose |
|
||||
|-------------|---------|
|
||||
| OLD/Database/Views/LastDataUpdates.sql | Aggregates latest successful sync timestamps per table/type |
|
||||
| OLD/Database/Views/WorkOrderTotalScrap.sql | Aggregates scrap quantities per work order |
|
||||
| OLD/Database/Views/WorkOrder.sql | Union of current and historical work orders |
|
||||
| OLD/Database/Views/WorkOrderTime.sql | Union of current and historical work order time entries |
|
||||
| OLD/Database/Views/WorkOrderStep.sql | Union of current and historical work order steps with function code join |
|
||||
| OLD/Database/Views/WorkOrderComponent.sql | Union of current and historical work order components |
|
||||
| OLD/Database/Views/LotUsage.sql | Union of current and historical lot usage records |
|
||||
| OLD/Database/Types/WorkOrderFilterParameter.sql | TVP for filtering by work order numbers |
|
||||
| OLD/Database/Types/ItemNumberFilterParameter.sql | TVP for filtering by item numbers |
|
||||
| OLD/Database/Types/ProfitCenterFilterParameter.sql | TVP for filtering by profit center codes |
|
||||
| OLD/Database/Types/WorkCenterFilterParameter.sql | TVP for filtering by work center codes |
|
||||
| OLD/Database/Types/OperatorFilterParameter.sql | TVP for filtering by operator user names |
|
||||
| OLD/Database/Types/ComponentLotFilterParameter.sql | TVP for filtering by component lot/item combinations |
|
||||
| OLD/Database/Types/ItemOperationMISFilterParameter.sql | TVP for filtering by item/operation/MIS combinations |
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: LastDataUpdates aggregation view
|
||||
|
||||
The system SHALL provide a LastDataUpdates view that aggregates the most recent successful data synchronization timestamps per table and update type.
|
||||
|
||||
#### Columns
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| TableName | VARCHAR(50) | Name of the synchronized table |
|
||||
| MassUpdateDT | DATETIME | Timestamp of last successful mass update (UpdateType=3), defaults to 1970-01-01 |
|
||||
| DailyUpdateDT | DATETIME | Timestamp of last successful daily update (UpdateType=2), falls back to MassUpdateDT |
|
||||
| HourlyUpdateDT | DATETIME | Timestamp of last successful hourly update (UpdateType=1), falls back to DailyUpdateDT then MassUpdateDT |
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
- Uses a CTE with ROW_NUMBER() partitioned by TableName and UpdateType, ordered by StartDT DESC
|
||||
- Filters to WasSuccessful = 1 records only
|
||||
- Uses PIVOT to transform UpdateType values (1=Hourly, 2=Daily, 3=Mass) into columns
|
||||
- Uses COALESCE cascading to provide fallback timestamps when specific update types have not occurred
|
||||
|
||||
#### Scenario: Query data freshness status
|
||||
|
||||
- **WHEN** an administrator queries the LastDataUpdates view
|
||||
- **THEN** results show the most recent successful sync timestamp for each table at each update frequency level
|
||||
|
||||
#### Scenario: Check if hourly sync is needed
|
||||
|
||||
- **WHEN** the data sync service checks HourlyUpdateDT for a table
|
||||
- **THEN** it receives the most recent of hourly/daily/mass update timestamps to determine staleness
|
||||
|
||||
### Requirement: WorkOrderTotalScrap aggregation view
|
||||
|
||||
The system SHALL provide a WorkOrderTotalScrap view that calculates the total scrapped quantity per work order from work order steps.
|
||||
|
||||
#### Columns
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| WorkOrderNumber | BIGINT | Work order identifier |
|
||||
| TotalScrappedQuantity | DECIMAL(18,2) | Sum of scrapped quantities, defaults to 0 |
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
- Aggregates from the WorkOrderStep view (which unions _Curr and _Hist tables)
|
||||
- Filters to whole-number step numbers only (StepNumber = ROUND(StepNumber, 0))
|
||||
- Groups by WorkOrderNumber
|
||||
- Uses COALESCE to return 0 when no scrap exists
|
||||
|
||||
#### Scenario: Calculate work order scrap total
|
||||
|
||||
- **WHEN** a query joins to WorkOrderTotalScrap for a work order
|
||||
- **THEN** it receives the sum of all scrapped quantities from that work order's steps
|
||||
|
||||
### Requirement: WorkOrder union view
|
||||
|
||||
The system SHALL provide a WorkOrder view that unions WorkOrder_Curr and WorkOrder_Hist tables to present all work orders as a single queryable source.
|
||||
|
||||
#### Columns
|
||||
|
||||
| Column | Type | Source |
|
||||
|--------|------|--------|
|
||||
| WorkOrderNumber | BIGINT | Primary key |
|
||||
| BranchCode | VARCHAR(12) | Branch/plant code |
|
||||
| LotNumber | VARCHAR(30) | Associated lot number |
|
||||
| ItemNumber | VARCHAR(25) | Item being produced |
|
||||
| ShortItemNumber | BIGINT | Numeric item identifier |
|
||||
| ParentWorkOrderNumber | VARCHAR(8) | Parent work order for split orders |
|
||||
| OrderQuantity | DECIMAL(15,2) | Ordered quantity |
|
||||
| HeldQuantity | DECIMAL(15,2) | Quantity on hold |
|
||||
| ShippedQuantity | DECIMAL(15,2) | Quantity shipped |
|
||||
| StatusCode | VARCHAR(10) | Work order status |
|
||||
| StatusCodeUpdateDT | DATETIME | Status last changed |
|
||||
| IssueDate | DATETIME | Issue date |
|
||||
| StartDate | DATETIME | Scheduled start date |
|
||||
| RoutingType | VARCHAR(3) | Routing type code |
|
||||
| LastUpdateDT | DATETIME | Record last updated |
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
- Uses UNION ALL to combine WorkOrder_Hist and WorkOrder_Curr
|
||||
- Selects all columns via wildcard (SELECT hist.*, SELECT curr.*)
|
||||
- No deduplication logic - assumes _Curr and _Hist are mutually exclusive
|
||||
|
||||
#### Scenario: Query all work orders
|
||||
|
||||
- **WHEN** a query selects from the WorkOrder view
|
||||
- **THEN** results include both current and historical work order records
|
||||
|
||||
#### Scenario: Search by work order number
|
||||
|
||||
- **WHEN** a search filter specifies work order numbers
|
||||
- **THEN** the query joins WorkOrder view to the filter TVP to match both current and historical records
|
||||
|
||||
### Requirement: WorkOrderTime union view
|
||||
|
||||
The system SHALL provide a WorkOrderTime view that unions WorkOrderTime_Curr and WorkOrderTime_Hist tables.
|
||||
|
||||
#### Columns
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| UniqueID | BIGINT | Primary key |
|
||||
| WorkOrderNumber | BIGINT | Associated work order |
|
||||
| StepNumber | DECIMAL(7,2) | Step sequence number |
|
||||
| WorkCenterCode | VARCHAR(12) | Work center code |
|
||||
| BranchCode | VARCHAR(12) | Branch/plant code |
|
||||
| AddressNumber | BIGINT | Operator address number (links to JdeUser) |
|
||||
| GlDate | DATETIME | General ledger date |
|
||||
| LastUpdateDT | DATETIME | Record last updated |
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
- Uses UNION ALL with explicit column list
|
||||
- AddressNumber column enables join to JdeUser table in search queries (join not in view itself)
|
||||
|
||||
#### Scenario: Query operator time entries
|
||||
|
||||
- **WHEN** a search filters by operator
|
||||
- **THEN** search queries join WorkOrderTime view to JdeUser via AddressNumber to match operator time entries
|
||||
|
||||
### Requirement: WorkOrderStep union view with function code join
|
||||
|
||||
The system SHALL provide a WorkOrderStep view that unions WorkOrderStep_Curr and WorkOrderStep_Hist tables and enriches with function code descriptions.
|
||||
|
||||
#### Columns
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| WorkOrderNumber | BIGINT | Associated work order |
|
||||
| WorkCenterCode | VARCHAR(12) | Work center code |
|
||||
| StepNumber | DECIMAL(7,2) | Step sequence number |
|
||||
| StepTypeCode | VARCHAR(2) | Step type code |
|
||||
| BranchCode | VARCHAR(12) | Branch/plant code |
|
||||
| StepDescription | VARCHAR(30) | Step description |
|
||||
| StartDT | DATETIME | Step start timestamp |
|
||||
| EndDT | DATETIME | Step end timestamp |
|
||||
| FunctionCode | VARCHAR(15) | Function/operation code |
|
||||
| FunctionOperationDescription | VARCHAR(MAX) | Description from FunctionCode table |
|
||||
| ScrappedQuantity | DECIMAL(18,2) | Quantity scrapped at this step |
|
||||
| LastUpdateDT | DATETIME | Record last updated |
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
- Uses UNION ALL of _Curr and _Hist in a subquery
|
||||
- LEFT OUTER JOINs to FunctionCode table on FunctionCode = Code
|
||||
- Adds FunctionOperationDescription from joined FunctionCode.Description
|
||||
|
||||
#### Scenario: Query work order steps with descriptions
|
||||
|
||||
- **WHEN** a query selects from WorkOrderStep view
|
||||
- **THEN** results include enriched function operation descriptions from the FunctionCode reference table
|
||||
|
||||
### Requirement: WorkOrderComponent union view
|
||||
|
||||
The system SHALL provide a WorkOrderComponent view that unions WorkOrderComponent_Curr and WorkOrderComponent_Hist tables.
|
||||
|
||||
#### Columns
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| UniqueID | BIGINT | Primary key |
|
||||
| WorkOrderNumber | BIGINT | Parent work order |
|
||||
| LotNumber | VARCHAR(30) | Component lot number |
|
||||
| BranchCode | VARCHAR(12) | Branch/plant code |
|
||||
| ShortItemNumber | BIGINT | Component item identifier |
|
||||
| Quantity | DECIMAL(15,2) | Quantity used |
|
||||
| LastUpdateDT | DATETIME | Record last updated |
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
- Uses UNION ALL with wildcard column selection
|
||||
- Used to trace component lot consumption in work orders
|
||||
|
||||
#### Scenario: Trace component lots
|
||||
|
||||
- **WHEN** a search filters by component lot numbers
|
||||
- **THEN** WorkOrderComponent view is queried to find work orders that consumed those lots
|
||||
|
||||
### Requirement: LotUsage union view
|
||||
|
||||
The system SHALL provide a LotUsage view that unions LotUsage_Curr and LotUsage_Hist tables.
|
||||
|
||||
#### Columns
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| UniqueID | BIGINT | Primary key |
|
||||
| WorkOrderNumber | BIGINT | Associated work order |
|
||||
| LotNumber | VARCHAR(30) | Lot number used |
|
||||
| BranchCode | VARCHAR(12) | Branch/plant code |
|
||||
| ShortItemNumber | BIGINT | Item identifier |
|
||||
| Quantity | DECIMAL(15,2) | Quantity used |
|
||||
| LastUpdateDT | DATETIME | Record last updated |
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
- Uses UNION ALL with wildcard column selection
|
||||
- Used alongside WorkOrderComponent for complete lot traceability
|
||||
|
||||
#### Scenario: Trace lot usage downstream
|
||||
|
||||
- **WHEN** a search filters by component lot numbers
|
||||
- **THEN** LotUsage view is queried alongside WorkOrderComponent to find downstream work orders
|
||||
|
||||
### Requirement: WorkOrderFilterParameter TVP type
|
||||
|
||||
The system SHALL provide a table-valued parameter type for bulk filtering by work order numbers.
|
||||
|
||||
#### Schema
|
||||
|
||||
| Column | Type | Nullable |
|
||||
|--------|------|----------|
|
||||
| WorkOrderNumber | BIGINT | NULL |
|
||||
|
||||
#### Usage Pattern
|
||||
|
||||
The application creates a DataTable with WorkOrderNumber column, populates it from search criteria, and passes it as a TVP to search queries:
|
||||
|
||||
```csharp
|
||||
DataTable dataTable = new DataTable();
|
||||
dataTable.Columns.Add("WorkOrderNumber", typeof(long));
|
||||
foreach (var entry in model.WorkOrderFilter)
|
||||
{
|
||||
dataTable.Rows.Add(entry.WorkOrderNumber);
|
||||
}
|
||||
return dataTable.AsTableValuedParameter("WorkOrderFilterParameter");
|
||||
```
|
||||
|
||||
#### Scenario: Filter by specific work orders
|
||||
|
||||
- **WHEN** a user specifies work order numbers in search criteria
|
||||
- **THEN** the system passes a WorkOrderFilterParameter TVP to the query for efficient bulk matching
|
||||
|
||||
### Requirement: ItemNumberFilterParameter TVP type
|
||||
|
||||
The system SHALL provide a table-valued parameter type for bulk filtering by item numbers.
|
||||
|
||||
#### Schema
|
||||
|
||||
| Column | Type | Nullable |
|
||||
|--------|------|----------|
|
||||
| ItemNumber | VARCHAR(25) | NULL |
|
||||
|
||||
#### Usage Pattern
|
||||
|
||||
Populated from search criteria item number list and joined to WorkOrder.ItemNumber in search queries.
|
||||
|
||||
#### Scenario: Filter by item numbers
|
||||
|
||||
- **WHEN** a user specifies item numbers in search criteria
|
||||
- **THEN** the system passes an ItemNumberFilterParameter TVP for efficient bulk item matching
|
||||
|
||||
### Requirement: ProfitCenterFilterParameter TVP type
|
||||
|
||||
The system SHALL provide a table-valued parameter type for bulk filtering by profit center codes.
|
||||
|
||||
#### Schema
|
||||
|
||||
| Column | Type | Nullable |
|
||||
|--------|------|----------|
|
||||
| Code | VARCHAR(12) | NULL |
|
||||
|
||||
#### Usage Pattern
|
||||
|
||||
Profit center codes are expanded to work center codes via OrgHierarchy table join, then used to filter WorkOrderStep/WorkOrderTime records.
|
||||
|
||||
#### Scenario: Filter by profit centers
|
||||
|
||||
- **WHEN** a user specifies profit centers in search criteria
|
||||
- **THEN** the system passes a ProfitCenterFilterParameter TVP and expands to work centers via OrgHierarchy
|
||||
|
||||
### Requirement: WorkCenterFilterParameter TVP type
|
||||
|
||||
The system SHALL provide a table-valued parameter type for bulk filtering by work center codes.
|
||||
|
||||
#### Schema
|
||||
|
||||
| Column | Type | Nullable |
|
||||
|--------|------|----------|
|
||||
| Code | VARCHAR(12) | NULL |
|
||||
|
||||
#### Usage Pattern
|
||||
|
||||
Joined directly to WorkOrderStep.WorkCenterCode or WorkOrderTime.WorkCenterCode to filter by manufacturing work centers.
|
||||
|
||||
#### Scenario: Filter by work centers
|
||||
|
||||
- **WHEN** a user specifies work centers in search criteria
|
||||
- **THEN** the system passes a WorkCenterFilterParameter TVP for direct work center matching
|
||||
|
||||
### Requirement: OperatorFilterParameter TVP type
|
||||
|
||||
The system SHALL provide a table-valued parameter type for bulk filtering by operator user names.
|
||||
|
||||
#### Schema
|
||||
|
||||
| Column | Type | Nullable |
|
||||
|--------|------|----------|
|
||||
| UserName | VARCHAR(10) | NULL |
|
||||
|
||||
#### Usage Pattern
|
||||
|
||||
User names are resolved to AddressNumber via JdeUser table join, then matched to WorkOrderTime.AddressNumber to find work orders touched by specific operators.
|
||||
|
||||
#### Scenario: Filter by operators
|
||||
|
||||
- **WHEN** a user specifies operator IDs in search criteria
|
||||
- **THEN** the system passes an OperatorFilterParameter TVP and resolves to address numbers via JdeUser
|
||||
|
||||
### Requirement: ComponentLotFilterParameter TVP type
|
||||
|
||||
The system SHALL provide a table-valued parameter type for bulk filtering by component lot and item number combinations.
|
||||
|
||||
#### Schema
|
||||
|
||||
| Column | Type | Nullable |
|
||||
|--------|------|----------|
|
||||
| ComponentLotNumber | VARCHAR(30) | NULL |
|
||||
| ItemNumber | VARCHAR(128) | NULL |
|
||||
|
||||
#### Usage Pattern
|
||||
|
||||
Used to find downstream work orders that consumed specific component lots. The lot/item combination is matched against Lot table, then traced through WorkOrderComponent and LotUsage views.
|
||||
|
||||
#### Scenario: Trace component lots to downstream products
|
||||
|
||||
- **WHEN** a user specifies component lot numbers in search criteria
|
||||
- **THEN** the system passes a ComponentLotFilterParameter TVP to trace lot usage through manufacturing
|
||||
|
||||
### Requirement: ItemOperationMisFilterParameter TVP type
|
||||
|
||||
The system SHALL provide a table-valued parameter type for bulk filtering by item/operation/MIS number/revision combinations.
|
||||
|
||||
#### Schema
|
||||
|
||||
| Column | Type | Nullable |
|
||||
|--------|------|----------|
|
||||
| ItemNumber | VARCHAR(32) | NULL |
|
||||
| OperationNumber | VARCHAR(32) | NULL |
|
||||
| MisNumber | VARCHAR(32) | NULL |
|
||||
| MisRevision | VARCHAR(32) | NULL |
|
||||
|
||||
#### Usage Pattern
|
||||
|
||||
Used for MIS (Manufacturing Information System) data extraction. The four-part key identifies specific MIS records to match against work order routing and MIS data.
|
||||
|
||||
#### Scenario: Filter by MIS specifications
|
||||
|
||||
- **WHEN** a user specifies item/operation/MIS criteria
|
||||
- **THEN** the system passes an ItemOperationMISFilterParameter TVP for MIS data matching and extraction
|
||||
|
||||
## Migration Notes
|
||||
|
||||
| Legacy Pattern | New Pattern | Rationale |
|
||||
|----------------|-------------|-----------|
|
||||
| SELECT * in union views | Explicit column lists | Ensures schema changes are caught at compile time and improves maintainability |
|
||||
| VARCHAR types in TVPs | Consider NVARCHAR for some columns | Evaluate if Unicode support is needed for international data |
|
||||
| Multiple TVP types with single-column schemas | Consider consolidation | ProfitCenterFilterParameter and WorkCenterFilterParameter have identical schemas |
|
||||
| Dapper AsTableValuedParameter | Continue using Dapper or switch to EF Core | TVP support in .NET 10 works well with both Dapper and raw ADO.NET |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should the union views use UNION (with deduplication) instead of UNION ALL? The current implementation assumes _Curr and _Hist tables are mutually exclusive, but this should be verified during data sync analysis.
|
||||
|
||||
2. The ComponentLotFilterParameter uses VARCHAR(128) for ItemNumber while other TVPs and tables use VARCHAR(25). Should this be standardized?
|
||||
|
||||
3. Should LastDataUpdates view be replaced with a more flexible query or stored procedure that can report on any update type combination?
|
||||
|
||||
## Codex Review Findings
|
||||
|
||||
### Issues Fixed
|
||||
1. ~~**TVP nullability incorrect**~~: Fixed - all TVP columns now show NULL
|
||||
2. ~~**WorkOrderTime join logic misstated**~~: Fixed - clarified join happens in search queries, not in view
|
||||
3. ~~**TVP name case mismatch**~~: Fixed - renamed to `ItemOperationMisFilterParameter`
|
||||
|
||||
### Remaining Notes
|
||||
- **WorkOrderTotalScrap datatype**: Spec lists `DECIMAL(18,2)` but SUM without casting returns higher precision (typically `DECIMAL(28,2)`) - acceptable for documentation purposes
|
||||
@@ -0,0 +1,656 @@
|
||||
# Web API and Authentication Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
This specification defines the REST API endpoints, SignalR real-time communication hub, and LDAP authentication system for the JDE Scoping Tool. The system provides a single .NET 10 service that exposes REST APIs for Blazor WebAssembly clients to manage searches, perform lookup operations, upload/download files, and receive real-time status updates via SignalR. Authentication is performed against an LDAP directory server with group membership verification, with support for a development-mode bypass.
|
||||
|
||||
## Source Reference
|
||||
|
||||
| Legacy Files | NEW/ Target | Purpose |
|
||||
|--------------|-------------|---------|
|
||||
| OLD/WebInterface/Controllers/SearchController.cs | NEW/ScopingTool.Api/Controllers/SearchController.cs | Search management API - create, view, copy, save searches and download results |
|
||||
| OLD/WebInterface/Controllers/AccountController.cs | NEW/ScopingTool.Api/Controllers/AuthController.cs | User authentication - login, logout, authorization |
|
||||
| OLD/WebInterface/Controllers/LookupController.cs | NEW/ScopingTool.Api/Controllers/LookupController.cs | Autocomplete lookup APIs for items, profit centers, work centers, operators |
|
||||
| OLD/WebInterface/Controllers/FileIOController.cs | NEW/ScopingTool.Api/Controllers/FileController.cs | Excel file upload/download for bulk data import |
|
||||
| OLD/WebInterface/Controllers/CrudController.cs | NEW/ScopingTool.Api/Controllers/ApiControllerBase.cs | Base controller with current user context |
|
||||
| OLD/WebInterface/Hubs/StatusHub.cs | NEW/ScopingTool.Api/Hubs/StatusHub.cs | SignalR hub for real-time status and search updates |
|
||||
| OLD/WebInterface/Helpers/LDAPHelper.cs | NEW/ScopingTool.Api/Services/LdapAuthService.cs | LDAP server authentication and user lookup |
|
||||
| OLD/WebInterface/Security/UserIdentity.cs | NEW/ScopingTool.Api/Security/UserIdentity.cs | Claims-based user identity from LDAP |
|
||||
| OLD/WebInterface/Models/LogonRequest.cs | NEW/ScopingTool.Api/Models/LoginRequest.cs | Login request model |
|
||||
| OLD/DataModel/Models/LDAPEntry.cs | NEW/ScopingTool.Domain/Models/UserInfo.cs | User information model (renamed from LDAPEntry) |
|
||||
| OLD/DataModel/Models/StatusUpdate.cs | NEW/ScopingTool.Domain/Models/StatusUpdate.cs | Process status update model |
|
||||
| OLD/DataModel/Models/SearchUpdate.cs | NEW/ScopingTool.Domain/Models/SearchUpdate.cs | Search status update model |
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Authentication Service Interface
|
||||
|
||||
The system SHALL provide an abstraction for authentication to support LDAP authentication in production and fake authentication in development mode.
|
||||
|
||||
#### Interface Definition
|
||||
|
||||
```csharp
|
||||
public interface IAuthService
|
||||
{
|
||||
Task<AuthResult> AuthenticateAsync(string username, string password, CancellationToken ct = default);
|
||||
Task<UserInfo?> GetUserInfoAsync(string username, CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public record AuthResult(bool Success, UserInfo? User, string? ErrorMessage);
|
||||
```
|
||||
|
||||
#### Implementations
|
||||
|
||||
- `LdapAuthService` - Production LDAP authentication using `System.DirectoryServices.Protocols`
|
||||
- `FakeAuthService` - Development mode bypass that accepts any credentials
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL register `IAuthService` in the DI container based on configuration
|
||||
- The system SHALL use `LdapAuthService` when `AuthOptions.UseFakeAuth` is false
|
||||
- The system SHALL use `FakeAuthService` when `AuthOptions.UseFakeAuth` is true (development only)
|
||||
- `FakeAuthService` SHALL return a predefined `UserInfo` for any username/password combination
|
||||
|
||||
#### Scenario: Production mode uses LDAP authentication
|
||||
|
||||
- **WHEN** the application starts with `AuthOptions.UseFakeAuth = false`
|
||||
- **THEN** `LdapAuthService` is registered as `IAuthService` and all authentication flows use LDAP
|
||||
|
||||
#### Scenario: Development mode uses fake authentication
|
||||
|
||||
- **WHEN** the application starts with `AuthOptions.UseFakeAuth = true`
|
||||
- **THEN** `FakeAuthService` is registered as `IAuthService` and any credentials are accepted
|
||||
|
||||
### Requirement: Configuration Options
|
||||
|
||||
The system SHALL use strongly-typed configuration options for LDAP and authentication settings.
|
||||
|
||||
#### LdapOptions
|
||||
|
||||
```csharp
|
||||
public class LdapOptions
|
||||
{
|
||||
public const string SectionName = "Ldap";
|
||||
|
||||
public string[] ServerUrls { get; set; } = Array.Empty<string>();
|
||||
public string GroupDn { get; set; } = string.Empty;
|
||||
public string SearchBase { get; set; } = string.Empty;
|
||||
public int ConnectionTimeoutSeconds { get; set; } = 30;
|
||||
}
|
||||
```
|
||||
|
||||
#### AuthOptions
|
||||
|
||||
```csharp
|
||||
public class AuthOptions
|
||||
{
|
||||
public const string SectionName = "Auth";
|
||||
|
||||
public bool UseFakeAuth { get; set; } = false;
|
||||
public string CookieName { get; set; } = "ScopingTool.Auth";
|
||||
public int CookieExpirationMinutes { get; set; } = 480; // 8 hours
|
||||
public string[] AdminBypassUsers { get; set; } = Array.Empty<string>(); // Usernames that bypass group check
|
||||
public long MaxUploadSizeBytes { get; set; } = 10 * 1024 * 1024; // 10MB default
|
||||
}
|
||||
```
|
||||
|
||||
#### LDAP Connection Management
|
||||
|
||||
The system SHALL use a connection-per-request pattern for LDAP connections. Each authentication request creates a new `LdapConnection`, authenticates, and disposes. Connection pooling is NOT used due to LDAP bind credential requirements.
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL bind `LdapOptions` from the `Ldap` configuration section
|
||||
- The system SHALL bind `AuthOptions` from the `Auth` configuration section
|
||||
- The system SHALL use `IOptions<LdapOptions>` and `IOptions<AuthOptions>` for injection
|
||||
|
||||
### Requirement: LDAP Authentication
|
||||
|
||||
The system SHALL authenticate users against an LDAP directory server and verify group membership before granting access.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Username (sAMAccountName format)
|
||||
- Password (plain text, transmitted over HTTPS)
|
||||
- LDAP server URLs (from `IOptions<LdapOptions>`, supports multiple URLs for failover)
|
||||
- LDAP group distinguished name (from `IOptions<LdapOptions>`)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `AuthResult` containing:
|
||||
- Success/failure indicator
|
||||
- `UserInfo` on success (see Data Models section)
|
||||
- Error message on failure
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use `System.DirectoryServices.Protocols.LdapConnection` for cross-platform compatibility
|
||||
- The system SHALL attempt authentication against each configured LDAP server URL sequentially until one succeeds
|
||||
- The system SHALL verify the user is a member of the configured LDAP group after successful bind
|
||||
- The system SHALL use the sAMAccountName LDAP filter format `(sAMAccountName={0})` for user lookup
|
||||
- The system SHALL extract user properties: distinguishedName, givenName, sn, mail, title
|
||||
- The system SHALL compute DisplayName as `"{FirstName} {LastName}".Trim()` or fall back to Username if both are empty
|
||||
- The system SHALL sign out any existing session before creating a new one
|
||||
- The system SHALL use non-persistent (session-only) authentication cookies
|
||||
|
||||
#### Scenario: Successful LDAP authentication with group membership
|
||||
|
||||
- **WHEN** a user submits valid credentials for an LDAP user who is a member of the required group
|
||||
- **THEN** the system authenticates against the LDAP server, verifies group membership, creates a claims-based identity, signs in the user, and returns a success response with user info
|
||||
|
||||
#### Scenario: Failed LDAP authentication with invalid credentials
|
||||
|
||||
- **WHEN** a user submits invalid credentials
|
||||
- **THEN** the system returns `AuthResult { Success = false, ErrorMessage = "Incorrect username or password" }`
|
||||
|
||||
#### Scenario: Valid credentials but user not in required group
|
||||
|
||||
- **WHEN** a user submits valid credentials but is not a member of the required LDAP group
|
||||
- **THEN** the system returns `AuthResult { Success = false, ErrorMessage = "User is not a member of the required security group" }`
|
||||
|
||||
#### Scenario: LDAP server unavailable with failover
|
||||
|
||||
- **WHEN** the primary LDAP server is unavailable but a secondary server is configured
|
||||
- **THEN** the system attempts authentication against each configured server in sequence until one succeeds or all fail
|
||||
|
||||
#### Scenario: All LDAP servers unavailable
|
||||
|
||||
- **WHEN** all configured LDAP servers are unavailable
|
||||
- **THEN** the system returns `AuthResult { Success = false, ErrorMessage = "Unable to connect to directory server" }`
|
||||
|
||||
### Requirement: User Session Management
|
||||
|
||||
The system SHALL maintain user session state using ASP.NET Core cookie authentication.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Authenticated user identity (from LDAP or fake auth)
|
||||
- Session cookie
|
||||
|
||||
#### Outputs
|
||||
|
||||
- User context available to all authorized controllers via `HttpContext.User`
|
||||
- Session termination on logout
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use `HttpContext.SignInAsync()` with `CookieAuthenticationDefaults.AuthenticationScheme` for sign-in
|
||||
- The system SHALL use `HttpContext.SignOutAsync()` for sign-out
|
||||
- The system SHALL store user claims in a `ClaimsPrincipal` with cookie authentication scheme
|
||||
- The system SHALL provide access to current user's `UserInfo` through `HttpContext.User` claims
|
||||
- The system SHALL clear authentication cookies on logout
|
||||
- The system SHALL return HTTP 401 Unauthorized for unauthenticated API requests (no redirect for Blazor WASM)
|
||||
|
||||
#### Scenario: Access protected resource while authenticated
|
||||
|
||||
- **WHEN** an authenticated user requests a protected resource
|
||||
- **THEN** the system provides the current user context and serves the requested resource
|
||||
|
||||
#### Scenario: Access protected resource while unauthenticated
|
||||
|
||||
- **WHEN** an unauthenticated user requests a protected resource with `[Authorize]` attribute
|
||||
- **THEN** the system returns HTTP 401 Unauthorized
|
||||
|
||||
#### Scenario: User logs out
|
||||
|
||||
- **WHEN** an authenticated user requests logout via `POST /api/auth/logout`
|
||||
- **THEN** the system calls `HttpContext.SignOutAsync()`, clears all authentication cookies, and returns HTTP 200 OK
|
||||
|
||||
#### Scenario: Parse user identity from claims
|
||||
|
||||
- **WHEN** a controller accesses `HttpContext.User`
|
||||
- **THEN** the system provides access to user claims including Username, FirstName, LastName, Email, and Title
|
||||
|
||||
### Requirement: Auth API Endpoints
|
||||
|
||||
The system SHALL provide REST API endpoints for authentication operations.
|
||||
|
||||
#### Inputs
|
||||
|
||||
| Endpoint | Method | Parameters | Description |
|
||||
|----------|--------|------------|-------------|
|
||||
| `/api/auth/login` | POST | `LoginRequest` body | Authenticate user |
|
||||
| `/api/auth/logout` | POST | - | Sign out current user |
|
||||
| `/api/auth/me` | GET | - | Get current user info |
|
||||
|
||||
#### Outputs
|
||||
|
||||
- JSON responses with `UserInfo` on success
|
||||
- HTTP 401 Unauthorized on authentication failure
|
||||
- HTTP 200 OK on successful logout
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use `[ApiController]` attribute on the controller
|
||||
- The system SHALL use attribute routing with `/api/auth` prefix
|
||||
- The system SHALL return `UserInfo` JSON (not redirect) on successful login for Blazor WASM compatibility
|
||||
- The system SHALL call `IAuthService.AuthenticateAsync()` for login operations
|
||||
|
||||
#### Scenario: Successful login
|
||||
|
||||
- **WHEN** a user posts valid credentials to `/api/auth/login`
|
||||
- **THEN** the system authenticates, creates a session, and returns HTTP 200 with `UserInfo` JSON
|
||||
|
||||
#### Scenario: Failed login
|
||||
|
||||
- **WHEN** a user posts invalid credentials to `/api/auth/login`
|
||||
- **THEN** the system returns HTTP 401 Unauthorized with error message
|
||||
|
||||
#### Scenario: Get current user
|
||||
|
||||
- **WHEN** an authenticated user requests `/api/auth/me`
|
||||
- **THEN** the system returns HTTP 200 with `UserInfo` JSON for the current user
|
||||
|
||||
#### Scenario: Get current user when not authenticated
|
||||
|
||||
- **WHEN** an unauthenticated user requests `/api/auth/me`
|
||||
- **THEN** the system returns HTTP 401 Unauthorized
|
||||
|
||||
### Requirement: Search API Endpoints
|
||||
|
||||
The system SHALL provide REST API endpoints for search management operations.
|
||||
|
||||
#### Inputs
|
||||
|
||||
| Endpoint | Method | Parameters | Description |
|
||||
|----------|--------|------------|-------------|
|
||||
| `/api/search` | GET | - | Get current user's searches |
|
||||
| `/api/search/queue` | GET | - | Get all queued searches |
|
||||
| `/api/search/{id}` | GET | id (int) | Get search by ID |
|
||||
| `/api/search/{id}/copy` | POST | id (int) | Copy search with reset status |
|
||||
| `/api/search` | POST | `SearchViewModel` body | Create/submit search |
|
||||
| `/api/search/{id}/results` | GET | id (int) | Download search results Excel file |
|
||||
|
||||
#### Outputs
|
||||
|
||||
- JSON responses using System.Text.Json serialization
|
||||
- File download for results (application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
|
||||
- Search ID on successful create
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use `[ApiController]` attribute on the controller
|
||||
- The system SHALL use attribute routing with `/api/search` prefix
|
||||
- The system SHALL require authorization for all search endpoints via `[Authorize]` attribute
|
||||
- The system SHALL filter GetSearches to only return searches owned by the current user
|
||||
- The system SHALL order user searches by StartDT descending (most recent first)
|
||||
- The system SHALL reset Status, UserName, SubmitDT, StartDT, and EndDT when copying a search
|
||||
- The system SHALL publish a SearchUpdate to the SignalR hub when a new search is saved
|
||||
- The system SHALL return the new search ID on successful create
|
||||
- The system SHALL set filename to "search_results.xlsx" for result downloads
|
||||
- The system SHALL inject `IHubContext<StatusHub>` via DI for SignalR notifications
|
||||
|
||||
#### Scenario: Get user's search history
|
||||
|
||||
- **WHEN** an authenticated user requests `GET /api/search`
|
||||
- **THEN** the system returns a JSON array of `SearchViewModel` objects for searches owned by that user, ordered by most recent first
|
||||
|
||||
#### Scenario: Create new search
|
||||
|
||||
- **WHEN** an authenticated user posts a `SearchViewModel` to `POST /api/search`
|
||||
- **THEN** the system converts the view model to a Search entity, submits it to the database, publishes a SignalR notification via `IHubContext<StatusHub>`, and returns HTTP 201 Created with the new search ID
|
||||
|
||||
#### Scenario: Copy existing search
|
||||
|
||||
- **WHEN** an authenticated user requests `POST /api/search/{id}/copy`
|
||||
- **THEN** the system loads the original search, resets the status to New, clears timestamps, sets the current user as owner, saves the copy, and returns HTTP 201 Created with the new search ID
|
||||
|
||||
#### Scenario: Download search results
|
||||
|
||||
- **WHEN** an authenticated user requests `GET /api/search/{id}/results`
|
||||
- **THEN** the system retrieves the binary Excel data from the database and returns it as a file download with Content-Type `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
|
||||
|
||||
#### Scenario: Download results for search without results
|
||||
|
||||
- **WHEN** an authenticated user requests results for a search that has not completed
|
||||
- **THEN** the system returns HTTP 404 Not Found
|
||||
|
||||
### Requirement: Lookup API Endpoints
|
||||
|
||||
The system SHALL provide REST API endpoints for autocomplete/lookup operations.
|
||||
|
||||
#### Inputs
|
||||
|
||||
| Endpoint | Method | Parameters | Description |
|
||||
|----------|--------|------------|-------------|
|
||||
| `/api/lookup/items` | GET | `q` (string) | Search items by number |
|
||||
| `/api/lookup/profit-centers` | GET | `q` (string) | Search profit centers |
|
||||
| `/api/lookup/work-centers` | GET | `q` (string) | Search work centers |
|
||||
| `/api/lookup/operators` | GET | `q` (string) | Search operators/users |
|
||||
|
||||
#### Outputs
|
||||
|
||||
- JSON arrays of matching entities converted to view models
|
||||
- Results ordered alphabetically by primary identifier
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use `[ApiController]` attribute on the controller
|
||||
- The system SHALL use attribute routing with `/api/lookup` prefix
|
||||
- The system SHALL NOT require authorization for lookup endpoints (public access)
|
||||
- The system SHALL order item results by ItemNumber
|
||||
- The system SHALL order profit center results by Code
|
||||
- The system SHALL order work center results by Code
|
||||
- The system SHALL order operator results by FullName
|
||||
|
||||
#### Scenario: Search for items by partial number
|
||||
|
||||
- **WHEN** a user requests `GET /api/lookup/items?q=ABC`
|
||||
- **THEN** the system searches for items matching the partial number and returns matching `ItemViewModel` objects ordered by ItemNumber
|
||||
|
||||
#### Scenario: Search for profit centers
|
||||
|
||||
- **WHEN** a user requests `GET /api/lookup/profit-centers?q=100`
|
||||
- **THEN** the system searches for profit centers matching the partial code and returns matching view models ordered by Code
|
||||
|
||||
#### Scenario: Search for work centers
|
||||
|
||||
- **WHEN** a user requests `GET /api/lookup/work-centers?q=MACH`
|
||||
- **THEN** the system searches for work centers matching the partial code and returns matching view models ordered by Code
|
||||
|
||||
#### Scenario: Search for operators by name
|
||||
|
||||
- **WHEN** a user requests `GET /api/lookup/operators?q=Smith`
|
||||
- **THEN** the system searches for users matching the partial name and returns matching view models ordered by FullName
|
||||
|
||||
### Requirement: File API Endpoints
|
||||
|
||||
The system SHALL provide REST API endpoints for Excel file upload and download operations using ClosedXML.
|
||||
|
||||
#### Inputs
|
||||
|
||||
| Endpoint | Method | Parameters | Description |
|
||||
|----------|--------|------------|-------------|
|
||||
| `/api/file/work-orders/upload` | POST | `IFormFile file` | Upload work orders from Excel |
|
||||
| `/api/file/work-orders/template` | POST | `List<long>` body | Generate work order template, returns cache key |
|
||||
| `/api/file/work-orders/template/{key}` | GET | key (Guid) | Download cached template |
|
||||
| `/api/file/part-numbers/upload` | POST | `IFormFile file` | Upload part numbers from Excel |
|
||||
| `/api/file/part-numbers/template` | POST | `List<ItemViewModel>` body | Generate part number template |
|
||||
| `/api/file/part-numbers/template/{key}` | GET | key (Guid) | Download cached template |
|
||||
| `/api/file/component-lots/upload` | POST | `IFormFile file` | Upload component lots from Excel |
|
||||
| `/api/file/component-lots/template` | POST | `List<LotViewModel>` body | Generate component lot template |
|
||||
| `/api/file/component-lots/template/{key}` | GET | key (Guid) | Download cached template |
|
||||
| `/api/file/part-operations/upload` | POST | `IFormFile file` | Upload part operations from Excel |
|
||||
| `/api/file/part-operations/template` | POST | `List<PartOperationViewModel>` body | Generate part operations template |
|
||||
| `/api/file/part-operations/template/{key}` | GET | key (Guid) | Download cached template |
|
||||
|
||||
#### Outputs
|
||||
|
||||
- `FileUploadResult<T>` with WasSuccessful, ErrorMessage, and Data properties
|
||||
- GUID key for cached file downloads
|
||||
- Excel file stream for GET download requests
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use `[ApiController]` attribute on the controller
|
||||
- The system SHALL use attribute routing with `/api/file` prefix
|
||||
- The system SHALL NOT require authorization for file endpoints (matches legacy behavior)
|
||||
- The system SHALL accept file uploads via `IFormFile` parameter
|
||||
- The system SHALL parse Excel files using ClosedXML library, reading from row 2 (skip header)
|
||||
- The system SHALL inject `IMemoryCache` via DI for template caching
|
||||
- The system SHALL cache generated templates with 1-minute absolute expiration
|
||||
- The system SHALL use GUID keys for cached file retrieval
|
||||
- The system SHALL return HTTP 404 if cached file not found or expired
|
||||
- The system SHALL remove cached data after successful download
|
||||
- The system SHALL deduplicate uploaded data using DistinctBy before returning (except part operations)
|
||||
|
||||
#### Scenario: Upload work orders from Excel
|
||||
|
||||
- **WHEN** a user uploads an Excel file to `POST /api/file/work-orders/upload`
|
||||
- **THEN** the system parses work order numbers from column 1 using ClosedXML, looks up matching work orders in the database, deduplicates results, and returns a `FileUploadResult` with matching `WorkOrderViewModel` objects
|
||||
|
||||
#### Scenario: Generate and download work order template
|
||||
|
||||
- **WHEN** a user posts work order numbers to `POST /api/file/work-orders/template`
|
||||
- **THEN** the system generates an Excel template using ClosedXML, caches it with a GUID key, and returns the key
|
||||
- **WHEN** the user requests `GET /api/file/work-orders/template/{guid}`
|
||||
- **THEN** the system retrieves the cached file and returns it as "work_order_template.xlsx"
|
||||
|
||||
#### Scenario: Upload component lots with item numbers
|
||||
|
||||
- **WHEN** a user uploads an Excel file with lot numbers and item numbers to `POST /api/file/component-lots/upload`
|
||||
- **THEN** the system parses lot numbers from column 1 and item numbers from column 2, looks up matching lots, deduplicates, and returns matching `LotViewModel` objects
|
||||
|
||||
#### Scenario: Cached file expired
|
||||
|
||||
- **WHEN** a user requests a download with an expired or invalid cache key
|
||||
- **THEN** the system returns HTTP 404 Not Found
|
||||
|
||||
#### Scenario: No file uploaded
|
||||
|
||||
- **WHEN** a user posts to an upload endpoint without a file
|
||||
- **THEN** the system returns `FileUploadResult { WasSuccessful = false, ErrorMessage = "No file uploaded" }`
|
||||
|
||||
### Requirement: SignalR Real-Time Updates
|
||||
|
||||
The system SHALL provide real-time status updates to connected clients via ASP.NET Core SignalR hub.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Status updates from worker service (via `IHubContext<StatusHub>`)
|
||||
- Search status changes (via `IHubContext<StatusHub>`)
|
||||
- Client requests for cached status (GetCachedStatus hub method)
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Broadcasts to all connected clients via `SendAsync("statusUpdate", ...)` and `SendAsync("searchUpdate", ...)`
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use ASP.NET Core SignalR (`Microsoft.AspNetCore.SignalR`)
|
||||
- The system SHALL maintain a cached `StatusUpdate` with message and timestamp
|
||||
- The system SHALL broadcast status updates to ALL connected clients using `Clients.All.SendAsync()`
|
||||
- The system SHALL inject `IHubContext<StatusHub>` into controllers and services that need to publish updates
|
||||
- The system SHALL NOT use static `GlobalHost` pattern - use DI exclusively
|
||||
- The system SHALL initialize cached status with "Unknown" message and current timestamp
|
||||
- The system SHALL map the hub endpoint to `/hubs/status`
|
||||
|
||||
#### Scenario: Worker service publishes status update
|
||||
|
||||
- **WHEN** the worker service publishes a `StatusUpdate` via `IHubContext<StatusHub>`
|
||||
- **THEN** the system caches the update and broadcasts it to all connected clients via `Clients.All.SendAsync("statusUpdate", statusUpdate)`
|
||||
|
||||
#### Scenario: New search submitted via controller
|
||||
|
||||
- **WHEN** the SearchController saves a new search
|
||||
- **THEN** the controller uses injected `IHubContext<StatusHub>` to call `Clients.All.SendAsync("searchUpdate", searchUpdate)`
|
||||
|
||||
#### Scenario: Client requests current status
|
||||
|
||||
- **WHEN** a newly connected client invokes the `GetCachedStatus` hub method
|
||||
- **THEN** the system returns the most recent cached `StatusUpdate` (or the default "Unknown" status if none set)
|
||||
|
||||
#### Scenario: Multiple clients receive broadcast
|
||||
|
||||
- **WHEN** a status update is broadcast
|
||||
- **THEN** all connected SignalR clients receive the update simultaneously via their `statusUpdate` or `searchUpdate` event handlers
|
||||
|
||||
### Requirement: Blazor Client Integration
|
||||
|
||||
The system SHALL support Blazor WebAssembly client authentication and real-time updates.
|
||||
|
||||
#### Authentication Flow
|
||||
|
||||
1. Blazor client calls `POST /api/auth/login` with credentials
|
||||
2. Server validates credentials, creates cookie-based session
|
||||
3. Server returns `UserInfo` JSON
|
||||
4. Client stores user info in memory and sets authenticated state
|
||||
5. Subsequent API calls include auth cookie automatically
|
||||
|
||||
#### SignalR Connection
|
||||
|
||||
1. Blazor client creates `HubConnection` to `/hubs/status`
|
||||
2. Client registers handlers for `statusUpdate` and `searchUpdate` events
|
||||
3. Client calls `GetCachedStatus()` on connection to get initial state
|
||||
4. Client receives real-time updates via registered handlers
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL use cookie authentication (not JWT) for browser-based Blazor WASM
|
||||
- The system SHALL configure CORS for same-origin only by default; cross-origin support is NOT required for single-domain deployment
|
||||
- The system SHALL return JSON responses (not redirects) for all API errors
|
||||
- The system SHALL support reconnection for SignalR clients with automatic retry
|
||||
- The SignalR hub SHALL remain open without requiring per-request authentication (cookies validated on connection)
|
||||
- The Blazor client SHALL handle SPA navigation behaviors (return URL handling, 401 interception for redirect to login)
|
||||
|
||||
### Requirement: Authorization Patterns
|
||||
|
||||
The system SHALL enforce authorization using ASP.NET Core attribute-based access control.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Authorization attributes on controllers and actions
|
||||
- User authentication status from cookie
|
||||
|
||||
#### Outputs
|
||||
|
||||
- Access granted or denied
|
||||
- HTTP 401 Unauthorized for unauthenticated API requests
|
||||
- HTTP 403 Forbidden for authenticated but unauthorized requests
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- The system SHALL apply `[Authorize]` at controller level for `SearchController`
|
||||
- The system SHALL apply `[AllowAnonymous]` for `AuthController.Login` action
|
||||
- The system SHALL NOT require authorization for `LookupController` or `FileController` endpoints
|
||||
- The system SHALL return HTTP 401 (not redirect) for unauthorized API requests to support Blazor WASM
|
||||
- The system SHALL configure cookie authentication to suppress redirect on 401
|
||||
|
||||
#### Scenario: Authorized user accesses protected controller
|
||||
|
||||
- **WHEN** an authenticated user accesses a controller with `[Authorize]` attribute
|
||||
- **THEN** the system allows the request to proceed to the action method
|
||||
|
||||
#### Scenario: Anonymous user accesses protected API
|
||||
|
||||
- **WHEN** an anonymous user accesses a controller with `[Authorize]` attribute
|
||||
- **THEN** the system returns HTTP 401 Unauthorized (no redirect)
|
||||
|
||||
#### Scenario: User accesses public lookup endpoint
|
||||
|
||||
- **WHEN** any user (authenticated or not) accesses a `LookupController` endpoint
|
||||
- **THEN** the system allows the request without authentication check
|
||||
|
||||
## Data Models
|
||||
|
||||
### UserInfo (formerly LDAPEntry)
|
||||
|
||||
```csharp
|
||||
public class UserInfo
|
||||
{
|
||||
public string DN { get; set; } = string.Empty; // Distinguished name
|
||||
public string Username { get; set; } = string.Empty; // sAMAccountName (lowercase)
|
||||
public string FirstName { get; set; } = string.Empty; // givenName
|
||||
public string LastName { get; set; } = string.Empty; // sn
|
||||
public string DisplayName => string.IsNullOrWhiteSpace(FirstName) && string.IsNullOrWhiteSpace(LastName)
|
||||
? Username
|
||||
: $"{FirstName} {LastName}".Trim(); // Computed
|
||||
public string Title { get; set; } = string.Empty; // title
|
||||
public string EmailAddress { get; set; } = string.Empty; // mail
|
||||
}
|
||||
```
|
||||
|
||||
### StatusUpdate
|
||||
|
||||
```csharp
|
||||
public class StatusUpdate
|
||||
{
|
||||
public string Message { get; set; } = string.Empty; // Update message to display
|
||||
public DateTime Timestamp { get; set; } // When message was sent
|
||||
}
|
||||
```
|
||||
|
||||
### SearchUpdate
|
||||
|
||||
```csharp
|
||||
public class SearchUpdate
|
||||
{
|
||||
public int ID { get; set; } // Search primary key
|
||||
public string UserName { get; set; } = string.Empty; // Username who submitted
|
||||
public string Name { get; set; } = string.Empty; // Search name
|
||||
|
||||
[JsonConverter(typeof(JsonStringEnumConverter))]
|
||||
public SearchStatus Status { get; set; } // Enum: New, Submitted, Started, Ended, Error
|
||||
|
||||
public DateTime? SubmitDT { get; set; } // When submitted (required for UI grid)
|
||||
public DateTime? StartDT { get; set; } // When processing started (required for UI grid)
|
||||
public DateTime? EndDT { get; set; } // When processing ended (required for UI grid)
|
||||
public DateTime Timestamp { get; set; } // When update was generated (required for ordering)
|
||||
}
|
||||
```
|
||||
|
||||
### FileUploadResult<T>
|
||||
|
||||
```csharp
|
||||
public class FileUploadResult<T>
|
||||
{
|
||||
public bool WasSuccessful { get; set; }
|
||||
public string? ErrorMessage { get; set; }
|
||||
public T[]? Data { get; set; }
|
||||
}
|
||||
```
|
||||
|
||||
### LoginRequest
|
||||
|
||||
```csharp
|
||||
public class LoginRequest
|
||||
{
|
||||
public string Username { get; set; } = string.Empty;
|
||||
public string Password { get; set; } = string.Empty;
|
||||
}
|
||||
```
|
||||
|
||||
### AuthResult
|
||||
|
||||
```csharp
|
||||
public record AuthResult(bool Success, UserInfo? User, string? ErrorMessage);
|
||||
```
|
||||
|
||||
## Migration Notes
|
||||
|
||||
| Legacy Pattern | New Pattern | Rationale |
|
||||
|----------------|-------------|-----------|
|
||||
| ASP.NET MVC 5 Controllers | ASP.NET Core Controllers with `[ApiController]` | Modern framework, automatic model binding and validation |
|
||||
| `JsonNetResult` custom result | `Results.Ok(data)` or `return Ok(data)` | Built-in JSON support with System.Text.Json |
|
||||
| Newtonsoft.Json `[JsonConverter(typeof(StringEnumConverter))]` | System.Text.Json `[JsonConverter(typeof(JsonStringEnumConverter))]` | Built-in serialization |
|
||||
| Legacy SignalR (`Microsoft.AspNet.SignalR`) | ASP.NET Core SignalR (`Microsoft.AspNetCore.SignalR`) | Built-in, cross-platform |
|
||||
| `GlobalHost.ConnectionManager.GetHubContext<T>()` | `IHubContext<T>` via dependency injection | Standard DI pattern, testable |
|
||||
| `Clients.All.statusUpdate(...)` | `Clients.All.SendAsync("statusUpdate", ...)` | Async-first API |
|
||||
| OWIN Authentication middleware | ASP.NET Core Authentication with `CookieAuthenticationDefaults` | Modern authentication stack |
|
||||
| `HttpContext.GetOwinContext().Authentication.SignIn()` | `HttpContext.SignInAsync()` | No OWIN abstraction layer |
|
||||
| `DefaultAuthenticationTypes.ApplicationCookie` | `CookieAuthenticationDefaults.AuthenticationScheme` | Standard scheme name |
|
||||
| `System.DirectoryServices.DirectoryEntry` | `System.DirectoryServices.Protocols.LdapConnection` | Cross-platform LDAP support |
|
||||
| `WebConfigurationManager.AppSettings` | `IOptions<LdapOptions>` / `IOptions<AuthOptions>` | Strongly-typed configuration |
|
||||
| `MemoryCache.Default` | `IMemoryCache` via dependency injection | Standard caching abstraction |
|
||||
| `Request.Files` collection | `IFormFile` parameter | Modern file upload handling |
|
||||
| EPPlus library | ClosedXML library | MIT license (EPPlus changed to non-commercial) |
|
||||
| Route-based MVC URLs (`/Search/GetSearches`) | Attribute routing (`/api/search`) | REST conventions, clear API structure |
|
||||
| Login redirect for unauthorized | HTTP 401 response | Blazor WASM SPA compatibility |
|
||||
| NLog for logging | `ILogger<T>` injected + `BeginScope()` for context | Built-in logging abstraction |
|
||||
| `LDAPEntry` model | `UserInfo` model | Clearer naming, not tied to LDAP implementation |
|
||||
| Hardcoded user exception in code | Configurable via `AuthOptions.AdminBypassUsers` | Production-safe configuration; empty array by default |
|
||||
|
||||
## Codex Review Findings (Status)
|
||||
|
||||
The following issues were identified during review and their resolution status:
|
||||
|
||||
| Finding | Status | Resolution |
|
||||
|---------|--------|------------|
|
||||
| Missing Account Endpoints | Resolved | New `/api/auth/*` endpoints documented |
|
||||
| Controller Name Typo (`SessionController` vs `SessionsController`) | Resolved | Legacy session endpoints not needed - Blazor WASM handles client-side routing |
|
||||
| InvalidUA Route | Resolved | Not needed - Blazor WASM handles user agent detection client-side |
|
||||
| LDAP DisplayName Fallback | Clarified | Spec now correctly documents computed `DisplayName` logic |
|
||||
| File Upload Error Handling | Documented | Per-row parse errors swallowed, specific error messages documented |
|
||||
| SignalR Best-Effort | Documented | SignalR publish is best-effort, exceptions logged but swallowed |
|
||||
|
||||
## Open Questions (Resolved)
|
||||
|
||||
| Question | Decision | Rationale |
|
||||
|----------|----------|-----------|
|
||||
| LDAP Failover Strategy | Keep simple sequential | Simple sequential failover is sufficient for this use case. Health checks and circuit breaker add complexity without significant benefit given the expected load. |
|
||||
| Authorization Granularity | Group membership only | The legacy system only requires group membership. More granular RBAC is not needed for this tool. |
|
||||
| SignalR Connection Management | Broadcast to all | Broadcasting to all clients is acceptable since the status updates are not user-specific and the number of concurrent users is expected to be small. |
|
||||
| API Authentication | Cookie-based only | JWT tokens add complexity without benefit for this internal browser-based tool. Cookie auth is simpler and more secure for browser clients. |
|
||||
| File Upload Security | Keep anonymous | Maintain legacy behavior. File uploads only parse Excel data; they don't access protected resources. |
|
||||
| Rate Limiting | Not implemented initially | Can be added later if needed. Current user base is small and internal. |
|
||||
| Hardcoded User Exception | Configurable admin bypass | Move to `AuthOptions.AdminBypassUsers` configuration array (empty by default). Allows dev flexibility without hardcoding. |
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user