dohertj2/jdescopingtool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e5fe2f06e94c9880e152ddec2f75393716e44821
jdescopingtool/openspec/specs/data-access/spec.md
T

2121 lines
73 KiB
Markdown
Raw Blame History

# Data Access Specification
## Purpose
The data access layer provides interface-based repositories with dependency injection for accessing three distinct data sources in the JDE Scoping Tool application:
1. **JDE Oracle** - Primary enterprise system (JD Edwards) for manufacturing data
2. **CMS (Oracle)** - Manufacturing Information System (MIS) data (legacy used DDTek.Oracle driver, migrating to Oracle.ManagedDataAccess.Core)
3. **SQL Server Cache (LotFinderDB)** - Local cache database for search operations
The layer follows a modern repository pattern with:
- **Interface-based repositories** (`ILotFinderRepository`, `IJdeRepository`, `ICmsRepository`) for testability and loose coupling
- **Connection factory abstraction** (`IDbConnectionFactory`) for database connections
- **Dependency injection** via extension methods for service registration
- **Async-first design** with `IAsyncEnumerable<T>` for streaming and `CancellationToken` support
- **Typed exception hierarchy** for consistent error handling
## Source Reference
| Legacy Files | Purpose |
|--------------|---------|
| OLD/DataModel/Process/LotFinderDB.cs | SQL Server cache - base class with connection management |
| OLD/DataModel/Process/LotFinderDB.SearchManagement.cs | SQL Server cache - search CRUD operations |
| OLD/DataModel/Process/LotFinderDB.Item.cs | SQL Server cache - item search/lookup |
| OLD/DataModel/Process/LotFinderDB.WorkOrder.cs | SQL Server cache - work order lookup |
| OLD/DataModel/Process/LotFinderDB.WorkCenter.cs | SQL Server cache - work center search/lookup |
| OLD/DataModel/Process/LotFinderDB.Lot.cs | SQL Server cache - lot lookup |
| OLD/DataModel/Process/LotFinderDB.ProfitCenter.cs | SQL Server cache - profit center search/lookup |
| OLD/DataModel/Process/LotFinderDB.User.cs | SQL Server cache - user search/lookup |
| OLD/DataModel/Process/LotFinderDB.MisData.cs | SQL Server cache - MIS data post-processing |
| OLD/DataModel/Process/JDE.cs | JDE Oracle - base class with connection management |
| OLD/DataModel/Process/JDE.WorkOrders.cs | JDE Oracle - work order queries |
| OLD/DataModel/Process/JDE.WorkOrderStep.cs | JDE Oracle - work order step queries |
| OLD/DataModel/Process/JDE.WorkOrderTime.cs | JDE Oracle - work order time transaction queries |
| OLD/DataModel/Process/JDE.WorkOrderRouting.cs | JDE Oracle - work order routing transaction queries |
| OLD/DataModel/Process/JDE.WorkOrderComponent.cs | JDE Oracle - work order component queries |
| OLD/DataModel/Process/JDE.Lots.cs | JDE Oracle - lot queries |
| OLD/DataModel/Process/JDE.LotUsage.cs | JDE Oracle - lot usage (cardex) queries |
| OLD/DataModel/Process/JDE.LotLocation.cs | JDE Oracle - lot location queries |
| OLD/DataModel/Process/JDE.Items.cs | JDE Oracle - item queries |
| OLD/DataModel/Process/JDE.Users.cs | JDE Oracle - user queries |
| OLD/DataModel/Process/JDE.BusinessUnits.cs | JDE Oracle - branch/profit center/work center queries |
| OLD/DataModel/Process/JDE.StatusCodes.cs | JDE Oracle - status code queries |
| OLD/DataModel/Process/JDE.FunctionCode.cs | JDE Oracle - function code queries |
| OLD/DataModel/Process/JDE.OrgHierarchy.cs | JDE Oracle - organization hierarchy queries |
| OLD/DataModel/Process/JDE.RouteMaster.cs | JDE Oracle - route master queries |
| OLD/DataModel/Process/CMS.cs | CMS - base class with connection management |
| OLD/DataModel/Process/CMS.MisData.cs | CMS - MIS data queries |
| OLD/DataModel/Process/QueryRepository.cs | External SQL query file management |
| OLD/DataModel/Config.cs | Connection string configuration |
---
## Requirements
### Requirement: Repository Interface Pattern
The system SHALL define repository interfaces for all data access operations.
#### Interface Definitions
```csharp
public interface ILotFinderRepository
{
// Search Management
Task<List<Search>> GetUserSearchesAsync(string userName, CancellationToken ct = default);
Task<List<Search>> GetQueuedSearchesAsync(CancellationToken ct = default);
Task<Search?> GetSearchAsync(int id, CancellationToken ct = default);
Task<byte[]?> GetSearchResultsAsync(int id, CancellationToken ct = default);
Task<int> SubmitSearchAsync(Search search, CancellationToken ct = default);
Task UpdateSearchStatusAsync(int id, SearchStatus status, CancellationToken ct = default);
Task UpdateSearchResultsAsync(int id, byte[] results, CancellationToken ct = default);
// Reference Data Lookup
Task<List<Item>> SearchItemsAsync(string filter, CancellationToken ct = default);
Task<List<Item>> LookupItemsAsync(List<string> itemNumbers, CancellationToken ct = default);
Task<List<WorkOrder>> LookupWorkordersAsync(List<long> workorderNumbers, CancellationToken ct = default);
Task<List<WorkCenter>> SearchWorkCentersAsync(string filter, CancellationToken ct = default);
Task<List<WorkCenter>> LookupWorkCentersAsync(List<string> codes, CancellationToken ct = default);
Task<List<ProfitCenter>> SearchProfitCentersAsync(string filter, CancellationToken ct = default);
Task<List<ProfitCenter>> LookupProfitCentersAsync(List<string> codes, CancellationToken ct = default);
Task<List<JdeUser>> SearchUsersAsync(string filter, CancellationToken ct = default);
Task<List<JdeUser>> LookupUsersAsync(List<string> userIds, CancellationToken ct = default);
Task<List<Lot>> LookupLotsAsync(List<LotViewModel> lots, CancellationToken ct = default);
// Data Sync Operations
Task<List<DataUpdate>> GetLastDataUpdatesAsync(CancellationToken ct = default);
Task<TableSpec> GetTableSpecAsync(string tableName, CancellationToken ct = default);
Task RebuildIndicesAsync(string tableName, CancellationToken ct = default);
Task PostProcessMisDataAsync(CancellationToken ct = default);
Task<int> BulkInsertAsync<T>(string tableName, IEnumerable<T> records, CancellationToken ct = default);
Task TruncateTableAsync(string tableName, CancellationToken ct = default);
}
public interface IJdeRepository
{
// Work Order Data
IAsyncEnumerable<WorkOrder> GetWorkOrdersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrder> GetWorkOrdersArchiveAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrderStep> GetWorkOrderStepsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrderStep> GetWorkOrderStepsArchiveAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrderTime> GetWorkOrderTimesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrderTime> GetWorkOrderTimesArchiveAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrderRouting> GetWorkOrderRoutingsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrderComponent> GetWorkOrderComponentsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkOrderComponent> GetWorkOrderComponentsArchiveAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
// Lot Data
IAsyncEnumerable<Lot> GetLotsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<LotUsage> GetLotUsagesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<LotUsage> GetLotUsagesArchiveAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<LotLocation> GetLotLocationsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
// Reference Data
IAsyncEnumerable<Item> GetItemsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<JdeUser> GetUsersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<Branch> GetBranchesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<ProfitCenter> GetProfitCentersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<WorkCenter> GetWorkCentersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<StatusCode> GetStatusCodesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<FunctionCode> GetFunctionCodesAsync(CancellationToken ct = default);
IAsyncEnumerable<OrgHierarchy> GetOrgHierarchyAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
IAsyncEnumerable<RouteMaster> GetRouteMastersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
}
public interface ICmsRepository
{
IAsyncEnumerable<MisData> GetMisDataAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default);
}
```
#### Business Rules
- All interfaces SHALL be registered with the DI container as scoped services
- Implementations SHALL accept `IDbConnectionFactory` via constructor injection
- Implementations SHALL accept `ILogger<T>` via constructor injection
- Implementations SHALL accept `IOptions<DataAccessOptions>` via constructor injection
#### Scenario: Repository injection
- **WHEN** a service requires data access and requests `ILotFinderRepository` via DI
- **THEN** the container provides a configured `LotFinderRepository` instance with connection factory, logger, and options injected
---
### Requirement: Connection Factory Pattern
The system SHALL provide a connection factory abstraction for database connections.
#### Interface Definition
```csharp
public interface IDbConnectionFactory
{
Task<SqlConnection> CreateLotFinderConnectionAsync(CancellationToken ct = default);
Task<OracleConnection> CreateJdeConnectionAsync(CancellationToken ct = default);
Task<OracleConnection> CreateJdeStageConnectionAsync(CancellationToken ct = default);
Task<OracleConnection> CreateCmsConnectionAsync(CancellationToken ct = default);
}
```
#### Business Rules
- Connection factory SHALL use `Microsoft.Data.SqlClient.SqlConnection` for SQL Server
- Connection factory SHALL use `Oracle.ManagedDataAccess.Core.OracleConnection` for all Oracle connections (JDE, JDE Stage, CMS)
- Connection strings SHALL be retrieved from `IConfiguration["ConnectionStrings:*"]`
- Secrets (passwords) SHALL be retrieved from Azure Key Vault or .NET Secret Manager (never stored in config files)
- Connections SHALL be opened asynchronously before returning
- Connection factory SHALL be registered as a singleton service
#### Scenario: Create SQL Server connection
- **WHEN** `CreateLotFinderConnectionAsync()` is called
- **THEN** a new `SqlConnection` is created with connection string from configuration, opened asynchronously, and returned
#### Scenario: Create Oracle connection
- **WHEN** `CreateJdeConnectionAsync()` is called
- **THEN** a new `OracleConnection` is created with connection string from configuration, opened asynchronously, and returned
---
### Requirement: Service Registration
The system SHALL provide DI extension methods for service registration.
#### Method Signature
```csharp
public static IServiceCollection AddDataAccess(this IServiceCollection services, IConfiguration configuration)
```
#### Business Rules
- Extension method SHALL register `IDbConnectionFactory` as singleton
- Extension method SHALL register `ILotFinderRepository` as scoped
- Extension method SHALL register `IJdeRepository` as scoped
- Extension method SHALL register `ICmsRepository` as scoped
- Extension method SHALL bind `DataAccessOptions` from configuration section "DataAccess"
#### Scenario: Register data access services
- **WHEN** `services.AddDataAccess(configuration)` is called during startup
- **THEN** all repository interfaces and connection factory are registered with appropriate lifetimes
---
### Requirement: Configuration Options
The system SHALL support configurable timeouts via `IOptions<DataAccessOptions>`.
#### Options Class
```csharp
public class DataAccessOptions
{
public int DefaultTimeoutSeconds { get; set; } = 600;
public int LotUsageTimeoutSeconds { get; set; } = 999999;
public int MisDataTimeoutSeconds { get; set; } = 60000;
public int RebuildIndexTimeoutSeconds { get; set; } = 600;
public string ProductionSchema { get; set; } = "PRODDTA";
public string ArchiveSchema { get; set; } = "ARCDTAPD";
public string StageSchema { get; set; } = "JDESTAGE";
}
```
#### Business Rules
- Default timeout SHALL be 600 seconds (10 minutes) for general queries
- LotUsage filtered queries SHALL use configurable timeout (default 999999 seconds)
- MIS data queries SHALL use configurable timeout (default 60000 seconds)
- Schema names SHALL be configurable for environment-specific deployments
---
### Requirement: Connection factory pattern
The system SHALL implement `IDbConnectionFactory` to provide database connections via dependency injection.
#### Implementation Pattern
```csharp
public class DbConnectionFactory : IDbConnectionFactory
{
private readonly IConfiguration _configuration;
private readonly ILogger<DbConnectionFactory> _logger;
public DbConnectionFactory(IConfiguration configuration, ILogger<DbConnectionFactory> logger)
{
_configuration = configuration;
_logger = logger;
}
public async Task<SqlConnection> CreateLotFinderConnectionAsync(CancellationToken ct = default)
{
var connectionString = _configuration.GetConnectionString("LotFinderDB")
?? throw new ConnectionException("LotFinderDB connection string not configured", "LotFinderDB");
var connection = new SqlConnection(connectionString);
try
{
await connection.OpenAsync(ct);
return connection;
}
catch (SqlException ex)
{
_logger.LogError(ex, "Failed to connect to LotFinderDB");
await connection.DisposeAsync();
throw new ConnectionException("LotFinderDB: failed to open connection to database.", "LotFinderDB", ex);
}
}
}
```
#### Business Rules
- Connection factory SHALL be registered as singleton
- Connections SHALL be opened asynchronously before returning
- Callers SHALL dispose returned connections when finished
- `ConnectionException` SHALL be thrown on connection failure with inner exception preserved
#### Scenario: Successful connection creation
- **WHEN** valid connection string exists in configuration
- **AND** `CreateLotFinderConnectionAsync()` is called
- **THEN** a new `SqlConnection` is created, opened, and returned
- **AND** the caller is responsible for disposal
#### Scenario: Connection failure handling
- **WHEN** database is unreachable
- **AND** `CreateLotFinderConnectionAsync()` is called
- **THEN** error is logged with exception details
- **AND** `ConnectionException` is thrown with descriptive message
- **AND** inner exception is preserved for debugging
---
### Requirement: Async streaming pattern
The system SHALL use `IAsyncEnumerable<T>` with Dapper's `QueryUnbufferedAsync` for streaming large result sets.
#### Implementation Pattern
```csharp
public async IAsyncEnumerable<WorkOrder> GetWorkOrdersAsync(
DateTime? lastUpdateDT = null,
[EnumeratorCancellation] CancellationToken ct = default)
{
await using var connection = await _connectionFactory.CreateJdeConnectionAsync(ct);
var sql = ApplySchemaPlaceholders(
lastUpdateDT.HasValue
? JdeQueries.SQL_GET_WORKORDERS_FILTERED
: JdeQueries.SQL_GET_WORKORDERS);
var command = new CommandDefinition(
sql,
parameters: lastUpdateDT.HasValue
? new { dateUpdated = ToJdeDate(lastUpdateDT.Value), timeUpdated = ToJdeTime(lastUpdateDT.Value) }
: null,
commandTimeout: _options.Value.DefaultTimeoutSeconds,
cancellationToken: ct);
await foreach (var workOrder in connection.QueryUnbufferedAsync<WorkOrder>(command).WithCancellation(ct))
{
yield return workOrder;
}
}
```
#### Business Rules
- All JDE/CMS collection queries SHALL return `IAsyncEnumerable<T>`
- Queries SHALL use `QueryUnbufferedAsync` to stream results
- Methods SHALL accept `CancellationToken` with `[EnumeratorCancellation]` attribute
- Cancellation SHALL be checked between row iterations via `WithCancellation(ct)`
#### Scenario: Stream large work order dataset
- **WHEN** `GetWorkOrdersAsync()` is called for 1 million work orders
- **THEN** results are streamed via `IAsyncEnumerable<WorkOrder>` one at a time
- **AND** memory usage remains constant regardless of result set size
- **AND** consumer can use `await foreach` syntax
#### Scenario: Cancel streaming operation
- **WHEN** cancellation is requested during `GetWorkOrdersAsync()` iteration
- **THEN** iteration stops after current row completes
- **AND** `OperationCanceledException` is thrown to consumer
- **AND** database connection is properly disposed
---
### Requirement: Schema placeholder replacement
The system SHALL replace schema placeholders in SQL queries from `DataAccessOptions`.
#### Implementation Pattern
```csharp
private string ApplySchemaPlaceholders(string sql)
{
return sql
.Replace("{ProductionSchema}", _options.Value.ProductionSchema)
.Replace("{ArchiveSchema}", _options.Value.ArchiveSchema)
.Replace("{StageSchema}", _options.Value.StageSchema);
}
```
#### Business Rules
- Schema names SHALL be configurable for environment-specific deployments
- Default values: PRODDTA (production), ARCDTAPD (archive), JDESTAGE (stage)
- Replacement SHALL occur at query execution time
#### Scenario: Query uses production schema
- **WHEN** SQL query contains `{ProductionSchema}.F4801`
- **AND** `DataAccessOptions.ProductionSchema` is "PRODDTA"
- **THEN** query is executed with `PRODDTA.F4801`
---
### Requirement: Table name whitelist validation
The system SHALL validate table names against an explicit whitelist to prevent SQL injection in `RebuildIndicesAsync`.
#### Implementation Pattern
```csharp
private static readonly HashSet<string> ValidTableNames = new(StringComparer.OrdinalIgnoreCase)
{
"Branch", "DataUpdate", "FunctionCode", "Item", "JdeUser",
"Lot", "LotLocation", "LotUsage_Curr", "LotUsage_Hist",
"MisData", "OrgHierarchy", "ProfitCenter", "RouteMaster",
"Search", "StatusCode", "WorkCenter",
"WorkOrder_Curr", "WorkOrder_Hist",
"WorkOrderComponent_Curr", "WorkOrderComponent_Hist",
"WorkOrderRouting",
"WorkOrderStep_Curr", "WorkOrderStep_Hist",
"WorkOrderTime_Curr", "WorkOrderTime_Hist"
};
public async Task RebuildIndicesAsync(string tableName, CancellationToken ct = default)
{
if (!ValidTableNames.Contains(tableName))
{
throw new ArgumentException($"Invalid table name: {tableName}", nameof(tableName));
}
await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
var sql = $"ALTER INDEX ALL ON [{tableName}] REBUILD WITH (FILLFACTOR = 95)";
await connection.ExecuteAsync(sql, commandTimeout: _options.Value.RebuildIndexTimeoutSeconds);
}
```
#### Business Rules
- Table name MUST be validated against explicit whitelist before execution
- `ArgumentException` SHALL be thrown for invalid table names
- Comparison SHALL be case-insensitive
#### Scenario: Valid table name accepted
- **WHEN** `RebuildIndicesAsync("WorkOrder_Curr")` is called
- **THEN** table name passes whitelist validation
- **AND** index rebuild executes successfully
#### Scenario: SQL injection attempt blocked
- **WHEN** `RebuildIndicesAsync("WorkOrder]; DROP TABLE Search;--")` is called
- **THEN** table name fails whitelist validation
- **AND** `ArgumentException` is thrown with message "Invalid table name"
- **AND** no SQL is executed
---
### Requirement: Exception logging with scope context
The system SHALL log exceptions with structured scope context before throwing.
#### Implementation Pattern
```csharp
catch (OracleException ex)
{
using (_logger.BeginScope(new Dictionary<string, object>
{
["DataSource"] = "JDE",
["Operation"] = nameof(GetWorkOrdersAsync),
["QueryName"] = "SQL_GET_WORKORDERS"
}))
{
_logger.LogError(ex, "Query execution failed");
}
throw new QueryException("Failed to execute work order query", "SQL_GET_WORKORDERS", ex);
}
```
#### Business Rules
- All exceptions SHALL be logged at throw site
- Log context SHALL include: DataSource, Operation, and QueryName where applicable
- Inner exceptions SHALL be preserved in thrown exceptions
- Structured logging SHALL enable log aggregation and analysis
#### Scenario: Query exception with full context
- **WHEN** JDE Oracle query fails with OracleException
- **THEN** error is logged with BeginScope containing DataSource, Operation, QueryName
- **AND** `QueryException` is thrown with descriptive message and inner exception
---
### Requirement: Search criteria extraction via SQL functions
The system SHALL use SQL extraction functions to retrieve filter criteria directly from the `Search.Criteria` JSON column.
#### Implementation Pattern
Search query building now uses SearchId to invoke extraction functions rather than passing filter values from C#:
```csharp
public SearchQueryResult BuildSearchQuery(int searchId)
{
// Query builder generates SQL that calls extraction functions
// Example generated SQL fragment:
// INSERT INTO #P_WorkOrders SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(@SearchId)
// INSERT INTO #P_ItemNumbers SELECT ItemNumber FROM dbo.fn_GetSearchItemNumbers(@SearchId)
return new SearchQueryResult(sql, new { SearchId = searchId });
}
```
#### Business Rules
- Query builder SHALL accept only `searchId` parameter (not full criteria object)
- SQL queries SHALL call extraction functions to populate temporary filter tables
- Extraction functions handle JSON parsing and validation in SQL Server
- Invalid JSON or missing criteria results in empty filter sets (no errors thrown)
#### Available Extraction Functions
| Function | Returns |
|----------|---------|
| `fn_GetSearchMinimumDt` | DATETIME2 scalar |
| `fn_GetSearchMaximumDt` | DATETIME2 scalar |
| `fn_GetSearchExtractMisData` | BIT scalar |
| `fn_GetSearchWorkOrders` | WorkOrderNumber table |
| `fn_GetSearchItemNumbers` | ItemNumber table |
| `fn_GetSearchProfitCenters` | Code table |
| `fn_GetSearchWorkCenters` | Code table |
| `fn_GetSearchOperatorIDs` | OperatorID table |
| `fn_GetSearchComponentLots` | LotNumber, ItemNumber table |
| `fn_GetSearchPartOperations` | ItemNumber, OperationNumber, MisNumber, MisRevision table |
#### Scenario: Build search query with extraction functions
- **WHEN** `BuildSearchQuery(123)` is called for a search with work order filter
- **THEN** generated SQL includes `SELECT WorkOrderNumber FROM dbo.fn_GetSearchWorkOrders(123)`
- **AND** only the `@SearchId` parameter is passed to the query
---
### Requirement: Table-valued parameter support for lookups
The system SHALL use DataTable for SQL Server table-valued parameters in reference data lookup methods.
#### Business Rules
- TVPs are used for batch lookups (LookupItemsAsync, LookupWorkordersAsync, etc.)
- TVPs are NOT used for search query execution (replaced by extraction functions)
- DataTable column names SHALL match TVP type column names
#### Scenario: Batch lookup with TVP
- **WHEN** `LookupItemsAsync(["ITEM001", "ITEM002", "ITEM003"])` is called
- **THEN** DataTable is created with ItemNumber column
- **AND** single query executes with TVP parameter
- **AND** matching items are returned
---
### Requirement: Service registration extension method
The system SHALL provide `AddDataAccess` extension method for DI registration.
#### Implementation Pattern
```csharp
public static IServiceCollection AddDataAccess(
this IServiceCollection services,
IConfiguration configuration)
{
services.Configure<DataAccessOptions>(
configuration.GetSection(DataAccessOptions.SectionName));
services.AddSingleton<IDbConnectionFactory, DbConnectionFactory>();
services.AddScoped<ILotFinderRepository, LotFinderRepository>();
services.AddScoped<IJdeRepository, JdeRepository>();
services.AddScoped<ICmsRepository, CmsRepository>();
return services;
}
```
#### Business Rules
- Extension method SHALL bind `DataAccessOptions` from "DataAccess" configuration section
- Connection factory SHALL be registered as singleton
- Repositories SHALL be registered as scoped services
- Method SHALL return `IServiceCollection` for chaining
#### Scenario: Register all data access services
- **WHEN** `services.AddDataAccess(configuration)` is called during startup
- **THEN** `DataAccessOptions` is bound from configuration
- **AND** `IDbConnectionFactory` is registered as singleton
- **AND** all repository interfaces are registered as scoped
## Exception Handling
### Requirement: Custom Exception Types
The system SHALL define custom exception types for data access errors.
#### Exception Hierarchy
```csharp
public class DataAccessException : Exception
{
public string? Operation { get; }
public string? Repository { get; }
public DataAccessException(string message, string? operation = null, string? repository = null, Exception? inner = null);
}
public class ConnectionException : DataAccessException
{
public string? DataSource { get; }
public ConnectionException(string message, string dataSource, Exception? inner = null);
}
public class QueryException : DataAccessException
{
public string? QueryName { get; }
public QueryException(string message, string queryName, Exception? inner = null);
}
public class DataAccessTimeoutException : DataAccessException
{
public int TimeoutSeconds { get; }
public DataAccessTimeoutException(string message, int timeoutSeconds, Exception? inner = null);
}
```
#### Business Rules
- All repository methods SHALL throw typed exceptions on error (never return null/empty on error)
- `ConnectionException` SHALL be thrown for connection failures
- `QueryException` SHALL be thrown for query execution failures
- `DataAccessTimeoutException` SHALL be thrown for timeout errors
- Exception SHALL be logged at throw site via `ILogger<T>` with `BeginScope()` for context
- Inner exceptions SHALL be preserved for debugging
#### Scenario: Connection failure logging
- **WHEN** a connection to JDE Oracle fails
- **THEN** error is logged with scope context (data source, operation) and `ConnectionException` is thrown with descriptive message and inner exception
#### Scenario: Query timeout
- **WHEN** a query exceeds configured timeout
- **THEN** `DataAccessTimeoutException` is thrown with timeout value and query name
---
## Query Management
### Requirement: QueryRepository component
The system SHALL manage SQL queries as embedded resources or compile-time constants.
#### Business Rules
- SQL queries SHALL be stored as embedded resources in the assembly or as `const string` fields
- Queries SHALL be loaded once at application startup and cached in memory
- Schema placeholder replacement (`PRODDTA`, `ARCDTAPD`, `JDESTAGE`) SHALL use values from `DataAccessOptions`
- Query loading SHALL throw `QueryException` if query is not found
#### Scenario: Load query from embedded resource
- **WHEN** repository is instantiated and SQL queries are needed
- **THEN** queries are loaded from embedded resources with schema placeholders replaced from configuration
#### Scenario: Query not found
- **WHEN** a requested query name does not exist in embedded resources
- **THEN** `QueryException` is thrown with message containing the query name
---
## Connection Management
### Requirement: Connection Management - LotFinderDB component
The system SHALL provide SQL Server cache database connections via `IDbConnectionFactory`.
#### Method: CreateLotFinderConnectionAsync
Opens a new connection to the LotFinderDB SQL Server database.
| Parameter | Type | Description |
|-----------|------|-------------|
| ct | CancellationToken | Cancellation token for async operation |
| Returns | Type | Description |
|---------|------|-------------|
| connection | Task<SqlConnection> | Open connection to LotFinderDB |
#### Business Rules
- Connection string loaded from `IConfiguration["ConnectionStrings:LotFinderDB"]`
- Password retrieved from Azure Key Vault or .NET Secret Manager
- Uses `Microsoft.Data.SqlClient.SqlConnection`
- Returns opened connection (caller must dispose)
- Throws `ConnectionException` on failure with descriptive message
- Default command timeout: configurable via `DataAccessOptions.DefaultTimeoutSeconds`
#### Scenario: Successful connection
- **WHEN** valid LotFinderDB connection string exists in configuration and `CreateLotFinderConnectionAsync()` is called
- **THEN** a new `SqlConnection` is created, opened asynchronously, and returned
#### Scenario: Connection failure
- **WHEN** invalid or unreachable database exists and `CreateLotFinderConnectionAsync()` is called
- **THEN** error is logged via `ILogger<T>` and `ConnectionException` is thrown with message "LotFinderDB: failed to open connection to database."
---
### Requirement: Connection Management - JDE component
The system SHALL provide JDE Oracle database connections via `IDbConnectionFactory`.
#### Method: CreateJdeConnectionAsync
Opens a new connection to the JDE Oracle database.
| Parameter | Type | Description |
|-----------|------|-------------|
| ct | CancellationToken | Cancellation token for async operation |
| Returns | Type | Description |
|---------|------|-------------|
| connection | Task<OracleConnection> | Open connection to JDE (Oracle.ManagedDataAccess.Core) |
#### Business Rules
- Connection string loaded from `IConfiguration["ConnectionStrings:JDE"]`
- Password retrieved from Azure Key Vault or .NET Secret Manager
- Uses `Oracle.ManagedDataAccess.Core.OracleConnection`
- Returns opened connection (caller must dispose)
- Throws `ConnectionException` on failure
#### Scenario: Successful JDE connection
- **WHEN** valid JDE Oracle connection string exists and `CreateJdeConnectionAsync()` is called
- **THEN** a new OracleConnection is created, opened asynchronously, and returned
---
### Requirement: Connection Management - JDE Stage component
The system SHALL provide a separate JDE connection for stage/view queries.
#### Method: CreateJdeStageConnectionAsync
Opens a new connection to the JDE Stage database (for JDESTAGE schema views).
#### Business Rules
- Connection string loaded from `IConfiguration["ConnectionStrings:JDEStage"]`
- Uses `Oracle.ManagedDataAccess.Core.OracleConnection`
- Used for `GetStatusCodes()` and `GetLotLocations()` methods
- Queries JDESTAGE schema views (F0005_VIEW, F41021_VIEW)
---
### Requirement: Connection Management - CMS component
The system SHALL provide CMS database connections via `IDbConnectionFactory`.
#### Method: CreateCmsConnectionAsync
Opens a new connection to the CMS database.
| Parameter | Type | Description |
|-----------|------|-------------|
| ct | CancellationToken | Cancellation token for async operation |
| Returns | Type | Description |
|---------|------|-------------|
| connection | Task<OracleConnection> | Open connection to CMS |
#### Business Rules
- Connection string loaded from `IConfiguration["ConnectionStrings:CMS"]`
- Password retrieved from Azure Key Vault or .NET Secret Manager
- Uses `Oracle.ManagedDataAccess.Core.OracleConnection` (consolidated from legacy DDTek.Oracle)
---
## Async Streaming
### Requirement: IAsyncEnumerable Streaming
The system SHALL use `IAsyncEnumerable<T>` for streaming large datasets.
#### Business Rules
- All JDE/CMS query methods returning collections SHALL return `IAsyncEnumerable<T>`
- Implementations SHALL use Dapper's `QueryUnbufferedAsync` for streaming results
- All streaming methods SHALL accept `CancellationToken` parameter
- Cancellation SHALL be checked between row iterations
#### Scenario: Stream work orders
- **WHEN** `GetWorkOrdersAsync()` is called with large dataset
- **THEN** results are streamed via `IAsyncEnumerable<WorkOrder>` without loading all rows into memory
#### Scenario: Cancel streaming operation
- **WHEN** cancellation is requested during `GetWorkOrdersAsync()` iteration
- **THEN** iteration stops and `OperationCanceledException` is thrown
---
## JDE Oracle Queries
### Requirement: JDE.GetWorkOrders component
The system SHALL fetch work order records from JDE F4801 table.
#### Method Signature
```csharp
IAsyncEnumerable<WorkOrder> GetWorkOrdersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| lastUpdateDT | DateTime? | Optional cutoff for incremental sync |
| ct | CancellationToken | Cancellation token |
#### Returns
- `IAsyncEnumerable<WorkOrder>` - Streaming results via QueryUnbufferedAsync
#### Query: SQL_GET_WORKORDERS
```sql
SELECT wo.WADOCO AS WorkOrderNumber,
TRIM(wo.WAMMCU) AS BranchCode,
TRIM(wo.WALOTN) AS LotNumber,
TRIM(wo.WALITM) AS ItemNumber,
wo.WAITM AS ShortItemNumber,
TRIM(wo.WAPARS) AS ParentWorkOrderNumber,
wo.WAUORG / 100.0 AS OrderQuantity,
wo.WASOBK / 100.0 AS HeldQuantity,
wo.WASOQS / 100.0 AS ShippedQuantity,
TRIM(wo.WASRST) AS StatusCode,
CASE wo.WADCG WHEN 0 THEN TO_DATE('1900-01-01', 'YYYY-MM-DD')
ELSE TO_DATE(wo.WADCG+1900000,'YYYYDDD') END AS StatusCodeUpdateDT,
CASE wo.WATRDJ WHEN 0 THEN TO_DATE('1900-01-01', 'YYYY-MM-DD')
ELSE TO_DATE(wo.WATRDJ+1900000,'YYYYDDD') END AS IssueDate,
CASE wo.WASTRT WHEN 0 THEN TO_DATE('1900-01-01', 'YYYY-MM-DD')
ELSE TO_DATE(wo.WASTRT+1900000,'YYYYDDD') END AS StartDate,
TRIM(wo.WATRT) AS RoutingType,
wo.WAUPMJ AS LastUpdateDate,
wo.WATDAY AS LastUpdateTime
FROM {ProductionSchema}.F4801 wo
```
#### Query: SQL_GET_WORKORDERS_FILTERED (Incremental)
Same SELECT with WHERE clause:
```sql
WHERE (wo.WAUPMJ > :dateUpdated OR
(wo.WAUPMJ = :dateUpdated AND wo.WATDAY >= :timeUpdated))
```
#### Query: SQL_GET_WORKORDERS_ARCHIVE
Same SELECT from `{ArchiveSchema}.F4801` (archive schema)
#### JDE Table Mapping
| JDE Column | Domain Property | Transformation |
|------------|-----------------|----------------|
| WADOCO | WorkOrderNumber | Direct |
| WAMMCU | BranchCode | TRIM |
| WALOTN | LotNumber | TRIM |
| WALITM | ItemNumber | TRIM |
| WAITM | ShortItemNumber | Direct |
| WAPARS | ParentWorkOrderNumber | TRIM |
| WAUORG | OrderQuantity | / 100.0 |
| WASOBK | HeldQuantity | / 100.0 |
| WASOQS | ShippedQuantity | / 100.0 |
| WASRST | StatusCode | TRIM |
| WADCG | StatusCodeUpdateDT | JDE date conversion |
| WATRDJ | IssueDate | JDE date conversion |
| WASTRT | StartDate | JDE date conversion |
| WATRT | RoutingType | TRIM |
| WAUPMJ | LastUpdateDate | JDE date (integer) |
| WATDAY | LastUpdateTime | JDE time (integer) |
#### Business Rules
- Quantities stored in JDE as integer * 100, divided for decimal values
- JDE dates stored as integer CYYDDD format, converted to DATE
- Zero dates converted to '1900-01-01'
- Uses configurable query timeout from `DataAccessOptions.DefaultTimeoutSeconds`
- Streaming results via `QueryUnbufferedAsync` for memory efficiency
- Schema placeholder `{ProductionSchema}` replaced from configuration
#### Scenario: Full sync (mass update)
- **WHEN** lastUpdateDT is null and `GetWorkOrdersAsync()` is called
- **THEN** SQL_GET_WORKORDERS query returns all work orders from JDE as async stream
#### Scenario: Incremental sync (daily/hourly update)
- **WHEN** lastUpdateDT is 2024-01-15 and `GetWorkOrdersAsync(lastUpdateDT)` is called
- **THEN** SQL_GET_WORKORDERS_FILTERED returns only records updated since that date
#### Scenario: JDE quantity conversion
- **WHEN** JDE WAUORG value is 10050 (integer) and mapping to WorkOrder.OrderQuantity
- **THEN** OrderQuantity equals 100.50 (decimal, divided by 100)
---
### Requirement: JDE.GetWorkOrderSteps component
The system SHALL fetch work order operation steps from JDE F3112 table.
#### Method Signature
```csharp
IAsyncEnumerable<WorkOrderStep> GetWorkOrderStepsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_WORKORDER_STEP
```sql
SELECT wos.WLDOCO AS WorkOrderNumber,
wos.WLOPSQ/10 AS StepNumber,
TRIM(wos.WLMCU) AS WorkCenterCode,
TRIM(wos.WLMMCU) AS BranchCode,
TRIM(wos.WLDSC1) AS StepDescription,
TRIM(mes.CFDS80) AS FunctionOperationDescription,
wos.WLOPSC AS StepTypeCode,
CASE wos.WLSTRT WHEN 0 THEN NULL
ELSE TO_DATE(wos.WLSTRT+1900000,'YYYYDDD') END AS StartDT,
CASE wos.WLSTRX WHEN 0 THEN NULL
ELSE TO_DATE(wos.WLSTRX+1900000,'YYYYDDD') END AS EndDT,
TRIM(wos.WLURRF) AS FunctionCode,
wos.WLSOCN / 100.0 AS ScrappedQuantity,
wos.WLUPMJ AS LastUpdateDate,
wos.WLTDAY AS LastUpdateTime
FROM {ProductionSchema}.F3112 wos LEFT OUTER JOIN
{ProductionSchema}.F00192 mes ON (wos.WLURRF = mes.CFKY)
WHERE TRIM(wos.WLMCU) IS NOT NULL AND
TRIM(wos.WLMMCU) IS NOT NULL
```
#### Business Rules
- StepNumber is WLOPSQ/10 (stored as tenths in JDE)
- LEFT OUTER JOIN to F00192 for FunctionOperationDescription
- Filters out records with NULL work center or branch
- Archive queries use `{ArchiveSchema}.F3112`
#### Scenario: Step number conversion
- **WHEN** JDE WLOPSQ value is 100 (tenths) and mapping to WorkOrderStep.StepNumber
- **THEN** StepNumber equals 10.0 (decimal)
#### Scenario: Enrich with function description
- **WHEN** step has FunctionCode "ASSY" and LEFT OUTER JOIN to F00192 succeeds
- **THEN** FunctionOperationDescription is populated from media object
---
### Requirement: JDE.GetWorkOrderTimes component
The system SHALL fetch work order time transactions (operator labor) from JDE F31122 table.
#### Method Signature
```csharp
IAsyncEnumerable<WorkOrderTime> GetWorkOrderTimesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_WORKORDER_TIMES
```sql
SELECT wot.WTUKID AS UniqueID,
wot.WTDOCO AS WorkOrderNumber,
wot.WTOPSQ/10 AS StepNumber,
TRIM(wot.WTMCU) AS WorkCenterCode,
TRIM(wot.WTMMCU) AS BranchCode,
wot.WTAN8 AS AddressNumber,
CASE wot.WTDGL WHEN 0 THEN NULL
ELSE TO_DATE(wot.WTDGL+1900000,'YYYYDDD') END AS GlDate,
wot.WTUPMJ AS LastUpdateDate,
wot.WTTDAY AS LastUpdateTime
FROM {ProductionSchema}.F31122 wot
WHERE TRIM(wot.WTMCU) IS NOT NULL AND
TRIM(wot.WTMMCU) IS NOT NULL
```
#### Business Rules
- Links operators to work orders via AddressNumber -> JdeUser
- Filters out records with NULL work center or branch
---
### Requirement: JDE.GetWorkOrderRoutings component
The system SHALL fetch work order routing transactions from JDE F3112Z1 table.
#### Method Signature
```csharp
IAsyncEnumerable<WorkOrderRouting> GetWorkOrderRoutingsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_WORKORDER_ROUTING
```sql
SELECT TRIM(woz.SZEDUS) AS UserID,
TRIM(woz.SZEDBT) AS BatchNumber,
TRIM(woz.SZEDTN) AS TransactionNumber,
woz.SZEDLN AS LineNumber,
woz.SZOPSQ / 10.0 AS StepNumber,
TRIM(woz.SZMCU) AS WorkCenterCode,
woz.SZDOCO AS WorkOrderNumber,
TRIM(woz.SZTRT) AS RoutingType,
TRIM(woz.SZMMCU) AS BranchCode,
TRIM(woz.SZDSC1) AS StepDescription,
TRIM(woz.SZURRF) AS FunctionCode,
woz.SZTRDJ AS TransactionDate_Date,
woz.SZUPMJ AS LastUpdateDate,
woz.SZTDAY AS LastUpdateTime
FROM {ProductionSchema}.F3112Z1 woz
WHERE woz.SZTYTN = 'JDERTG' AND
woz.SZDRIN = '2' AND
woz.SZTNAC = '02' AND
woz.SZPID = 'ER31410' AND
TRIM(woz.SZEDUS) IS NOT NULL AND
TRIM(woz.SZEDBT) IS NOT NULL AND
TRIM(woz.SZEDTN) IS NOT NULL AND
TRIM(woz.SZMCU) IS NOT NULL
```
#### Business Rules
- Filters by specific transaction type (JDERTG), direction (2), action (02), program ID (ER31410)
- Used for MIS matching to find original operation sequence
- **Data validation**: Skips records where LastUpdateDT or TransactionDate year < 1900 or > 2500
- No archive table (single table for all data)
---
### Requirement: JDE.GetWorkOrderComponents component
The system SHALL fetch work order component usage from JDE F3111 table.
#### Method Signature
```csharp
IAsyncEnumerable<WorkOrderComponent> GetWorkOrderComponentsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_WORKORDER_COMPONENTS
```sql
SELECT woc.WMUKID AS UniqueID,
woc.WMDOCO AS WorkOrderNumber,
TRIM(woc.WMLOTN) AS LotNumber,
TRIM(woc.WMCMCU) AS BranchCode,
woc.WMCPIT AS ShortItemNumber,
woc.WMTRQT / 100.0 AS Quantity,
woc.WMUPMJ AS LastUpdateDate,
woc.WMTDAY AS LastUpdateTime
FROM {ProductionSchema}.F3111 woc
WHERE TRIM(woc.WMLOTN) IS NOT NULL
```
#### Business Rules
- Filters records with NULL lot number
- Archive queries use `{ArchiveSchema}.F3111`
---
### Requirement: JDE.GetLots component
The system SHALL fetch lot master data from JDE F4108 table.
#### Method Signature
```csharp
IAsyncEnumerable<Lot> GetLotsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_LOTS
```sql
SELECT TRIM(lot.IOLOTN) AS LotNumber,
TRIM(lot.IOMCU) AS BranchCode,
lot.IOITM AS ShortItemNumber,
TRIM(lot.IOLITM) AS ItemNumber,
lot.IOVEND AS SupplierCode,
lot.IOLOTS AS StatusCode,
lot.IOLOT1 AS Memo1,
lot.IOLOT2 AS Memo2,
lot.IOLOT3 AS Memo3,
lot.IOUPMJ AS LastUpdateDate,
lot.IOTDAY AS LastUpdateTime
FROM {ProductionSchema}.F4108 lot
WHERE TRIM(lot.IOLOTN) IS NOT NULL AND
TRIM(lot.IOMCU) IS NOT NULL
```
#### Business Rules
- StatusCode (IOLOTS) is single character
- Filters out records with NULL lot number or branch code
---
### Requirement: JDE.GetLotUsages component
The system SHALL fetch lot usage (cardex) transactions from JDE F4111 table.
#### Method Signature
```csharp
IAsyncEnumerable<LotUsage> GetLotUsagesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_LOT_USAGES
```sql
SELECT lu.ILUKID AS UniqueID,
lu.ILDOCO AS WorkOrderNumber,
TRIM(lu.ILLOTN) AS LotNumber,
TRIM(lu.ILMCU) AS BranchCode,
lu.ILITM AS ShortItemNumber,
lu.ILTRQT AS Quantity,
lu.ILTRDJ AS LastUpdateDate,
lu.ILTDAY AS LastUpdateTime
FROM {ProductionSchema}.F4111 lu
WHERE lu.ILDCT = 'IM' AND
TRIM(lu.ILLOTN) IS NOT NULL
```
#### Business Rules
- Filters by document type 'IM' (inventory management)
- Filters out records with NULL lot number
- **Special timeout**: Filtered query uses `DataAccessOptions.LotUsageTimeoutSeconds` (default 999999)
- Archive queries use `{ArchiveSchema}.F4111`
---
### Requirement: JDE.GetLotLocations component
The system SHALL fetch lot location tracking from JDESTAGE.F41021_VIEW.
#### Method Signature
```csharp
IAsyncEnumerable<LotLocation> GetLotLocationsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_LOT_LOCATIONS
```sql
SELECT TRIM(il.LOT_LILOTN) AS LotNumber,
il.IDENTIFIERSHORTITEM_LIITM AS ShortItemNumber,
TRIM(il.COSTCENTER_LIMCU) AS BranchCode,
COALESCE(TRIM(il.LOCATION_LILOCN), ' ') AS Location,
il.DATEUPDATED_LIUPMJ + FLOOR(il.TIMEOFDAY_LITDAY / 10000) / 24 +
FLOOR(MOD(il.TIMEOFDAY_LITDAY, 10000) / 100) / 1440 +
MOD(il.TIMEOFDAY_LITDAY, 100) / 86400 AS LastUpdateDT
FROM {StageSchema}.F41021_VIEW il
WHERE TRIM(il.LOT_LILOTN) IS NOT NULL
```
#### Business Rules
- Uses JDESTAGE view via `CreateJdeStageConnectionAsync()` (different connection than main JDE)
- Location defaults to space if NULL
- LastUpdateDT computed inline from JDE date/time integers
---
### Requirement: JDE.GetItems component
The system SHALL fetch item (part number) master data from JDE F4101 table.
#### Method Signature
```csharp
IAsyncEnumerable<Item> GetItemsAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_ITEMS
```sql
SELECT pn.IMITM AS ShortItemNumber,
TRIM(pn.IMLITM) AS ItemNumber,
TRIM(pn.IMDSC1) AS Description,
TRIM(pn.IMPRP4) AS PlanningFamily,
TRIM(pn.IMSTKT) AS StockingType,
pn.IMUPMJ AS LastUpdateDate,
pn.IMTDAY AS LastUpdateTime
FROM {ProductionSchema}.F4101 pn
WHERE TRIM(pn.IMLITM) IS NOT NULL
```
---
### Requirement: JDE.GetUsers component
The system SHALL fetch user/operator data from JDE F0101 and SY920.F0092 tables.
#### Method Signature
```csharp
IAsyncEnumerable<JdeUser> GetUsersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_USERS
```sql
WITH USER_CTE AS (
SELECT ab.ABAN8 AS AddressNumber,
TRIM(pro.ULUSER) AS UserID,
TRIM(ab.ABALPH) AS FullName,
ab.ABUPMJ AS LastUpdateDate,
ab.ABUPMT AS LastUpdateTime,
ROW_NUMBER() OVER (PARTITION BY ab.ABAN8 ORDER BY ab.ABUPMJ DESC, ab.ABUPMT DESC) RN
FROM {ProductionSchema}.F0101 ab LEFT OUTER JOIN
SY920.F0092 pro ON (ab.ABAN8 = pro.ULAN8)
)
SELECT AddressNumber,
UserID,
FullName,
LastUpdateDate,
LastUpdateTime
FROM USER_CTE
WHERE RN = 1
```
#### Business Rules
- CTE with ROW_NUMBER to get latest record per AddressNumber
- LEFT OUTER JOIN to SY920 schema for UserID lookup
- **Note**: Incremental filtering not supported for users (full sync always)
---
### Requirement: JDE.GetBranches / GetProfitCenters / GetWorkCenters component
The system SHALL fetch business unit reference data from JDE F0006 table.
#### Method Signature
```csharp
IAsyncEnumerable<Branch> GetBranchesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
IAsyncEnumerable<ProfitCenter> GetProfitCentersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
IAsyncEnumerable<WorkCenter> GetWorkCentersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_BUSINESS_UNITS
```sql
SELECT TRIM(wc.MCMCU) AS Code,
TRIM(wc.MCDL01) AS Description,
wc.MCUPMJ AS LastUpdateDate,
wc.MCUPMT AS LastUpdateTime
FROM {ProductionSchema}.F0006 wc
WHERE wc.MCSTYL = :typeCode
```
#### Parameters
| Entity | typeCode Parameter |
|--------|-------------------|
| Branch | 'BP' |
| ProfitCenter | 'I3' |
| WorkCenter | 'WC' |
---
### Requirement: JDE.GetStatusCodes component
The system SHALL fetch work order status codes from JDESTAGE.F0005_VIEW.
#### Method Signature
```csharp
IAsyncEnumerable<StatusCode> GetStatusCodesAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_STATUS_CODES
```sql
SELECT TRIM(sc.USERDEFINEDCODE_DRKY) AS CODE,
TRIM(sc.DESCRIPTION001_DRDL01) AS Description,
sc.DATEUPDATED_DRUPMJ + FLOOR(sc.TIMELASTUPDATED_DRUPMT / 10000) / 24 +
FLOOR(MOD(sc.TIMELASTUPDATED_DRUPMT, 10000) / 100) / 1440 +
MOD(sc.TIMELASTUPDATED_DRUPMT, 100) / 86400 AS LastUpdateDT
FROM {StageSchema}.F0005_VIEW sc
WHERE TRIM(sc.PRODUCTCODE_DRSY) = '00' AND
sc.USERDEFINEDCODES_DRRT = 'SS' AND
TRIM(sc.USERDEFINEDCODE_DRKY) IS NOT NULL
```
#### Business Rules
- Uses JDE Stage connection via `CreateJdeStageConnectionAsync()` (different connection than main JDE)
- Filters by product code '00' and UDC type 'SS'
- LastUpdateDT computed inline from date/time integers
---
### Requirement: JDE.GetFunctionCodes component
The system SHALL fetch function code lookup data from JDE F00192 table.
#### Method Signature
```csharp
IAsyncEnumerable<FunctionCode> GetFunctionCodesAsync(CancellationToken ct = default)
```
#### Query: SQL_GET_FUNCTION_CODES
```sql
SELECT Code,
TRIM(LISTAGG(Description, ' ') WITHIN GROUP(ORDER BY Description) ||
CASE WHEN MAX(total_lengthb) > 4000 THEN '...' ELSE '' END) Description,
SYSDATE AS LastUpdateDT
FROM (
SELECT TRIM(fc.CFKY) AS Code,
TRIM(ASCIISTR(fc.CFDS80)) AS Description,
SUM(LENGTHB(TRIM(fc.CFDS80))+1) OVER(PARTITION BY TRIM(fc.CFKY)
ORDER BY TRIM(fc.CFDS80)) - 1 cumul_lengthb,
SUM(LENGTHB(TRIM(fc.CFDS80))+1) OVER(PARTITION BY TRIM(fc.CFKY)) - 1 total_lengthb,
COUNT(*) OVER(PARTITION BY TRIM(fc.CFKY)) num_values
FROM {ProductionSchema}.F00192 fc
WHERE TRIM(fc.CFKY) IS NOT NULL
)
WHERE total_lengthb <= 4000 OR cumul_lengthb <= 4000 - length('...')
GROUP BY Code
```
#### Business Rules
- Aggregates multiple description rows per code using LISTAGG
- Truncates at 4000 bytes with '...' suffix
- Uses ASCIISTR to handle special characters
- LastUpdateDT set to SYSDATE (current timestamp)
- **Note**: Does not support incremental filtering (no lastUpdateDT parameter)
---
### Requirement: JDE.GetOrgHierarchy component
The system SHALL fetch organization hierarchy (work center to profit center mapping) from JDE F30006 table.
#### Method Signature
```csharp
IAsyncEnumerable<OrgHierarchy> GetOrgHierarchyAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_ORG_HIERARCHY
```sql
SELECT TRIM(oh.IWMCUW) AS ProfitCenterCode,
TRIM(oh.IWMCU) AS WorkCenterCode,
TRIM(oh.IWMMCU) AS BranchCode,
oh.IWUPMJ AS LastUpdateDate,
oh.IWTDAY AS LastUpdateTime
FROM {ProductionSchema}.F30006 oh
WHERE TRIM(oh.IWMCU) IS NOT NULL AND
TRIM(oh.IWMMCU) IS NOT NULL
```
---
### Requirement: JDE.GetRouteMasters component
The system SHALL fetch item routing master data from JDE F3003 table.
#### Method Signature
```csharp
IAsyncEnumerable<RouteMaster> GetRouteMastersAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_ROUTE_MASTER
```sql
SELECT TRIM(rm.IRMMCU) AS BranchCode,
TRIM(rm.IRKITL) AS ItemNumber,
TRIM(rm.IRTRT) AS RoutingType,
rm.IROPSQ / 10.0 AS SequenceNumber,
TRIM(rm.IRURRF) AS FunctionCode,
TRIM(rm.IRMCU) AS WorkCenterCode,
rm.IREFFF AS StartDate_Date,
rm.IREFFT AS EndDate_Date,
rm.IRUPMJ AS LastUpdateDate,
rm.IRTDAY AS LastUpdateTime
FROM {ProductionSchema}.F3003 rm
WHERE TRIM(rm.IRKITL) IS NOT NULL
```
#### Business Rules
- SequenceNumber is IROPSQ/10 (stored as tenths)
- StartDate_Date and EndDate_Date are JDE integer format (converted in model)
---
## CMS Queries
### Requirement: CMS.GetMisData component
The system SHALL fetch Manufacturing Information System (MIS) data from CMS database.
#### Method Signature
```csharp
IAsyncEnumerable<MisData> GetMisDataAsync(DateTime? lastUpdateDT = null, CancellationToken ct = default)
```
#### Query: SQL_GET_MIS_DATA
```sql
SELECT DISTINCT
mis.P_PART_NUMBER AS ItemNumber,
mis.P_OPERATION_NUMBER AS SequenceNumber,
item.PITEM_ID AS MISNumber,
itemrev.PITEM_REVISION_ID AS RevID,
TRIM(mis.P_SITE) AS BranchCode,
zim_test_details.P_SEQ_NUMBER AS CharNumber,
zim_test_details.P_TEST_DESC AS TestDescription,
zim_test_details.P_SAMPL_TYPE AS SamplingType,
zim_test_details.P_SAMPL_VALUE AS SamplingValue,
zim_test_details.P_TOOLS AS ToolsGauges,
zim_test_details.P_WORK_INTR AS WorkInstructions,
Status.PNAME AS Status,
Status.PDATE_RELEASED AS ReleaseDate
FROM INFODBA.PITEM item
INNER JOIN INFODBA.PITEMREVISION itemrev ON (item.PUID = itemrev.RITEMS_TAGU)
INNER JOIN INFODBA.PRELEASE_STATUS_LIST listing ON (itemrev.PUID = listing.PUID)
INNER JOIN INFODBA.PRELEASESTATUS Status ON (listing.PVALU_0 = Status.PUID)
INNER JOIN INFODBA.PIMANRELATION imanrel ON (itemrev.PUID = imanrel.RPRIMARY_OBJECTU)
INNER JOIN INFODBA.PFORM form ON (imanrel.RSECONDARY_OBJECTU = form.PUID)
INNER JOIN INFODBA.PZIMMERMISDETAILS zim_mis ON (form.RDATA_FILEU = zim_mis.PUID)
INNER JOIN INFODBA.P_TEST_DETAILS test_details ON (zim_mis.PUID = test_details.PUID)
INNER JOIN INFODBA.P_PART_ASSOCIATION ppa ON (ppa.PUID = test_details.PUID)
INNER JOIN INFODBA.PMISDATAOBJECT mis ON (mis.PUID = ppa.PVALU_0)
INNER JOIN INFODBA.PZIMTESTDETAILS zim_test_details ON (test_details.PVALU_0 = zim_test_details.PUID)
WHERE Status.PNAME IN ('Current', 'BackLevel')
```
#### Business Rules
- Complex 10-table JOIN through CMS schema (INFODBA)
- Filters for Status 'Current' or 'BackLevel' only
- **Special timeout**: Uses `DataAccessOptions.MisDataTimeoutSeconds` (default 60000 seconds)
- **Data transformation**: ReleaseDate converted to local time
- Uses CMS connection via `CreateCmsConnectionAsync()`
#### Scenario: Fetch MIS data from CMS
- **WHEN** CMS connection is available and `GetMisDataAsync()` is called
- **THEN** 10-table JOIN returns Current and BackLevel MIS records as async stream
#### Scenario: Handle long-running query
- **WHEN** MIS data query may take extended time and query executes
- **THEN** configured timeout allows completion without error
---
## SQL Server Cache Operations
### Requirement: LotFinderDB.GetUserSearches component
The system SHALL retrieve searches for a specific user.
#### Method Signature
```csharp
Task<List<Search>> GetUserSearchesAsync(string userName, CancellationToken ct = default)
```
#### Query
```sql
SELECT s.ID, s.Name, s.Status, s.SubmitDT, s.StartDT, s.EndDT
FROM dbo.Search s
WHERE s.UserName = @userName
ORDER BY s.SubmitDT
```
#### Business Rules
- Throws `QueryException` on error (does not return empty list on error)
- Does not include Criteria or Results (lightweight query)
- Ordered by SubmitDT ascending
#### Scenario: Get user's search history
- **WHEN** user "jdoe" has 5 previous searches and `GetUserSearchesAsync("jdoe")` is called
- **THEN** 5 Search objects are returned without Criteria/Results (lightweight)
---
### Requirement: LotFinderDB.GetQueuedSearches component
The system SHALL retrieve all searches pending processing.
#### Method Signature
```csharp
Task<List<Search>> GetQueuedSearchesAsync(CancellationToken ct = default)
```
#### Query
```sql
SELECT s.ID, s.UserName, s.Name, s.Status, s.SubmitDT, s.StartDT, s.EndDT
FROM dbo.Search s
WHERE s.Status < 3
ORDER BY s.SubmitDT
```
#### Business Rules
- Status < 3 = New (0), Submitted (1), Started (2)
- Throws `QueryException` on error
- Ordered by SubmitDT ascending (FIFO processing)
#### Scenario: Worker polls for pending searches
- **WHEN** 3 searches with Status = Submitted exist and `GetQueuedSearchesAsync()` is called
- **THEN** 3 Search objects are returned ordered by SubmitDT (FIFO)
#### Scenario: Exclude completed searches
- **WHEN** 2 searches with Status = Ended (3) exist and `GetQueuedSearchesAsync()` is called
- **THEN** those searches are not included (Status < 3 filter)
---
### Requirement: LotFinderDB.GetSearch component
The system SHALL retrieve a single search by ID with criteria.
#### Method Signature
```csharp
Task<Search?> GetSearchAsync(int id, CancellationToken ct = default)
```
#### Query
```sql
SELECT s.UserName, s.Name, s.Status, s.SubmitDT, s.StartDT, s.EndDT,
s.Criteria as CriteriaJSON
FROM dbo.Search s
WHERE s.ID = @id
```
#### Business Rules
- Returns null if not found
- Throws `QueryException` on database error
- Deserializes CriteriaJSON to SearchCriteria object
- Sets ID property manually after query (not in SELECT)
---
### Requirement: LotFinderDB.GetSearchResults component
The system SHALL retrieve Excel results for a completed search.
#### Method Signature
```csharp
Task<byte[]?> GetSearchResultsAsync(int id, CancellationToken ct = default)
```
#### Query
```sql
SELECT s.Results
FROM dbo.Search AS s
WHERE s.ID = @id
```
#### Business Rules
- Returns null if not found
- Throws `QueryException` on database error
- Results is VARBINARY(MAX) containing Excel file
---
### Requirement: LotFinderDB.SubmitSearch component
The system SHALL create a new search request.
#### Method Signature
```csharp
Task<int> SubmitSearchAsync(Search search, CancellationToken ct = default)
```
#### Stored Procedure
```sql
EXEC dbo.usp_SubmitSearch
@p_UserName = ...,
@p_Name = ...,
@p_Criteria = ...,
@o_SearchID = ... OUTPUT
```
#### Business Rules
- Sets Status = Submitted and SubmitDT = DateTime.UtcNow before insert
- Serializes Criteria to JSON via ToJSON() method
- Throws `QueryException` on error
- Uses stored procedure with OUTPUT parameter for new ID
---
### Requirement: LotFinderDB.SearchItems component
The system SHALL provide autocomplete search for items by number or description.
#### Method Signature
```csharp
Task<List<Item>> SearchItemsAsync(string filter, CancellationToken ct = default)
```
#### Query
```sql
SELECT TOP 25 i.ShortItemNumber, i.ItemNumber, i.Description, i.LastUpdateDT
FROM dbo.Item AS i
WHERE i.ItemNumber LIKE '%' + @filter + '%' OR
i.Description LIKE '%' + @filter + '%'
ORDER BY i.ItemNumber
```
#### Business Rules
- Returns top 25 matches
- Case-insensitive LIKE search on both ItemNumber and Description
- Throws `QueryException` on error
---
### Requirement: LotFinderDB.LookupItems component
The system SHALL provide batch lookup of items by exact item numbers.
#### Method Signature
```csharp
Task<List<Item>> LookupItemsAsync(List<string> itemNumbers, CancellationToken ct = default)
```
#### Query
```sql
SELECT i.ShortItemNumber, i.ItemNumber, i.Description, i.LastUpdateDT
FROM dbo.Item AS i
INNER JOIN @itemNumbers AS i2 ON (i.ItemNumber = i2.ItemNumber)
ORDER BY i.ItemNumber
```
#### Business Rules
- Uses table-valued parameter `ItemNumberFilterParameter`
- Throws `QueryException` on error
---
### Requirement: LotFinderDB.LookupWorkorders component
The system SHALL provide batch lookup of work orders by work order numbers.
#### Method Signature
```csharp
Task<List<WorkOrder>> LookupWorkordersAsync(List<long> workorderNumbers, CancellationToken ct = default)
```
#### Query
```sql
SELECT *
FROM dbo.WorkOrder AS wo
INNER JOIN @workOrderNumbers wo2 ON (wo.WorkOrderNumber = wo2.WorkOrderNumber)
```
#### Business Rules
- Uses table-valued parameter `WorkOrderFilterParameter`
- SELECT * returns all columns from WorkOrder view
- Throws `QueryException` on error
---
### Requirement: LotFinderDB.SearchWorkCenters / LookupWorkCenters component
The system SHALL provide autocomplete and batch lookup for work centers.
#### SearchWorkCenters Query
```sql
SELECT TOP 25 wc.Code, wc.Description, wc.LastUpdateDT
FROM dbo.WorkCenter AS wc
WHERE wc.Code LIKE '%' + @filter + '%' OR
wc.Description LIKE '%' + @filter + '%'
ORDER BY wc.Code
```
#### LookupWorkCenters Query
```sql
SELECT wc.Code, wc.Description, wc.LastUpdateDT
FROM dbo.WorkCenter AS wc
INNER JOIN @workCenterCodes wc2 ON (wc.Code = wc2.Code)
ORDER BY wc.Code
```
---
### Requirement: LotFinderDB.SearchProfitCenters / LookupProfitCenters component
The system SHALL provide autocomplete and batch lookup for profit centers.
#### SearchProfitCenters Query
```sql
SELECT TOP 25 pc.Code, pc.Description, pc.LastUpdateDT
FROM dbo.ProfitCenter AS pc
WHERE pc.Code LIKE '%' + @filter + '%' OR
pc.Description LIKE '%' + @filter + '%'
ORDER BY pc.Code
```
#### LookupProfitCenters Query
```sql
SELECT pc.Code, pc.Description, pc.LastUpdateDT
FROM dbo.ProfitCenter AS pc
INNER JOIN @profitCenterCodes AS pc2 ON (pc.Code = pc2.Code)
ORDER BY pc.Code
```
---
### Requirement: LotFinderDB.SearchUsers / LookupUsers component
The system SHALL provide autocomplete and batch lookup for JDE users.
#### SearchUsers Query
```sql
SELECT TOP 25 u.AddressNumber, COALESCE(u.UserID,' ') AS UserID, u.FullName, u.LastUpdateDT
FROM dbo.JdeUser AS u
WHERE u.UserID LIKE '%' + @filter + '%' OR
u.FullName LIKE '%' + @filter + '%' OR
CAST(u.AddressNumber AS VARCHAR(10)) LIKE '%' + @filter + '%'
ORDER BY u.UserID, u.FullName
```
#### LookupUsers Query
```sql
SELECT u.AddressNumber, u.UserID, u.FullName, u.LastUpdateDT
FROM dbo.JdeUser AS u
INNER JOIN @userIDs u2 ON (u.UserID = u2.UserName OR CAST(u.AddressNumber AS VARCHAR(20)) = u2.UserName)
ORDER BY u.UserID
```
#### Business Rules
- SearchUsers: COALESCE UserID to space if null
- LookupUsers: Matches by UserID OR AddressNumber (cast to string)
- Uses table-valued parameter `OperatorFilterParameter`
---
### Requirement: LotFinderDB.LookupLots component
The system SHALL provide batch lookup of lots by lot number and item number.
#### Method Signature
```csharp
Task<List<Lot>> LookupLotsAsync(List<LotViewModel> lots, CancellationToken ct = default)
```
#### Query
```sql
SELECT DISTINCT l.LotNumber, l.BranchCode, l.ShortItemNumber, l.ItemNumber,
l.SupplierCode, l.LastUpdateDT
FROM dbo.Lot AS l
INNER JOIN @lotNumbers ln ON (l.LotNumber = ln.ComponentLotNumber AND
((l.ItemNumber IS NULL AND ln.ItemNumber IS NULL) OR l.ItemNumber = ln.ItemNumber))
```
#### Business Rules
- Uses table-valued parameter `ComponentLotFilterParameter`
- Matches on LotNumber AND ItemNumber (with NULL handling)
- DISTINCT to eliminate duplicates
---
### Requirement: LotFinderDB.PostProcessMisData component
The system SHALL post-process imported MIS data to set obsolete dates.
#### Method Signature
```csharp
Task PostProcessMisDataAsync(CancellationToken ct = default)
```
#### Query
```sql
SET ANSI_WARNINGS OFF;
-- Set ObsoleteDate for Current status when BackLevel exists
WITH cte AS (
SELECT md.MisNumber, md.RevID, md.Status, MIN(md.ReleaseDate) Released
FROM dbo.MisData AS md
GROUP BY md.MisNumber, md.RevID, md.Status
)
UPDATE dbo.MisData
SET ObsoleteDate = bl.Released
FROM cte bl
WHERE MisData.MisNumber = bl.MisNumber AND
MisData.RevID = bl.RevID AND
MisData.Status = 'Current' AND
bl.Status = 'BackLevel';
-- Set ObsoleteDate for remaining records based on next revision
WITH cte AS (
SELECT md.MisNumber, md.RevID, md.Status, MIN(md.ReleaseDate) Released
FROM dbo.MisData AS md
GROUP BY md.MisNumber, md.RevID, md.Status
)
UPDATE dbo.MisData
SET ObsoleteDate = (SELECT TOP 1 nl.Released
FROM cte nl
WHERE MisData.MisNumber = nl.MisNumber AND
MisData.RevID < nl.RevID AND
MisData.Status = nl.Status
ORDER BY nl.RevID)
WHERE ObsoleteDate IS NULL;
ALTER INDEX [PK_MisData] ON [dbo].[MisData] REBUILD;
```
#### Business Rules
- Two-phase update: first BackLevel, then next revision
- Rebuilds primary key index after updates
- Disables ANSI_WARNINGS for aggregate operations
---
### Requirement: LotFinderDB.GetLastDataUpdates component
The system SHALL retrieve most recent successful update per table/type.
#### Method Signature
```csharp
Task<List<DataUpdate>> GetLastDataUpdatesAsync(CancellationToken ct = default)
```
#### Query
```sql
WITH DU_CTE AS (
SELECT du.*,
ROW_NUMBER() OVER (PARTITION BY du.TableName, du.UpdateType ORDER BY du.StartDT DESC) RN
FROM dbo.DataUpdate AS du
)
SELECT cte.SourceSystem, cte.SourceData, cte.TableName, cte.StartDT, cte.EndDT,
cte.UpdateType, cte.WasSuccessful, cte.NumberRecords
FROM DU_CTE cte
WHERE cte.RN = 1
```
#### Business Rules
- Returns latest record per (TableName, UpdateType) combination
- Used to determine incremental update windows
---
### Requirement: LotFinderDB.GetTableSpec component
The system SHALL retrieve table schema for dynamic operations.
#### Method Signature
```csharp
Task<TableSpec> GetTableSpecAsync(string tableName, CancellationToken ct = default)
```
#### Queries
```sql
-- Get columns
SELECT c.name AS Name,
CASE t2.name
WHEN 'varchar' THEN 'VARCHAR(' + CAST(c.max_length AS VARCHAR(10)) + ')'
WHEN 'decimal' THEN 'DECIMAL(' + CAST(c.precision AS VARCHAR(4)) + ',' + CAST(c.scale AS VARCHAR(4)) + ')'
ELSE UPPER(t2.name)
END AS Definition
FROM sys.columns c
INNER JOIN sys.types AS t2 ON (c.system_type_id = t2.system_type_id)
INNER JOIN sys.tables t ON (c.object_id = t.object_id)
WHERE t.name = @name
ORDER BY c.column_id
-- Get primary key columns
SELECT COLUMN_NAME AS Name
FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE
WHERE OBJECTPROPERTY(OBJECT_ID(CONSTRAINT_SCHEMA + '.' + QUOTENAME(CONSTRAINT_NAME)), 'IsPrimaryKey') = 1 AND
TABLE_NAME = @name
ORDER BY ORDINAL_POSITION
```
#### Business Rules
- Used for dynamic temp table creation during bulk operations
- Maps SQL Server types to DDL definitions
---
### Requirement: LotFinderDB.RebuildIndices component
The system SHALL rebuild all indices on a table with SQL injection protection.
#### Method Signature
```csharp
Task RebuildIndicesAsync(string tableName, CancellationToken ct = default)
```
#### Query
```sql
ALTER INDEX ALL ON [{tableName}] REBUILD WITH (FILLFACTOR = 95);
```
#### Valid Table Names (Whitelist)
The following table names are valid for index rebuilding:
- `Branch`
- `DataUpdate`
- `FunctionCode`
- `Item`
- `JdeUser`
- `Lot`
- `LotLocation`
- `LotUsage_Curr`
- `LotUsage_Hist`
- `MisData`
- `OrgHierarchy`
- `ProfitCenter`
- `RouteMaster`
- `Search`
- `StatusCode`
- `WorkCenter`
- `WorkOrder_Curr`
- `WorkOrder_Hist`
- `WorkOrderComponent_Curr`
- `WorkOrderComponent_Hist`
- `WorkOrderRouting`
- `WorkOrderStep_Curr`
- `WorkOrderStep_Hist`
- `WorkOrderTime_Curr`
- `WorkOrderTime_Hist`
#### Business Rules
- Table name MUST be validated against whitelist before execution
- Throws `ArgumentException` if table name is not in whitelist
- FILLFACTOR = 95 leaves 5% free space for inserts
- Uses `DataAccessOptions.RebuildIndexTimeoutSeconds` for timeout
#### Scenario: Valid table name
- **WHEN** `RebuildIndicesAsync("WorkOrder_Curr")` is called
- **THEN** index rebuild executes successfully because table name is in whitelist
#### Scenario: Invalid table name (SQL injection attempt)
- **WHEN** `RebuildIndicesAsync("WorkOrder]; DROP TABLE Search;--")` is called
- **THEN** `ArgumentException` is thrown with message "Invalid table name"
---
### Requirement: LotFinderDB.GenerateTableParameter component
The system SHALL create DataTable for table-valued parameters.
#### Method Signature
```csharp
DataTable GenerateTableParameter<T>(List<T> keys)
```
#### Business Rules
- Creates single-column DataTable with column named "Key"
- Column type matches generic type T
- Used for simple single-value TVPs
---
## Domain Model Cross-Reference
| Repository Method | Domain Model | Cache Table |
|-------------------|--------------|-------------|
| JDE.GetWorkOrders | WorkOrder | WorkOrder_Curr/WorkOrder_Hist |
| JDE.GetWorkOrderSteps | WorkOrderStep | WorkOrderStep_Curr/WorkOrderStep_Hist |
| JDE.GetWorkOrderTimes | WorkOrderTime | WorkOrderTime_Curr/WorkOrderTime_Hist |
| JDE.GetWorkOrderRoutings | WorkOrderRouting | WorkOrderRouting |
| JDE.GetWorkOrderComponents | WorkOrderComponent | WorkOrderComponent_Curr/WorkOrderComponent_Hist |
| JDE.GetLots | Lot | Lot |
| JDE.GetLotUsages | LotUsage | LotUsage_Curr/LotUsage_Hist |
| JDE.GetLotLocations | LotLocation | LotLocation |
| JDE.GetItems | Item | Item |
| JDE.GetUsers | JdeUser | JdeUser |
| JDE.GetBranches | Branch | Branch |
| JDE.GetProfitCenters | ProfitCenter | ProfitCenter |
| JDE.GetWorkCenters | WorkCenter | WorkCenter |
| JDE.GetStatusCodes | StatusCode | StatusCode |
| JDE.GetFunctionCodes | FunctionCode | FunctionCode |
| JDE.GetOrgHierarchy | OrgHierarchy | OrgHierarchy |
| JDE.GetRouteMasters | RouteMaster | RouteMaster |
| CMS.GetMisData | MisData | MisData |
| LotFinderDB.GetSearch | Search | Search |
| LotFinderDB.GetLastDataUpdates | DataUpdate | DataUpdate |
---
## JDE Table Reference
| JDE Table | Schema | Description |
|-----------|--------|-------------|
| F4801 | PRODDTA / ARCDTAPD | Work Order Master |
| F3112 | PRODDTA / ARCDTAPD | Work Order Routing (Steps) |
| F31122 | PRODDTA / ARCDTAPD | Work Order Time Transactions |
| F3112Z1 | PRODDTA | Work Order Routing Transactions |
| F3111 | PRODDTA / ARCDTAPD | Work Order Parts List (Components) |
| F4108 | PRODDTA | Lot Master |
| F4111 | PRODDTA / ARCDTAPD | Item Ledger (Cardex) |
| F41021 | JDESTAGE | Item Location |
| F4101 | PRODDTA | Item Master |
| F0101 | PRODDTA | Address Book Master |
| F0092 | SY920 | User Profile |
| F0006 | PRODDTA | Business Unit Master |
| F0005 | JDESTAGE | User Defined Codes |
| F00192 | PRODDTA | Media Object (Function Codes) |
| F30006 | PRODDTA | Work Center Master |
| F3003 | PRODDTA | Routing Master |
---
## Migration Notes
| Legacy Pattern | New Pattern | Rationale |
|----------------|-------------|-----------|
| `System.Data.SqlClient` | `Microsoft.Data.SqlClient` | Modern SQL Server client with better performance and security |
| `Oracle.ManagedDataAccess.Client` + `DDTek.Oracle` | `Oracle.ManagedDataAccess.Core` | Consolidated to single Oracle driver for .NET Core/.NET 5+ |
| Static partial classes | Instance-based repositories with interfaces | Testability, dependency injection, loose coupling |
| External .sql files | Embedded resources or `const string` fields | Deployment simplicity, no runtime file I/O |
| `EncryptionHelper.Decrypt` with hardcoded key | Azure Key Vault or .NET Secret Manager | Industry standard secrets management |
| `buffered: false` Dapper | `QueryUnbufferedAsync` with `IAsyncEnumerable<T>` | Native async streaming, memory efficiency |
| `string.Format` SQL (RebuildIndices) | Table name whitelist validation | SQL injection prevention |
| Synchronous methods | Async/await with `CancellationToken` | Scalability, responsiveness, cancellation support |
| NLog static logger | `ILogger<T>` injected + `BeginScope()` for context | Modern structured logging with DI |
| Exception wrapping with generic message | Custom typed exceptions (`DataAccessException`, `ConnectionException`, `QueryException`, `DataAccessTimeoutException`) | Better error handling, debugging, and error classification |
| `Config.*CS` static properties | `IConfiguration["ConnectionStrings:*"]` | Configuration abstraction, environment support |
| Hardcoded timeout constants | `IOptions<DataAccessOptions>` with configurable values | Centralized configuration, runtime flexibility |
| DataTable for TVPs | Keep DataTable (Dapper compatible) | Dapper TVP support, no breaking change needed |
| Return null/empty on error | Throw typed exceptions | Consistent error handling, explicit failure semantics |
---
## Resolved Design Decisions
The following questions from the legacy analysis have been resolved with architectural decisions:
1. **Oracle driver consolidation**: Consolidated to `Oracle.ManagedDataAccess.Core` for all Oracle connections (JDE, JDE Stage, CMS). Single driver simplifies deployment and maintenance.
2. **External SQL files vs embedded resources**: SQL queries stored as embedded resources or compile-time constants. Eliminates runtime file I/O and simplifies deployment.
3. **Password encryption**: Replaced custom `EncryptionHelper.Decrypt` with Azure Key Vault or .NET Secret Manager. Industry-standard approach with proper key management.
4. **Streaming vs buffered**: All large dataset queries use `IAsyncEnumerable<T>` with `QueryUnbufferedAsync`. Streaming pattern preserved for memory efficiency.
5. **SQL injection in RebuildIndices**: Implemented table name whitelist validation. Only explicitly listed table names are allowed.
6. **Error handling inconsistency**: Standardized on throwing typed exceptions. No methods return null/empty on error conditions.
7. **GetUsers incremental bug**: Documented as intentional behavior (full sync always for users). No incremental filtering parameter.
8. **Special timeouts**: Made configurable via `DataAccessOptions`. LotUsage and MisData have separate configurable timeout values.
9. **Schema placeholders**: Schema names (`ProductionSchema`, `ArchiveSchema`, `StageSchema`) are configurable in `DataAccessOptions` for environment-specific deployments.
10. **StatusCode/LotLocation connection**: These queries use `CreateJdeStageConnectionAsync()` for JDESTAGE schema views. Separate connection configuration allows different credentials if needed.
11. **CMS connection visibility**: All connection methods are on `IDbConnectionFactory` interface, providing consistent public access pattern.
12. **Async support**: Full async support with `Task<T>` and `IAsyncEnumerable<T>` return types. All methods accept `CancellationToken`.
13. **Connection pooling**: Connection pooling is configured at the connection string level. `IDbConnectionFactory` opens connections; callers are responsible for disposal.
14. **Transaction support**: Transactions handled at the service layer when needed. Repository methods are single-unit operations.
15. **F41021_VIEW vs F41021**: JDESTAGE views provide flattened/optimized access. Original design preserved; uses JDE Stage connection.
16. **Archive methods**: Explicitly defined as separate interface methods (`GetWorkOrdersArchiveAsync`, etc.) for clarity.
17. **_FILTERED query variants**: Filtered queries use same method with optional `lastUpdateDT` parameter. Null = full sync, non-null = incremental.
---
## Codex Review Findings
The following observations were noted during legacy code analysis:
1. **Mixed driver usage**: Legacy code uses both `Oracle.ManagedDataAccess` and `DDTek.Oracle`. New design consolidates to single driver.
2. **Password decryption location**: Legacy decrypts in `Config.*CS` properties. New design retrieves secrets from secure store at connection time.
3. **LotFinderDB timeout comment**: Legacy comment says "ms" but value is seconds. New design uses explicit `TimeoutSeconds` naming.
4. **Query repository SetQuery unused**: Legacy `SetQuery` method writes to disk but appears unused. Not carried forward to new design.
5. **Inconsistent error patterns**: Some legacy methods return empty/null on error, others throw. New design standardizes on exceptions.
Reference in New Issue View Git Blame Copy Permalink