Initial commit: JDE Scoping Tool migration project
Set up repository with legacy .NET Framework 4.8 source (OLD/), new .NET 10 Blazor solution (NEW/), OpenSpec specifications, documentation, and project configuration.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,376 @@
|
||||
# Data Access Design
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the architecture and implementation approach for the data access layer, including repository interfaces, connection factory, exception handling, and service registration.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Repository Pattern
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Service Layer │
|
||||
│ (SearchProcessor, DataSyncService, etc.) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ILotFinder │ │IJde │ │ICms │
|
||||
│Repository │ │Repository │ │Repository │
|
||||
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
|
||||
│ │ │
|
||||
┌──────▼───────┐ ┌──────▼───────┐ ┌──────▼───────┐
|
||||
│LotFinder │ │Jde │ │Cms │
|
||||
│Repository │ │Repository │ │Repository │
|
||||
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
|
||||
│ │ │
|
||||
└────────────────┼────────────────┘
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ IDbConnectionFactory │
|
||||
└─────────────┬───────────┘
|
||||
│
|
||||
┌─────────────────────┼─────────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
|
||||
│ SqlConnection │ │OracleConnection│ │OracleConnection│
|
||||
│ (LotFinderDB) │ │ (JDE/Stage) │ │ (CMS) │
|
||||
└───────────────┘ └───────────────┘ └───────────────┘
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
NEW/src/JdeScoping.DataAccess/
|
||||
├── Exceptions/
|
||||
│ ├── DataAccessException.cs
|
||||
│ ├── ConnectionException.cs
|
||||
│ ├── QueryException.cs
|
||||
│ └── DataAccessTimeoutException.cs
|
||||
├── Interfaces/
|
||||
│ ├── IDbConnectionFactory.cs
|
||||
│ ├── ILotFinderRepository.cs
|
||||
│ ├── IJdeRepository.cs
|
||||
│ └── ICmsRepository.cs
|
||||
├── Repositories/
|
||||
│ ├── LotFinderRepository.cs
|
||||
│ ├── JdeRepository.cs
|
||||
│ └── CmsRepository.cs
|
||||
├── Queries/
|
||||
│ ├── LotFinderQueries.cs (const string SQL statements)
|
||||
│ ├── JdeQueries.cs (const string SQL statements)
|
||||
│ └── CmsQueries.cs (const string SQL statements)
|
||||
├── Configuration/
|
||||
│ └── DataAccessOptions.cs
|
||||
├── DbConnectionFactory.cs
|
||||
├── ServiceCollectionExtensions.cs
|
||||
└── JdeScoping.DataAccess.csproj
|
||||
```
|
||||
|
||||
## Connection Factory
|
||||
|
||||
### IDbConnectionFactory Interface
|
||||
|
||||
```csharp
|
||||
public interface IDbConnectionFactory
|
||||
{
|
||||
Task<SqlConnection> CreateLotFinderConnectionAsync(CancellationToken ct = default);
|
||||
Task<OracleConnection> CreateJdeConnectionAsync(CancellationToken ct = default);
|
||||
Task<OracleConnection> CreateJdeStageConnectionAsync(CancellationToken ct = default);
|
||||
Task<OracleConnection> CreateCmsConnectionAsync(CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
### Implementation Details
|
||||
|
||||
- Registered as **singleton** (stateless, creates new connections)
|
||||
- Connection strings read from `IConfiguration["ConnectionStrings:*"]`
|
||||
- Secrets retrieved from .NET Secret Manager (local) or Azure Key Vault (production)
|
||||
- Connections opened asynchronously before returning
|
||||
- Caller responsible for disposing returned connections
|
||||
|
||||
### Connection String Keys
|
||||
|
||||
| Key | Database | Driver |
|
||||
|-----|----------|--------|
|
||||
| `ConnectionStrings:LotFinderDB` | SQL Server cache | Microsoft.Data.SqlClient |
|
||||
| `ConnectionStrings:JDE` | JDE Oracle (PRODDTA) | Oracle.ManagedDataAccess.Core |
|
||||
| `ConnectionStrings:JDEStage` | JDE Oracle (JDESTAGE) | Oracle.ManagedDataAccess.Core |
|
||||
| `ConnectionStrings:CMS` | CMS Oracle (INFODBA) | Oracle.ManagedDataAccess.Core |
|
||||
|
||||
## Repository Interfaces
|
||||
|
||||
### Registration Lifetimes
|
||||
|
||||
| Interface | Lifetime | Rationale |
|
||||
|-----------|----------|-----------|
|
||||
| `IDbConnectionFactory` | Singleton | Stateless, creates new connections |
|
||||
| `ILotFinderRepository` | Scoped | Per-request, uses scoped DbContext pattern |
|
||||
| `IJdeRepository` | Scoped | Per-request, creates connections as needed |
|
||||
| `ICmsRepository` | Scoped | Per-request, creates connections as needed |
|
||||
|
||||
### Constructor Dependencies
|
||||
|
||||
All repository implementations receive:
|
||||
- `IDbConnectionFactory` - For database connections
|
||||
- `ILogger<T>` - For structured logging
|
||||
- `IOptions<DataAccessOptions>` - For configurable timeouts and schemas
|
||||
|
||||
## Async Streaming Pattern
|
||||
|
||||
### IAsyncEnumerable for Large Datasets
|
||||
|
||||
JDE and CMS repositories return `IAsyncEnumerable<T>` for all collection queries:
|
||||
|
||||
```csharp
|
||||
public async IAsyncEnumerable<WorkOrder> GetWorkOrdersAsync(
|
||||
DateTime? lastUpdateDT = null,
|
||||
[EnumeratorCancellation] CancellationToken ct = default)
|
||||
{
|
||||
await using var connection = await _connectionFactory.CreateJdeConnectionAsync(ct);
|
||||
|
||||
var sql = lastUpdateDT.HasValue
|
||||
? JdeQueries.SQL_GET_WORKORDERS_FILTERED
|
||||
: JdeQueries.SQL_GET_WORKORDERS;
|
||||
|
||||
var parameters = BuildWorkOrderParameters(lastUpdateDT);
|
||||
|
||||
await foreach (var workOrder in connection.QueryUnbufferedAsync<WorkOrder>(
|
||||
sql, parameters, commandTimeout: _options.Value.DefaultTimeoutSeconds)
|
||||
.WithCancellation(ct))
|
||||
{
|
||||
yield return workOrder;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Benefits
|
||||
|
||||
- Memory efficient: rows streamed one at a time
|
||||
- Cancellation support: stops iteration on cancellation
|
||||
- Backpressure: consumer controls iteration speed
|
||||
- Compatible with `await foreach` syntax
|
||||
|
||||
## Query Management
|
||||
|
||||
### SQL Query Storage
|
||||
|
||||
SQL queries stored as compile-time constants in static classes:
|
||||
|
||||
```csharp
|
||||
public static class JdeQueries
|
||||
{
|
||||
public const string SQL_GET_WORKORDERS = @"
|
||||
SELECT wo.WADOCO AS WorkOrderNumber,
|
||||
TRIM(wo.WAMMCU) AS BranchCode,
|
||||
-- ... rest of query
|
||||
FROM {ProductionSchema}.F4801 wo";
|
||||
|
||||
public const string SQL_GET_WORKORDERS_FILTERED = SQL_GET_WORKORDERS + @"
|
||||
WHERE (wo.WAUPMJ > :dateUpdated OR
|
||||
(wo.WAUPMJ = :dateUpdated AND wo.WATDAY >= :timeUpdated))";
|
||||
}
|
||||
```
|
||||
|
||||
### Schema Placeholder Replacement
|
||||
|
||||
Schema names replaced at runtime from `DataAccessOptions`:
|
||||
|
||||
```csharp
|
||||
private string ApplySchemaPlaceholders(string sql)
|
||||
{
|
||||
return sql
|
||||
.Replace("{ProductionSchema}", _options.Value.ProductionSchema)
|
||||
.Replace("{ArchiveSchema}", _options.Value.ArchiveSchema)
|
||||
.Replace("{StageSchema}", _options.Value.StageSchema);
|
||||
}
|
||||
```
|
||||
|
||||
## Exception Handling
|
||||
|
||||
### Exception Hierarchy
|
||||
|
||||
```
|
||||
Exception
|
||||
└── DataAccessException (base for all data access errors)
|
||||
├── ConnectionException (connection failures)
|
||||
├── QueryException (query execution failures)
|
||||
└── DataAccessTimeoutException (timeout errors)
|
||||
```
|
||||
|
||||
### Exception Properties
|
||||
|
||||
```csharp
|
||||
public class DataAccessException : Exception
|
||||
{
|
||||
public string? Operation { get; } // Method name (e.g., "GetWorkOrdersAsync")
|
||||
public string? Repository { get; } // Repository name (e.g., "JdeRepository")
|
||||
}
|
||||
|
||||
public class ConnectionException : DataAccessException
|
||||
{
|
||||
public string? DataSource { get; } // Database identifier (e.g., "JDE", "CMS")
|
||||
}
|
||||
|
||||
public class QueryException : DataAccessException
|
||||
{
|
||||
public string? QueryName { get; } // Query identifier (e.g., "SQL_GET_WORKORDERS")
|
||||
}
|
||||
|
||||
public class DataAccessTimeoutException : DataAccessException
|
||||
{
|
||||
public int TimeoutSeconds { get; } // Configured timeout value
|
||||
}
|
||||
```
|
||||
|
||||
### Logging Pattern
|
||||
|
||||
All exceptions logged at throw site with scope context:
|
||||
|
||||
```csharp
|
||||
try
|
||||
{
|
||||
// Execute query
|
||||
}
|
||||
catch (OracleException ex) when (ex.Number == 1017) // Invalid credentials
|
||||
{
|
||||
using (_logger.BeginScope(new Dictionary<string, object>
|
||||
{
|
||||
["DataSource"] = "JDE",
|
||||
["Operation"] = "GetWorkOrdersAsync"
|
||||
}))
|
||||
{
|
||||
_logger.LogError(ex, "Failed to connect to JDE Oracle database");
|
||||
}
|
||||
throw new ConnectionException("JDE: Failed to connect to database", "JDE", ex);
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### DataAccessOptions Class
|
||||
|
||||
```csharp
|
||||
public class DataAccessOptions
|
||||
{
|
||||
public const string SectionName = "DataAccess";
|
||||
|
||||
public int DefaultTimeoutSeconds { get; set; } = 600;
|
||||
public int LotUsageTimeoutSeconds { get; set; } = 999999;
|
||||
public int MisDataTimeoutSeconds { get; set; } = 60000;
|
||||
public int RebuildIndexTimeoutSeconds { get; set; } = 600;
|
||||
public string ProductionSchema { get; set; } = "PRODDTA";
|
||||
public string ArchiveSchema { get; set; } = "ARCDTAPD";
|
||||
public string StageSchema { get; set; } = "JDESTAGE";
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Binding
|
||||
|
||||
```json
|
||||
{
|
||||
"DataAccess": {
|
||||
"DefaultTimeoutSeconds": 600,
|
||||
"LotUsageTimeoutSeconds": 999999,
|
||||
"MisDataTimeoutSeconds": 60000,
|
||||
"RebuildIndexTimeoutSeconds": 600,
|
||||
"ProductionSchema": "PRODDTA",
|
||||
"ArchiveSchema": "ARCDTAPD",
|
||||
"StageSchema": "JDESTAGE"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Service Registration
|
||||
|
||||
### AddDataAccess Extension Method
|
||||
|
||||
```csharp
|
||||
public static class ServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddDataAccess(
|
||||
this IServiceCollection services,
|
||||
IConfiguration configuration)
|
||||
{
|
||||
// Bind options
|
||||
services.Configure<DataAccessOptions>(
|
||||
configuration.GetSection(DataAccessOptions.SectionName));
|
||||
|
||||
// Register connection factory (singleton)
|
||||
services.AddSingleton<IDbConnectionFactory, DbConnectionFactory>();
|
||||
|
||||
// Register repositories (scoped)
|
||||
services.AddScoped<ILotFinderRepository, LotFinderRepository>();
|
||||
services.AddScoped<IJdeRepository, JdeRepository>();
|
||||
services.AddScoped<ICmsRepository, CmsRepository>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## SQL Injection Prevention
|
||||
|
||||
### RebuildIndicesAsync Whitelist
|
||||
|
||||
Table names validated against explicit whitelist:
|
||||
|
||||
```csharp
|
||||
private static readonly HashSet<string> ValidTableNames = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
"Branch", "DataUpdate", "FunctionCode", "Item", "JdeUser",
|
||||
"Lot", "LotLocation", "LotUsage_Curr", "LotUsage_Hist",
|
||||
"MisData", "OrgHierarchy", "ProfitCenter", "RouteMaster",
|
||||
"Search", "StatusCode", "WorkCenter",
|
||||
"WorkOrder_Curr", "WorkOrder_Hist",
|
||||
"WorkOrderComponent_Curr", "WorkOrderComponent_Hist",
|
||||
"WorkOrderRouting",
|
||||
"WorkOrderStep_Curr", "WorkOrderStep_Hist",
|
||||
"WorkOrderTime_Curr", "WorkOrderTime_Hist"
|
||||
};
|
||||
|
||||
public async Task RebuildIndicesAsync(string tableName, CancellationToken ct = default)
|
||||
{
|
||||
if (!ValidTableNames.Contains(tableName))
|
||||
{
|
||||
throw new ArgumentException($"Invalid table name: {tableName}", nameof(tableName));
|
||||
}
|
||||
|
||||
await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
|
||||
var sql = $"ALTER INDEX ALL ON [{tableName}] REBUILD WITH (FILLFACTOR = 95)";
|
||||
await connection.ExecuteAsync(sql, commandTimeout: _options.Value.RebuildIndexTimeoutSeconds);
|
||||
}
|
||||
```
|
||||
|
||||
## NuGet Dependencies
|
||||
|
||||
### Required Packages
|
||||
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.*" />
|
||||
<PackageReference Include="Oracle.ManagedDataAccess.Core" Version="23.4.*" />
|
||||
<PackageReference Include="Dapper" Version="2.1.*" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="9.0.*" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.*" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.*" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- Mock `IDbConnectionFactory` to return mock connections
|
||||
- Use in-memory test data for query result mapping
|
||||
- Verify exception handling scenarios
|
||||
- Test cancellation token propagation
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- Use Docker containers for SQL Server and Oracle
|
||||
- Test actual query execution
|
||||
- Verify streaming behavior for large datasets
|
||||
- Test connection pooling under load
|
||||
@@ -0,0 +1,69 @@
|
||||
# Implement Data Access
|
||||
|
||||
## Summary
|
||||
|
||||
Implement the data access layer with repository interfaces and implementations for accessing SQL Server (LotFinderDB), JDE Oracle, and CMS Oracle databases. This provides the foundation for all data operations in the migrated application.
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
|
||||
- `IDbConnectionFactory` interface and `DbConnectionFactory` implementation
|
||||
- `ILotFinderRepository` interface with all SQL Server cache methods
|
||||
- `IJdeRepository` interface with all JDE Oracle query methods
|
||||
- `ICmsRepository` interface with CMS MIS data methods
|
||||
- `LotFinderRepository`, `JdeRepository`, `CmsRepository` implementations
|
||||
- `DataAccessOptions` configuration class
|
||||
- Custom exception hierarchy (`DataAccessException`, `ConnectionException`, `QueryException`, `DataAccessTimeoutException`)
|
||||
- `AddDataAccess` service registration extension method
|
||||
- SQL queries as embedded resources or compile-time constants
|
||||
- Unit tests for repository methods
|
||||
|
||||
### Out of Scope
|
||||
|
||||
- Database schema changes (handled by migrate-database-schema)
|
||||
- Data sync scheduling (Phase 5: data-sync)
|
||||
- Search processing logic (Phase 6: search-processing)
|
||||
- Azure Key Vault integration (will use .NET Secret Manager for local dev)
|
||||
|
||||
## Motivation
|
||||
|
||||
The legacy data access layer uses static partial classes which are difficult to test and tightly coupled. The new design provides:
|
||||
- Interface-based repositories for dependency injection and testability
|
||||
- Connection factory abstraction for consistent connection management
|
||||
- Async-first design with `IAsyncEnumerable<T>` for memory-efficient streaming
|
||||
- Typed exceptions for consistent error handling
|
||||
- Configurable timeouts via options pattern
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. All three repository interfaces defined with methods matching the spec
|
||||
2. `IDbConnectionFactory` provides connections for all four database connections (LotFinderDB, JDE, JDE Stage, CMS)
|
||||
3. All repository implementations use Dapper for query execution
|
||||
4. JDE/CMS streaming queries use `IAsyncEnumerable<T>` with `QueryUnbufferedAsync`
|
||||
5. All methods accept `CancellationToken` parameter
|
||||
6. Custom exceptions thrown on errors (never return null/empty on error)
|
||||
7. `AddDataAccess` extension method registers all services with appropriate lifetimes
|
||||
8. SQL injection prevented via whitelist validation in `RebuildIndicesAsync`
|
||||
9. Unit tests pass with mocked dependencies
|
||||
10. `openspec validate implement-data-access --strict` passes
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `migrate-database-schema` - Database schema must exist for repository queries
|
||||
- NuGet packages: `Microsoft.Data.SqlClient`, `Oracle.ManagedDataAccess.Core`, `Dapper`
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Oracle driver compatibility | Test early with Oracle.ManagedDataAccess.Core against target databases |
|
||||
| Query translation errors | Copy SQL exactly from legacy, validate with Codex MCP review |
|
||||
| Streaming memory issues | Use `QueryUnbufferedAsync` for all large result sets |
|
||||
| Connection pooling misconfiguration | Use default ADO.NET pooling, document connection string settings |
|
||||
|
||||
## Related Specs
|
||||
|
||||
- `data-access` - All repository interface and method definitions
|
||||
- `domain-models` - Entity types returned by repositories
|
||||
- `database-schema` - SQL Server tables accessed by LotFinderRepository
|
||||
@@ -0,0 +1,324 @@
|
||||
# Data Access - Implementation Patterns
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Connection factory pattern
|
||||
|
||||
The system SHALL implement `IDbConnectionFactory` to provide database connections via dependency injection.
|
||||
|
||||
#### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
public class DbConnectionFactory : IDbConnectionFactory
|
||||
{
|
||||
private readonly IConfiguration _configuration;
|
||||
private readonly ILogger<DbConnectionFactory> _logger;
|
||||
|
||||
public DbConnectionFactory(IConfiguration configuration, ILogger<DbConnectionFactory> logger)
|
||||
{
|
||||
_configuration = configuration;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public async Task<SqlConnection> CreateLotFinderConnectionAsync(CancellationToken ct = default)
|
||||
{
|
||||
var connectionString = _configuration.GetConnectionString("LotFinderDB")
|
||||
?? throw new ConnectionException("LotFinderDB connection string not configured", "LotFinderDB");
|
||||
|
||||
var connection = new SqlConnection(connectionString);
|
||||
try
|
||||
{
|
||||
await connection.OpenAsync(ct);
|
||||
return connection;
|
||||
}
|
||||
catch (SqlException ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to connect to LotFinderDB");
|
||||
await connection.DisposeAsync();
|
||||
throw new ConnectionException("LotFinderDB: failed to open connection to database.", "LotFinderDB", ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Connection factory SHALL be registered as singleton
|
||||
- Connections SHALL be opened asynchronously before returning
|
||||
- Callers SHALL dispose returned connections when finished
|
||||
- `ConnectionException` SHALL be thrown on connection failure with inner exception preserved
|
||||
|
||||
#### Scenario: Successful connection creation
|
||||
|
||||
- **WHEN** valid connection string exists in configuration
|
||||
- **AND** `CreateLotFinderConnectionAsync()` is called
|
||||
- **THEN** a new `SqlConnection` is created, opened, and returned
|
||||
- **AND** the caller is responsible for disposal
|
||||
|
||||
#### Scenario: Connection failure handling
|
||||
|
||||
- **WHEN** database is unreachable
|
||||
- **AND** `CreateLotFinderConnectionAsync()` is called
|
||||
- **THEN** error is logged with exception details
|
||||
- **AND** `ConnectionException` is thrown with descriptive message
|
||||
- **AND** inner exception is preserved for debugging
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Async streaming pattern
|
||||
|
||||
The system SHALL use `IAsyncEnumerable<T>` with Dapper's `QueryUnbufferedAsync` for streaming large result sets.
|
||||
|
||||
#### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
public async IAsyncEnumerable<WorkOrder> GetWorkOrdersAsync(
|
||||
DateTime? lastUpdateDT = null,
|
||||
[EnumeratorCancellation] CancellationToken ct = default)
|
||||
{
|
||||
await using var connection = await _connectionFactory.CreateJdeConnectionAsync(ct);
|
||||
|
||||
var sql = ApplySchemaPlaceholders(
|
||||
lastUpdateDT.HasValue
|
||||
? JdeQueries.SQL_GET_WORKORDERS_FILTERED
|
||||
: JdeQueries.SQL_GET_WORKORDERS);
|
||||
|
||||
var command = new CommandDefinition(
|
||||
sql,
|
||||
parameters: lastUpdateDT.HasValue
|
||||
? new { dateUpdated = ToJdeDate(lastUpdateDT.Value), timeUpdated = ToJdeTime(lastUpdateDT.Value) }
|
||||
: null,
|
||||
commandTimeout: _options.Value.DefaultTimeoutSeconds,
|
||||
cancellationToken: ct);
|
||||
|
||||
await foreach (var workOrder in connection.QueryUnbufferedAsync<WorkOrder>(command).WithCancellation(ct))
|
||||
{
|
||||
yield return workOrder;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- All JDE/CMS collection queries SHALL return `IAsyncEnumerable<T>`
|
||||
- Queries SHALL use `QueryUnbufferedAsync` to stream results
|
||||
- Methods SHALL accept `CancellationToken` with `[EnumeratorCancellation]` attribute
|
||||
- Cancellation SHALL be checked between row iterations via `WithCancellation(ct)`
|
||||
|
||||
#### Scenario: Stream large work order dataset
|
||||
|
||||
- **WHEN** `GetWorkOrdersAsync()` is called for 1 million work orders
|
||||
- **THEN** results are streamed via `IAsyncEnumerable<WorkOrder>` one at a time
|
||||
- **AND** memory usage remains constant regardless of result set size
|
||||
- **AND** consumer can use `await foreach` syntax
|
||||
|
||||
#### Scenario: Cancel streaming operation
|
||||
|
||||
- **WHEN** cancellation is requested during `GetWorkOrdersAsync()` iteration
|
||||
- **THEN** iteration stops after current row completes
|
||||
- **AND** `OperationCanceledException` is thrown to consumer
|
||||
- **AND** database connection is properly disposed
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Schema placeholder replacement
|
||||
|
||||
The system SHALL replace schema placeholders in SQL queries from `DataAccessOptions`.
|
||||
|
||||
#### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
private string ApplySchemaPlaceholders(string sql)
|
||||
{
|
||||
return sql
|
||||
.Replace("{ProductionSchema}", _options.Value.ProductionSchema)
|
||||
.Replace("{ArchiveSchema}", _options.Value.ArchiveSchema)
|
||||
.Replace("{StageSchema}", _options.Value.StageSchema);
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Schema names SHALL be configurable for environment-specific deployments
|
||||
- Default values: PRODDTA (production), ARCDTAPD (archive), JDESTAGE (stage)
|
||||
- Replacement SHALL occur at query execution time
|
||||
|
||||
#### Scenario: Query uses production schema
|
||||
|
||||
- **WHEN** SQL query contains `{ProductionSchema}.F4801`
|
||||
- **AND** `DataAccessOptions.ProductionSchema` is "PRODDTA"
|
||||
- **THEN** query is executed with `PRODDTA.F4801`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Table name whitelist validation
|
||||
|
||||
The system SHALL validate table names against an explicit whitelist to prevent SQL injection in `RebuildIndicesAsync`.
|
||||
|
||||
#### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
private static readonly HashSet<string> ValidTableNames = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
"Branch", "DataUpdate", "FunctionCode", "Item", "JdeUser",
|
||||
"Lot", "LotLocation", "LotUsage_Curr", "LotUsage_Hist",
|
||||
"MisData", "OrgHierarchy", "ProfitCenter", "RouteMaster",
|
||||
"Search", "StatusCode", "WorkCenter",
|
||||
"WorkOrder_Curr", "WorkOrder_Hist",
|
||||
"WorkOrderComponent_Curr", "WorkOrderComponent_Hist",
|
||||
"WorkOrderRouting",
|
||||
"WorkOrderStep_Curr", "WorkOrderStep_Hist",
|
||||
"WorkOrderTime_Curr", "WorkOrderTime_Hist"
|
||||
};
|
||||
|
||||
public async Task RebuildIndicesAsync(string tableName, CancellationToken ct = default)
|
||||
{
|
||||
if (!ValidTableNames.Contains(tableName))
|
||||
{
|
||||
throw new ArgumentException($"Invalid table name: {tableName}", nameof(tableName));
|
||||
}
|
||||
|
||||
await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
|
||||
var sql = $"ALTER INDEX ALL ON [{tableName}] REBUILD WITH (FILLFACTOR = 95)";
|
||||
await connection.ExecuteAsync(sql, commandTimeout: _options.Value.RebuildIndexTimeoutSeconds);
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Table name MUST be validated against explicit whitelist before execution
|
||||
- `ArgumentException` SHALL be thrown for invalid table names
|
||||
- Comparison SHALL be case-insensitive
|
||||
|
||||
#### Scenario: Valid table name accepted
|
||||
|
||||
- **WHEN** `RebuildIndicesAsync("WorkOrder_Curr")` is called
|
||||
- **THEN** table name passes whitelist validation
|
||||
- **AND** index rebuild executes successfully
|
||||
|
||||
#### Scenario: SQL injection attempt blocked
|
||||
|
||||
- **WHEN** `RebuildIndicesAsync("WorkOrder]; DROP TABLE Search;--")` is called
|
||||
- **THEN** table name fails whitelist validation
|
||||
- **AND** `ArgumentException` is thrown with message "Invalid table name"
|
||||
- **AND** no SQL is executed
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Exception logging with scope context
|
||||
|
||||
The system SHALL log exceptions with structured scope context before throwing.
|
||||
|
||||
#### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
catch (OracleException ex)
|
||||
{
|
||||
using (_logger.BeginScope(new Dictionary<string, object>
|
||||
{
|
||||
["DataSource"] = "JDE",
|
||||
["Operation"] = nameof(GetWorkOrdersAsync),
|
||||
["QueryName"] = "SQL_GET_WORKORDERS"
|
||||
}))
|
||||
{
|
||||
_logger.LogError(ex, "Query execution failed");
|
||||
}
|
||||
throw new QueryException("Failed to execute work order query", "SQL_GET_WORKORDERS", ex);
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- All exceptions SHALL be logged at throw site
|
||||
- Log context SHALL include: DataSource, Operation, and QueryName where applicable
|
||||
- Inner exceptions SHALL be preserved in thrown exceptions
|
||||
- Structured logging SHALL enable log aggregation and analysis
|
||||
|
||||
#### Scenario: Query exception with full context
|
||||
|
||||
- **WHEN** JDE Oracle query fails with OracleException
|
||||
- **THEN** error is logged with BeginScope containing DataSource, Operation, QueryName
|
||||
- **AND** `QueryException` is thrown with descriptive message and inner exception
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Table-valued parameter support
|
||||
|
||||
The system SHALL use DataTable for SQL Server table-valued parameters in lookup methods.
|
||||
|
||||
#### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
public async Task<List<Item>> LookupItemsAsync(List<string> itemNumbers, CancellationToken ct = default)
|
||||
{
|
||||
await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(ct);
|
||||
|
||||
var table = new DataTable();
|
||||
table.Columns.Add("ItemNumber", typeof(string));
|
||||
foreach (var itemNumber in itemNumbers)
|
||||
{
|
||||
table.Rows.Add(itemNumber);
|
||||
}
|
||||
|
||||
var parameters = new DynamicParameters();
|
||||
parameters.Add("@itemNumbers", table.AsTableValuedParameter("dbo.ItemNumberFilterParameter"));
|
||||
|
||||
return (await connection.QueryAsync<Item>(
|
||||
LotFinderQueries.SQL_LOOKUP_ITEMS,
|
||||
parameters,
|
||||
commandTimeout: _options.Value.DefaultTimeoutSeconds))
|
||||
.AsList();
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- TVPs SHALL use `AsTableValuedParameter` extension with correct type name
|
||||
- DataTable column names SHALL match TVP type column names
|
||||
- TVPs enable efficient batch lookups with single database round-trip
|
||||
|
||||
#### Scenario: Batch lookup with TVP
|
||||
|
||||
- **WHEN** `LookupItemsAsync(["ITEM001", "ITEM002", "ITEM003"])` is called
|
||||
- **THEN** DataTable is created with ItemNumber column
|
||||
- **AND** single query executes with TVP parameter
|
||||
- **AND** matching items are returned
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Service registration extension method
|
||||
|
||||
The system SHALL provide `AddDataAccess` extension method for DI registration.
|
||||
|
||||
#### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
public static IServiceCollection AddDataAccess(
|
||||
this IServiceCollection services,
|
||||
IConfiguration configuration)
|
||||
{
|
||||
services.Configure<DataAccessOptions>(
|
||||
configuration.GetSection(DataAccessOptions.SectionName));
|
||||
|
||||
services.AddSingleton<IDbConnectionFactory, DbConnectionFactory>();
|
||||
services.AddScoped<ILotFinderRepository, LotFinderRepository>();
|
||||
services.AddScoped<IJdeRepository, JdeRepository>();
|
||||
services.AddScoped<ICmsRepository, CmsRepository>();
|
||||
|
||||
return services;
|
||||
}
|
||||
```
|
||||
|
||||
#### Business Rules
|
||||
|
||||
- Extension method SHALL bind `DataAccessOptions` from "DataAccess" configuration section
|
||||
- Connection factory SHALL be registered as singleton
|
||||
- Repositories SHALL be registered as scoped services
|
||||
- Method SHALL return `IServiceCollection` for chaining
|
||||
|
||||
#### Scenario: Register all data access services
|
||||
|
||||
- **WHEN** `services.AddDataAccess(configuration)` is called during startup
|
||||
- **THEN** `DataAccessOptions` is bound from configuration
|
||||
- **AND** `IDbConnectionFactory` is registered as singleton
|
||||
- **AND** all repository interfaces are registered as scoped
|
||||
@@ -0,0 +1,208 @@
|
||||
# Tasks: Implement Data Access
|
||||
|
||||
## Phase 1: Project Setup
|
||||
|
||||
- [x] Create JdeScoping.DataAccess project
|
||||
- Location: `NEW/src/JdeScoping.DataAccess/JdeScoping.DataAccess.csproj`
|
||||
- Target: net10.0
|
||||
- Validation: Project builds successfully
|
||||
- Dependencies: None
|
||||
|
||||
- [x] Add NuGet package references
|
||||
- Packages: Microsoft.Data.SqlClient, Oracle.ManagedDataAccess.Core, Dapper, Microsoft.Extensions.Options, Microsoft.Extensions.Logging.Abstractions, Microsoft.Extensions.Configuration.Abstractions
|
||||
- Validation: `dotnet restore` succeeds
|
||||
|
||||
- [x] Create folder structure
|
||||
- Folders: Exceptions/, Interfaces/, Repositories/, Queries/, Configuration/
|
||||
- Validation: Directories exist
|
||||
|
||||
## Phase 2: Exception Types
|
||||
|
||||
- [x] Create DataAccessException base class
|
||||
- Location: `Exceptions/DataAccessException.cs`
|
||||
- Properties: Operation, Repository, Message, InnerException
|
||||
- Validation: Class compiles
|
||||
|
||||
- [x] Create ConnectionException class
|
||||
- Location: `Exceptions/ConnectionException.cs`
|
||||
- Properties: DataSource (inherits from DataAccessException)
|
||||
- Validation: Class compiles
|
||||
|
||||
- [x] Create QueryException class
|
||||
- Location: `Exceptions/QueryException.cs`
|
||||
- Properties: QueryName (inherits from DataAccessException)
|
||||
- Validation: Class compiles
|
||||
|
||||
- [x] Create DataAccessTimeoutException class
|
||||
- Location: `Exceptions/DataAccessTimeoutException.cs`
|
||||
- Properties: TimeoutSeconds (inherits from DataAccessException)
|
||||
- Validation: Class compiles
|
||||
|
||||
## Phase 3: Configuration
|
||||
|
||||
- [x] Create DataAccessOptions class
|
||||
- Location: `Configuration/DataAccessOptions.cs`
|
||||
- Properties: DefaultTimeoutSeconds, LotUsageTimeoutSeconds, MisDataTimeoutSeconds, RebuildIndexTimeoutSeconds, ProductionSchema, ArchiveSchema, StageSchema
|
||||
- Validation: Class compiles with default values
|
||||
|
||||
## Phase 4: Connection Factory
|
||||
|
||||
- [x] Create IDbConnectionFactory interface
|
||||
- Location: `Interfaces/IDbConnectionFactory.cs`
|
||||
- Methods: CreateLotFinderConnectionAsync, CreateJdeConnectionAsync, CreateJdeStageConnectionAsync, CreateCmsConnectionAsync
|
||||
- Validation: Interface compiles
|
||||
|
||||
- [x] Create DbConnectionFactory implementation
|
||||
- Location: `DbConnectionFactory.cs`
|
||||
- Dependencies: IConfiguration, ILogger<DbConnectionFactory>
|
||||
- Validation: Compiles, logs connection attempts
|
||||
|
||||
## Phase 5: SQL Query Constants
|
||||
|
||||
- [x] Create LotFinderQueries static class
|
||||
- Location: `Queries/LotFinderQueries.cs`
|
||||
- Contains: All SQL Server queries from spec (GetUserSearches, GetQueuedSearches, GetSearch, etc.)
|
||||
- Validation: All queries compile as const strings
|
||||
|
||||
- [x] Create JdeQueries static class
|
||||
- Location: `Queries/JdeQueries.cs`
|
||||
- Contains: All JDE Oracle queries from spec (GetWorkOrders, GetWorkOrderSteps, GetLots, etc.)
|
||||
- Note: Include both full and filtered variants
|
||||
- Validation: All queries compile as const strings
|
||||
|
||||
- [x] Create CmsQueries static class
|
||||
- Location: `Queries/CmsQueries.cs`
|
||||
- Contains: SQL_GET_MIS_DATA query from spec
|
||||
- Validation: Query compiles as const string
|
||||
|
||||
## Phase 6: Repository Interfaces
|
||||
|
||||
- [x] Create ILotFinderRepository interface
|
||||
- Location: `Interfaces/ILotFinderRepository.cs`
|
||||
- Methods: All 17 methods from spec (GetUserSearchesAsync, GetQueuedSearchesAsync, GetSearchAsync, GetSearchResultsAsync, SubmitSearchAsync, UpdateSearchStatusAsync, UpdateSearchResultsAsync, SearchItemsAsync, LookupItemsAsync, LookupWorkordersAsync, SearchWorkCentersAsync, LookupWorkCentersAsync, SearchProfitCentersAsync, LookupProfitCentersAsync, SearchUsersAsync, LookupUsersAsync, LookupLotsAsync, GetLastDataUpdatesAsync, GetTableSpecAsync, RebuildIndicesAsync, PostProcessMisDataAsync, BulkInsertAsync, TruncateTableAsync)
|
||||
- Validation: Interface compiles
|
||||
|
||||
- [x] Create IJdeRepository interface
|
||||
- Location: `Interfaces/IJdeRepository.cs`
|
||||
- Methods: All 18 methods from spec (GetWorkOrdersAsync, GetWorkOrdersArchiveAsync, GetWorkOrderStepsAsync, GetWorkOrderStepsArchiveAsync, GetWorkOrderTimesAsync, GetWorkOrderTimesArchiveAsync, GetWorkOrderRoutingsAsync, GetWorkOrderComponentsAsync, GetWorkOrderComponentsArchiveAsync, GetLotsAsync, GetLotUsagesAsync, GetLotUsagesArchiveAsync, GetLotLocationsAsync, GetItemsAsync, GetUsersAsync, GetBranchesAsync, GetProfitCentersAsync, GetWorkCentersAsync, GetStatusCodesAsync, GetFunctionCodesAsync, GetOrgHierarchyAsync, GetRouteMastersAsync)
|
||||
- Return types: IAsyncEnumerable<T> for streaming
|
||||
- Validation: Interface compiles
|
||||
|
||||
- [x] Create ICmsRepository interface
|
||||
- Location: `Interfaces/ICmsRepository.cs`
|
||||
- Methods: GetMisDataAsync
|
||||
- Return type: IAsyncEnumerable<MisData>
|
||||
- Validation: Interface compiles
|
||||
|
||||
## Phase 7: LotFinderRepository Implementation
|
||||
|
||||
- [x] Create LotFinderRepository class
|
||||
- Location: `Repositories/LotFinderRepository.cs`
|
||||
- Dependencies: IDbConnectionFactory, ILogger<LotFinderRepository>, IOptions<DataAccessOptions>
|
||||
- Validation: Class compiles
|
||||
|
||||
- [x] Implement Search Management methods
|
||||
- Methods: GetUserSearchesAsync, GetQueuedSearchesAsync, GetSearchAsync, GetSearchResultsAsync, SubmitSearchAsync, UpdateSearchStatusAsync, UpdateSearchResultsAsync
|
||||
- Validation: Methods compile, use Dapper QueryAsync/ExecuteAsync
|
||||
|
||||
- [x] Implement Reference Data Lookup methods
|
||||
- Methods: SearchItemsAsync, LookupItemsAsync, LookupWorkordersAsync, SearchWorkCentersAsync, LookupWorkCentersAsync, SearchProfitCentersAsync, LookupProfitCentersAsync, SearchUsersAsync, LookupUsersAsync, LookupLotsAsync
|
||||
- Note: Use DataTable for table-valued parameters
|
||||
- Validation: Methods compile
|
||||
|
||||
- [x] Implement Data Sync methods
|
||||
- Methods: GetLastDataUpdatesAsync, GetTableSpecAsync, RebuildIndicesAsync, PostProcessMisDataAsync, BulkInsertAsync, TruncateTableAsync
|
||||
- Note: RebuildIndicesAsync includes table name whitelist validation
|
||||
- Validation: Methods compile
|
||||
|
||||
## Phase 8: JdeRepository Implementation
|
||||
|
||||
- [x] Create JdeRepository class
|
||||
- Location: `Repositories/JdeRepository.cs`
|
||||
- Dependencies: IDbConnectionFactory, ILogger<JdeRepository>, IOptions<DataAccessOptions>
|
||||
- Validation: Class compiles
|
||||
|
||||
- [x] Implement Work Order methods
|
||||
- Methods: GetWorkOrdersAsync, GetWorkOrdersArchiveAsync, GetWorkOrderStepsAsync, GetWorkOrderStepsArchiveAsync, GetWorkOrderTimesAsync, GetWorkOrderTimesArchiveAsync, GetWorkOrderRoutingsAsync, GetWorkOrderComponentsAsync, GetWorkOrderComponentsArchiveAsync
|
||||
- Pattern: IAsyncEnumerable<T> with Query (buffered: false) for streaming
|
||||
- Validation: Methods compile
|
||||
|
||||
- [x] Implement Lot methods
|
||||
- Methods: GetLotsAsync, GetLotUsagesAsync, GetLotUsagesArchiveAsync, GetLotLocationsAsync
|
||||
- Note: GetLotLocationsAsync uses JDE Stage connection
|
||||
- Validation: Methods compile
|
||||
|
||||
- [x] Implement Reference Data methods
|
||||
- Methods: GetItemsAsync, GetUsersAsync, GetBranchesAsync, GetProfitCentersAsync, GetWorkCentersAsync, GetStatusCodesAsync, GetFunctionCodesAsync, GetOrgHierarchyAsync, GetRouteMastersAsync
|
||||
- Note: GetStatusCodesAsync uses JDE Stage connection
|
||||
- Validation: Methods compile
|
||||
|
||||
- [x] Implement schema placeholder replacement
|
||||
- Method: Private ApplySchemaReplacements method
|
||||
- Replaces: {ProductionSchema}, {ArchiveSchema}, {StageSchema}
|
||||
- Validation: Placeholders replaced correctly
|
||||
|
||||
## Phase 9: CmsRepository Implementation
|
||||
|
||||
- [x] Create CmsRepository class
|
||||
- Location: `Repositories/CmsRepository.cs`
|
||||
- Dependencies: IDbConnectionFactory, ILogger<CmsRepository>, IOptions<DataAccessOptions>
|
||||
- Validation: Class compiles
|
||||
|
||||
- [x] Implement GetMisDataAsync method
|
||||
- Pattern: IAsyncEnumerable<MisData> with Query (buffered: false) for streaming
|
||||
- Timeout: Uses MisDataTimeoutSeconds from options
|
||||
- Validation: Method compiles
|
||||
|
||||
## Phase 10: Service Registration
|
||||
|
||||
- [x] Create ServiceCollectionExtensions class
|
||||
- Location: `Extensions/ServiceCollectionExtensions.cs`
|
||||
- Method: AddDataAccess(this IServiceCollection services, IConfiguration configuration)
|
||||
- Registers: DataAccessOptions, IDbConnectionFactory (singleton), all repositories (scoped)
|
||||
- Validation: Extension method compiles
|
||||
|
||||
## Phase 11: Unit Tests
|
||||
|
||||
- [x] Create test project
|
||||
- Location: `NEW/tests/JdeScoping.DataAccess.Tests/JdeScoping.DataAccess.Tests.csproj`
|
||||
- Dependencies: xUnit, NSubstitute, Shouldly
|
||||
- Validation: Project builds
|
||||
|
||||
- [x] Create DbConnectionFactory tests
|
||||
- Tests: Connection creation, error handling, logging
|
||||
- Validation: Tests pass
|
||||
|
||||
- [x] Create LotFinderRepository tests
|
||||
- Tests: Search methods, lookup methods, exception handling
|
||||
- Mock: IDbConnectionFactory
|
||||
- Validation: Tests pass
|
||||
|
||||
- [x] Create JdeRepository tests
|
||||
- Tests: Streaming methods, cancellation, schema replacement
|
||||
- Mock: IDbConnectionFactory
|
||||
- Validation: Tests pass
|
||||
|
||||
- [x] Create CmsRepository tests
|
||||
- Tests: GetMisDataAsync, timeout configuration
|
||||
- Mock: IDbConnectionFactory
|
||||
- Validation: Tests pass
|
||||
|
||||
## Phase 12: Verification
|
||||
|
||||
- [x] Build complete solution
|
||||
- Command: `dotnet build NEW/JdeScoping.slnx`
|
||||
- Validation: No build errors
|
||||
|
||||
- [x] Run all unit tests
|
||||
- Command: `dotnet test NEW/tests/JdeScoping.DataAccess.Tests/`
|
||||
- Validation: All 124 tests pass
|
||||
|
||||
- [x] Validate OpenSpec change
|
||||
- Command: `openspec validate implement-data-access --strict`
|
||||
- Validation: No validation errors
|
||||
|
||||
- [x] Codex MCP review
|
||||
- Review: Repository implementations against spec
|
||||
- Verify: All methods match spec signatures
|
||||
- Verify: Query SQL matches legacy exactly
|
||||
Reference in New Issue
Block a user