refactor: relocate options classes to dedicated Options folders

Move configuration options from Core/DataAccess/DataSync/ExcelIO to
dedicated Options folders within each project for better organization.
Update all references and tests accordingly.
This commit is contained in:
Joseph Doherty
2026-01-03 08:55:08 -05:00
parent 3cb73eb09f
commit ec4c8fab87
52 changed files with 4628 additions and 202 deletions
+134
View File
@@ -0,0 +1,134 @@
# Encrypted Login Design
## Overview
Consolidate login models into Core project and implement RSA-OAEP encryption for login credentials between Blazor WASM client and API.
## Decisions
| Decision | Choice | Rationale |
|----------|--------|-----------|
| Encryption algorithm | RSA-OAEP (SHA-256) | Simple direct encryption, suitable for small payloads |
| Key size | 2048-bit | Industry standard, ~20 year security horizon |
| Key distribution | Fetch at runtime | Allows key rotation without client redeployment |
| Key storage | Auto-generate, persist to file | Self-bootstrapping, no manual key management |
| Key expiration | None | Internal enterprise app, manual rotation if needed |
## New Models (Core)
All in `JdeScoping.Core/Models/Auth/`:
```csharp
// LoginModel.cs - shared form/request model
public class LoginModel
{
[Required(ErrorMessage = "Username is required")]
public string Username { get; set; } = string.Empty;
[Required(ErrorMessage = "Password is required")]
public string Password { get; set; } = string.Empty;
}
// LoginResultModel.cs - API response (unencrypted)
public record LoginResultModel(
bool Success,
string? ErrorMessage,
UserInfo? User);
// EncryptedLoginRequest.cs - encrypted payload wrapper
public record EncryptedLoginRequest(string EncryptedData);
// PublicKeyResponse.cs - public key endpoint response
public record PublicKeyResponse(string PublicKeyPem);
```
## RSA Key Service (Infrastructure)
**Interface:** `JdeScoping.Core/Interfaces/IRsaKeyService.cs`
```csharp
public interface IRsaKeyService
{
string GetPublicKeyPem();
byte[] Decrypt(byte[] ciphertext);
}
```
**Implementation:** `JdeScoping.Infrastructure/Security/RsaKeyService.cs`
- Auto-generates 2048-bit RSA key on first startup if not exists
- Persists private key to configurable file path
- Exports public key as PEM format
- Decrypts using RSA-OAEP with SHA-256 padding
## API Changes
**New endpoint:**
```
GET /api/auth/public-key → PublicKeyResponse
```
**Modified endpoint:**
```
POST /api/auth/login
Body: EncryptedLoginRequest { EncryptedData: base64 }
Response: LoginResultModel
```
Flow:
1. Receive base64-encoded encrypted data
2. Decode and decrypt with RSA private key
3. Deserialize JSON to LoginModel
4. Authenticate via IAuthService
5. Return LoginResultModel (unencrypted)
## Client Changes
**New service:** `ICryptoService` / `CryptoService`
- Uses `Blazor.SubtleCrypto` NuGet package
- Fetches and caches server public key
- Encrypts LoginModel JSON with RSA-OAEP
- Returns base64-encoded ciphertext
**Modified AuthService:**
- Injects ICryptoService
- Encrypts LoginModel before sending
- Sends EncryptedLoginRequest to API
## Files to Delete
- `JdeScoping.Api/Models/LoginRequest.cs`
- `JdeScoping.Client/Models/LoginModel.cs`
## Dependencies
**JdeScoping.Client.csproj:**
```xml
<PackageReference Include="Blazor.SubtleCrypto" Version="..." />
```
## DI Registration
**API/Host:**
```csharp
services.AddSingleton<IRsaKeyService>(sp =>
new RsaKeyService(Path.Combine(appDataPath, "rsa-key.bin")));
```
**Client:**
```csharp
services.AddScoped<ICryptoService, CryptoService>();
```
## Test Coverage
### RsaKeyServiceTests (Api.Tests)
- GetPublicKeyPem returns valid PEM format
- Decrypt with valid ciphertext returns plaintext
- Constructor loads existing key from file
- Constructor generates and persists new key if missing
### CryptoServiceTests (Client.Tests)
- EncryptLoginAsync returns valid base64
- EncryptLoginAsync caches public key (single HTTP call)
### Integration (optional)
- Full roundtrip: encrypt on client → decrypt on API → authenticate
File diff suppressed because it is too large Load Diff
+334
View File
@@ -0,0 +1,334 @@
# ETL Pipeline Design for DataSync
**Date:** 2026-01-03
**Status:** Reviewed (Codex MCP)
**Purpose:** Replace the existing strongly-typed fetcher + source-generated DataReader + BulkMergeHelper system with a flexible, configuration-driven ETL pipeline.
## Problem Statement
The current DataSync system requires:
1. A source-generated `IDataReader` implementation per entity type
2. Strongly-typed `IDataFetcher<T>` classes for each data source
3. Code changes to modify queries or add new sync operations
**Primary drivers for change:**
- **Flexibility** - Enable ad-hoc queries and schema changes without code modifications
- **Observability** - Better error handling and row count tracking across pipeline steps
## Design Overview
```
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ ImportSource│───▶│ Transformer │───▶│ Destination │
│ (IDataReader) │ (decorates) │ │ (bulk ops) │
└─────────────┘ └──────────────┘ └─────────────┘
▲ │
│ ▼
┌──────┴──────┐ ┌──────────────┐
│ Pre-Scripts │ │ Post-Scripts │
└─────────────┘ └──────────────┘
```
All steps report to `PipelineResult` with timing and row counts.
## Core Interfaces
### IImportSource
Reads data from a source system, returns `IDataReader` for streaming.
```csharp
public interface IImportSource : IAsyncDisposable
{
Task<IDataReader> ReadDataAsync(CancellationToken cancellationToken = default);
string SourceName { get; }
}
```
**Implementations:**
- `DbQuerySource` - Executes SQL against Oracle/Sybase/SQL Server via `IDbConnectionFactory`
- `FileSource` - Reads CSV files with configurable delimiter/header options
### IDataTransformer
Wraps an `IDataReader` and applies transformations (column rename, drop, type conversion).
```csharp
public interface IDataTransformer
{
IDataReader Transform(IDataReader source);
string TransformerName { get; }
}
```
**Implementations:**
- `JdeDateTransformer` - Merges Julian date + time columns into `DateTime`
- `ColumnRenameTransformer` - Renames columns
- `ColumnDropTransformer` - Removes columns from output
- `ColumnAddTransformer` - Adds computed columns
Transformers are registered by name in `ITransformerRegistry` for config-based hydration.
### IImportDestination
Writes data to a target table using bulk operations.
```csharp
public interface IImportDestination
{
Task<DestinationResult> WriteAsync(
IDataReader source,
CancellationToken cancellationToken = default);
string DestinationName { get; }
}
public record DestinationResult(
long RowsProcessed,
int BatchCount,
TimeSpan Elapsed);
```
**Implementations:**
#### DbBulkImportDestination (Full Refresh)
1. Truncate destination table
2. Bulk copy all data in batches
3. Per-batch commits
#### DbBulkMergeDestination (Incremental)
1. Create temp table from destination schema
2. For each batch:
- Bulk copy to temp table
- Execute MERGE to destination
- Truncate temp table
3. Drop temp table on completion
**Configuration options:**
- `tableName` - Target table
- `matchColumns` - PK columns for MERGE ON clause
- `updateColumns` - Columns to update (default: all non-match columns)
- `batchSize` - Rows per batch (default: 10000)
### IScriptRunner
Executes SQL scripts before or after the main pipeline.
```csharp
public interface IScriptRunner
{
Task ExecuteAsync(CancellationToken cancellationToken = default);
string ScriptName { get; }
}
```
**Common scripts (factory methods):**
- `CommonScripts.DisableIndexes(factory, tableName)`
- `CommonScripts.RebuildIndexes(factory, tableName)`
- `CommonScripts.UpdateStatistics(factory, tableName)`
- `CommonScripts.CustomSql(factory, sql, name)`
### Pipeline Results
```csharp
public record PipelineResult(
bool Success,
long TotalRows,
TimeSpan Elapsed,
IReadOnlyList<StepResult> Steps,
Exception? Error = null);
public record StepResult(
string StepName,
string StepType, // "Source", "Transform", "Destination", "Script"
long RowsAffected,
TimeSpan Elapsed);
```
## EtlPipeline Orchestration
```csharp
public class EtlPipeline
{
public string PipelineName { get; }
public async Task<PipelineResult> ExecuteAsync(CancellationToken cancellationToken = default)
{
// 1. Run pre-scripts (fail-fast)
// 2. Open source, get IDataReader
// 3. Chain transformers (decorators)
// 4. Write to destination
// 5. Run post-scripts
// 6. Return PipelineResult with all step metrics
}
}
```
**Error handling:**
- Fail-fast on any error
- No cross-batch transactions (per-batch commits)
- Caller marks overall sync operation as failed
- Restart from beginning on failure (no resume)
## Pipeline Construction
### Fluent Builder
```csharp
var pipeline = new EtlPipelineBuilder()
.WithName("WorkOrderSync")
.WithSource(new DbQuerySource(connFactory, "JDE", workOrderQuery))
.WithTransformer(new JdeDateTransformer("UPMJ", "TDAY", "UpdatedAt"))
.WithDestination(new DbBulkMergeDestination(connFactory, "WorkOrder", ["OrderNumber"]))
.WithPreScript(CommonScripts.DisableIndexes(connFactory, "WorkOrder"))
.WithPostScript(CommonScripts.RebuildIndexes(connFactory, "WorkOrder"))
.WithLogger(logger)
.Build();
var result = await pipeline.ExecuteAsync(cancellationToken);
```
### Configuration-Based
```json
{
"name": "WorkOrderSync",
"source": {
"type": "db",
"connectionName": "JDE",
"query": "SELECT * FROM F4801 WHERE UPMJ >= :minDate"
},
"transformers": [
{ "type": "jde-date", "options": { "dateColumn": "UPMJ", "timeColumn": "TDAY", "outputColumn": "UpdatedAt" } }
],
"destination": {
"type": "bulk-merge",
"tableName": "WorkOrder",
"matchColumns": ["OrderNumber"],
"batchSize": 10000
},
"preScripts": [
{ "type": "disable-indexes", "tableName": "WorkOrder" }
],
"postScripts": [
{ "type": "rebuild-indexes", "tableName": "WorkOrder" }
]
}
```
Hydrated via `EtlPipelineFactory.CreateFromConfig(config)`.
## Project Structure
```
NEW/src/JdeScoping.DataSync/
├── Contracts/
│ ├── IImportSource.cs
│ ├── IDataTransformer.cs
│ ├── IImportDestination.cs
│ ├── IScriptRunner.cs
│ └── ITransformerRegistry.cs
├── Etl/
│ ├── Pipeline/
│ │ ├── EtlPipeline.cs
│ │ ├── EtlPipelineBuilder.cs
│ │ └── EtlPipelineFactory.cs
│ ├── Sources/
│ │ ├── DbQuerySource.cs
│ │ └── FileSource.cs
│ ├── Transformers/
│ │ ├── DataTransformerBase.cs
│ │ ├── TransformingDataReader.cs
│ │ ├── JdeDateTransformer.cs
│ │ ├── ColumnRenameTransformer.cs
│ │ ├── ColumnDropTransformer.cs
│ │ └── TransformerRegistry.cs
│ ├── Destinations/
│ │ ├── DbBulkImportDestination.cs
│ │ └── DbBulkMergeDestination.cs
│ ├── Scripts/
│ │ ├── SqlScriptRunner.cs
│ │ └── CommonScripts.cs
│ └── Results/
│ ├── PipelineResult.cs
│ ├── StepResult.cs
│ └── DestinationResult.cs
└── Config/
└── PipelineConfig.cs
```
## Integration Strategy
1. **Parallel existence** - New ETL pipeline lives alongside existing `IDataFetcher<T>` + `IBulkMergeHelper`
2. **Gradual migration** - Convert one sync operation at a time to new pipeline
3. **Shared infrastructure** - Both use `IDbConnectionFactory`, logging, etc.
4. **Deprecation path** - Once all syncs migrated, remove old `Fetchers/` and source generator
## DI Registration
```csharp
public static class EtlServiceCollectionExtensions
{
public static IServiceCollection AddEtlPipeline(this IServiceCollection services)
{
services.AddSingleton<ITransformerRegistry, TransformerRegistry>();
services.AddTransient<EtlPipelineFactory>();
return services;
}
}
```
## Design Decisions
Based on Codex MCP review, the following decisions were made:
### Atomicity & Consistency
| Concern | Decision | Rationale |
|---------|----------|-----------|
| Full refresh partial visibility | **Accept** | Caller retries on failure; consumers tolerate brief inconsistency |
| Orphaned disabled indexes | **Accept risk** | Next successful run rebuilds; manual intervention rare |
| Temp table connection lifetime | **Destination owns connection** | `DbBulkMergeDestination` holds single connection for all batches |
### Implementation Requirements
1. **TransformingDataReader schema accuracy** - Transformers MUST correctly implement:
- `GetSchemaTable()` - Accurate after column changes
- `GetName(ordinal)` - Reflects renames
- `GetOrdinal(name)` - Works with new names
- `GetFieldType(ordinal)` - Correct after type transforms
- `FieldCount` - Accurate after drops/adds
2. **Transform step row counts** - `StepResult.RowsAffected` is `0` for transform steps (inline decorators). Only destination reports actual rows processed.
3. **Cancellation handling** - Destinations use `DbDataReader.ReadAsync()` internally, checking cancellation between rows. Source `IDataReader` interface stays synchronous for ADO.NET compatibility.
4. **Schema validation** - Optional `ValidateSchema` step available to compare source columns against expected schema before pipeline runs. Not mandatory.
5. **MERGE duplicate keys** - Source data MUST have unique keys for match columns. Pipeline does not dedupe; caller's responsibility to ensure uniqueness.
### Connection Lifetime Rules
```
DbQuerySource:
- Opens connection in ReadDataAsync()
- Holds connection until DisposeAsync()
- Connection closed when pipeline completes or fails
DbBulkImportDestination:
- Opens connection in WriteAsync()
- Holds for all batches
- Closes on completion or failure
DbBulkMergeDestination:
- Opens connection in WriteAsync()
- Creates #temp table
- Holds same connection for all batch cycles
- Drops temp table in finally block
- Closes connection on completion or failure
```
## Open Questions
1. Should `DbQuerySource` support parameterized queries with different parameter styles (`:param` for Oracle, `@param` for SQL Server)?
2. Should we support multiple destinations per pipeline (e.g., write to both current and history tables)?
3. How should pipeline configurations be stored and loaded (appsettings.json, separate files, database)?
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,116 @@
# Options Classes Relocation Design
## Summary
Move options classes from Core project to the projects that use them, establishing an `Options/` folder convention across all projects.
## Current State
All 7 options classes live in `JdeScoping.Core/Options/`:
- `AuthOptions` - used by Api + Infrastructure
- `DataSourceOptions` - used by Infrastructure
- `DataSyncOptions` - duplicate (real one in DataSync)
- `ExcelExportOptions` - duplicate (real one in ExcelIO)
- `LdapOptions` - used by Infrastructure
- `SearchOptions` - unused (future use in DataAccess)
- `SearchProcessingOptions` - used by DataAccess
## Target State
### Folder Structure
```
src/
├── JdeScoping.Api/
│ └── Options/
│ └── AuthOptions.cs
├── JdeScoping.DataAccess/
│ └── Options/
│ ├── SearchOptions.cs
│ └── SearchProcessingOptions.cs
├── JdeScoping.Infrastructure/
│ └── Options/
│ ├── DataSourceOptions.cs
│ └── LdapOptions.cs
├── JdeScoping.DataSync/
│ └── Options/ # Renamed from Configuration/
│ └── DataSyncOptions.cs
├── JdeScoping.ExcelIO/
│ └── Options/ # Renamed from Configuration/
│ └── ExcelExportOptions.cs
└── JdeScoping.Core/
└── (Options folder deleted)
```
### Property Moves
**LdapOptions** gains properties from AuthOptions:
- `UseFakeAuth` (bool, default: false)
- `AdminBypassUsers` (string[], default: [])
**AuthOptions** loses those properties, keeping only:
- `CookieName` (string, default: "ScopingTool.Auth")
- `CookieExpirationMinutes` (int, default: 480)
### Namespace Changes
| Old Namespace | New Namespace |
|---------------|---------------|
| `JdeScoping.Core.Options.AuthOptions` | `JdeScoping.Api.Options.AuthOptions` |
| `JdeScoping.Core.Options.SearchOptions` | `JdeScoping.DataAccess.Options.SearchOptions` |
| `JdeScoping.Core.Options.SearchProcessingOptions` | `JdeScoping.DataAccess.Options.SearchProcessingOptions` |
| `JdeScoping.Core.Options.DataSourceOptions` | `JdeScoping.Infrastructure.Options.DataSourceOptions` |
| `JdeScoping.Core.Options.LdapOptions` | `JdeScoping.Infrastructure.Options.LdapOptions` |
| `JdeScoping.DataSync.Configuration.*` | `JdeScoping.DataSync.Options.*` |
| `JdeScoping.ExcelIO.Configuration.*` | `JdeScoping.ExcelIO.Options.*` |
### DI Registration Changes
**Infrastructure/DependencyInjection.cs:**
- Remove `AuthOptions` registration
- Change `authOptions?.UseFakeAuth` check to use `ldapOptions?.UseFakeAuth`
**Api/DependencyInjection.cs:**
- Add `AuthOptions` registration
### Configuration (appsettings.json)
Move settings from `"Auth"` to `"Ldap"` section:
```json
{
"Ldap": {
"ServerUrls": ["ldap.corp.example.com"],
"GroupDn": "CN=ScopingTool-Users,OU=Groups,DC=corp,DC=example,DC=com",
"SearchBase": "DC=corp,DC=example,DC=com",
"ConnectionTimeoutSeconds": 30,
"UseFakeAuth": false,
"AdminBypassUsers": []
},
"Auth": {
"CookieName": "ScopingTool.Auth",
"CookieExpirationMinutes": 480
}
}
```
## Files to Delete
- `src/JdeScoping.Core/Options/AuthOptions.cs`
- `src/JdeScoping.Core/Options/DataSourceOptions.cs`
- `src/JdeScoping.Core/Options/DataSyncOptions.cs`
- `src/JdeScoping.Core/Options/ExcelExportOptions.cs`
- `src/JdeScoping.Core/Options/LdapOptions.cs`
- `src/JdeScoping.Core/Options/SearchOptions.cs`
- `src/JdeScoping.Core/Options/SearchProcessingOptions.cs`
- `src/JdeScoping.Core/Options/` (folder)
## Files to Update
- `Host/Program.cs` - update using statements
- `Infrastructure/DependencyInjection.cs` - remove AuthOptions, use LdapOptions for UseFakeAuth
- `Infrastructure/Auth/LdapAuthService.cs` - update using, use LdapOptions for AdminBypassUsers
- `Api/DependencyInjection.cs` - add AuthOptions registration
- `DataAccess/DependencyInjection.cs` - update using statements
- `DataSync/**` - rename Configuration namespace to Options
- `ExcelIO/**` - rename Configuration namespace to Options
- `appsettings.json` - move UseFakeAuth/AdminBypassUsers to Ldap section