Initial commit: JDE Scoping Tool migration project

Set up repository with legacy .NET Framework 4.8 source (OLD/), new .NET 10 Blazor solution (NEW/), OpenSpec specifications, documentation, and project configuration.
2026-01-02 07:43:29 -05:00
commit 26ff8d9b4f
1761 changed files with 596509 additions and 0 deletions
@@ -0,0 +1,259 @@
+# TableSyncOperation Migration Design
+
+## Goal
+
+Migrate `TableSyncOperation` from using `IStagingTableManager` with `TableSpec` metadata to using `IBulkMergeHelper` with expression-based `IMergeConfiguration<T>` classes. Remove old staging table code after migration.
+
+## Architecture Decisions
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| Column metadata source | Expression lambdas | Compile-time safety, explicit configuration |
+| Configuration style | Fluent `IMergeConfiguration<T>` classes | Clean Architecture - keeps domain entities clean |
+| Configuration location | DataSync project | Infrastructure concern, not domain |
+| Mass update support | Add to `BulkMergeHelper` | Unified bulk operations API |
+| Index management | Built into `BulkMergeHelper` | Simple API, matches current behavior |
+| DataReader generation | Source generator (fix first) | Best long-term solution, block until working |
+| Old code removal | Full removal | Clean break, no dead code |
+| DI registration | Explicit per-entity | Visibility and control |
+
+---
+
+## Component Design
+
+### 1. IMergeConfiguration Interface
+
+Location: `src/JdeScoping.DataSync/Contracts/IMergeConfiguration.cs`
+
+```csharp
+public interface IMergeConfiguration<T> where T : class
+{
+    /// <summary>Table name in SQL Server cache.</summary>
+    string TableName { get; }
+
+    /// <summary>Columns to match on (primary key).</summary>
+    Expression<Func<T, object>> MatchOn { get; }
+
+    /// <summary>Columns to update when matched. Null = all non-PK columns.</summary>
+    Expression<Func<T, object>>? UpdateColumns { get; }
+
+    /// <summary>Condition for when to update. Null = always update on match.</summary>
+    Expression<Func<T, T, bool>>? UpdateWhen { get; }
+
+    /// <summary>Columns to insert. Null = all columns.</summary>
+    Expression<Func<T, object>>? InsertColumns { get; }
+}
+```
+
+### 2. Example Merge Configuration
+
+Location: `src/JdeScoping.DataSync/Configuration/MergeConfigurations/WorkOrderMergeConfiguration.cs`
+
+```csharp
+public class WorkOrderMergeConfiguration : IMergeConfiguration<WorkOrder>
+{
+    public string TableName => "WorkOrder";
+
+    public Expression<Func<WorkOrder, object>> MatchOn =>
+        x => new { x.WorkOrderNumber, x.BranchCode };
+
+    public Expression<Func<WorkOrder, object>>? UpdateColumns =>
+        x => new { x.Status, x.Quantity, x.LastUpdateDt };
+
+    public Expression<Func<WorkOrder, WorkOrder, bool>>? UpdateWhen =>
+        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;
+
+    public Expression<Func<WorkOrder, object>>? InsertColumns => null; // All columns
+}
+```
+
+### 3. Extended IBulkMergeHelper
+
+Added to: `src/JdeScoping.DataSync/Contracts/IBulkMergeHelper.cs`
+
+```csharp
+/// <summary>
+/// Performs a mass insert (full table refresh) with optional index management.
+/// Truncates table, disables non-clustered indexes, bulk copies data, rebuilds indexes.
+/// </summary>
+Task<MassInsertResult> MassInsertAsync<T>(
+    IAsyncEnumerable<T> data,
+    string destinationTable,
+    bool rebuildIndexes = true,
+    int batchSize = 0,
+    CancellationToken cancellationToken = default) where T : class;
+```
+
+New result type:
+```csharp
+public record MassInsertResult(
+    long TotalRowsInserted,
+    TimeSpan Elapsed,
+    bool IndexesRebuilt);
+```
+
+### 4. MergeConfigurationRegistry
+
+Location: `src/JdeScoping.DataSync/Services/MergeConfigurationRegistry.cs`
+
+```csharp
+public interface IMergeConfigurationRegistry
+{
+    IMergeConfiguration<T> GetConfiguration<T>() where T : class;
+    bool HasConfiguration<T>() where T : class;
+}
+
+internal class MergeConfigurationRegistry : IMergeConfigurationRegistry
+{
+    private readonly IServiceProvider _serviceProvider;
+
+    public MergeConfigurationRegistry(IServiceProvider serviceProvider)
+    {
+        _serviceProvider = serviceProvider;
+    }
+
+    public IMergeConfiguration<T> GetConfiguration<T>() where T : class
+    {
+        return _serviceProvider.GetService<IMergeConfiguration<T>>()
+            ?? throw new InvalidOperationException(
+                $"No merge configuration registered for {typeof(T).Name}");
+    }
+
+    public bool HasConfiguration<T>() where T : class
+    {
+        return _serviceProvider.GetService<IMergeConfiguration<T>>() != null;
+    }
+}
+```
+
+### 5. Refactored TableSyncOperation
+
+Key changes to `src/JdeScoping.DataSync/Services/TableSyncOperation.cs`:
+
+- Remove `IStagingTableManager` dependency
+- Add `IBulkMergeHelper` dependency
+- Add `IMergeConfigurationRegistry` dependency
+- Remove `ILotFinderRepository` dependency (no longer need `GetTableSpecAsync`)
+
+```csharp
+private async Task<long> ExecuteIncrementalUpdateAsync<T>(
+    IAsyncEnumerable<T> data,
+    DataUpdateTask task,
+    CancellationToken ct) where T : class
+{
+    var config = _configRegistry.GetConfiguration<T>();
+
+    var result = await _bulkMergeHelper.MergeAsync(
+        data,
+        config.TableName,
+        config.MatchOn,
+        config.UpdateColumns,
+        config.UpdateWhen,
+        config.InsertColumns,
+        batchSize: _options.Value.BatchSize,
+        cancellationToken: ct);
+
+    return result.TotalRowsProcessed;
+}
+
+private async Task<long> ExecuteMassUpdateAsync<T>(
+    IAsyncEnumerable<T> data,
+    DataUpdateTask task,
+    CancellationToken ct) where T : class
+{
+    var config = _configRegistry.GetConfiguration<T>();
+
+    var result = await _bulkMergeHelper.MassInsertAsync(
+        data,
+        config.TableName,
+        rebuildIndexes: task.ScheduleConfig.ReIndexData,
+        batchSize: _options.Value.BulkCopyBatchSize,
+        cancellationToken: ct);
+
+    return result.TotalRowsInserted;
+}
+```
+
+### 6. DI Registration
+
+In `ServiceCollectionExtensions.cs`:
+
+```csharp
+// Merge configuration registry
+services.AddSingleton<IMergeConfigurationRegistry, MergeConfigurationRegistry>();
+
+// Merge configurations - explicit registration
+services.AddSingleton<IMergeConfiguration<WorkOrder>, WorkOrderMergeConfiguration>();
+services.AddSingleton<IMergeConfiguration<LotUsage>, LotUsageMergeConfiguration>();
+services.AddSingleton<IMergeConfiguration<Item>, ItemMergeConfiguration>();
+// ... one line per synced entity
+```
+
+---
+
+## Source Generator Fix
+
+The generator at `JdeScoping.DataSync.SourceGenerators` must produce `IDataReader` implementations.
+
+**Verification steps:**
+1. Confirm `.csproj` reference has `OutputItemType="Analyzer"` and `ReferenceOutputAssembly="false"`
+2. Add diagnostic logging to generator
+3. Ensure `[GenerateDataReader]` attribute is accessible
+4. Generator emits `{EntityName}DataReader : IDataReader` per attributed entity
+
+**Generated output pattern:**
+```csharp
+public sealed class WorkOrderDataReader : IDataReader
+{
+    private readonly IAsyncEnumerator<WorkOrder> _enumerator;
+    private WorkOrder? _current;
+
+    private static readonly string[] ColumnNames =
+        ["WorkOrderNumber", "BranchCode", "Status", "Quantity", "LastUpdateDt"];
+
+    public object GetValue(int i) => i switch
+    {
+        0 => _current!.WorkOrderNumber,
+        1 => _current!.BranchCode,
+        // ...
+    };
+}
+```
+
+---
+
+## Code Removal (After Migration Verified)
+
+**Delete:**
+- `src/JdeScoping.DataSync/Contracts/IStagingTableManager.cs`
+- `src/JdeScoping.DataSync/Services/StagingTableManager.cs`
+- `tests/JdeScoping.DataSync.Tests/StagingTableManagerTests.cs`
+- `tests/JdeScoping.DataSync.IntegrationTests/StagingTableManagerTests.cs`
+
+**Update:**
+- Remove `IStagingTableManager`/`StagingTableManager` from `ServiceCollectionExtensions.cs`
+- Remove `GetTableSpecAsync` usage from DataSync (if no longer needed elsewhere)
+
+---
+
+## Testing Strategy
+
+**Unit tests - new:**
+- `MergeConfigurationRegistryTests`
+- `BulkMergeHelper.MassInsertAsync` tests
+
+**Unit tests - update:**
+- `TableSyncOperationTests` - update mocks
+
+**Integration tests:**
+- `BulkMergeHelperTests` - add mass insert tests
+- `TableSyncOperationTests` - end-to-end verification
+
+**Source generator tests:**
+- Verify generated `IDataReader` implementations
+- Test with `SqlBulkCopy`
+
+**Verification before old code removal:**
+1. All existing tests pass
+2. Integration tests verify same behavior
+3. Manual smoke test if possible