jdescopingtool/PLANS/2026-01-01-tablesyncop-migration-design.md

# TableSyncOperation Migration Design

## Goal

Migrate `TableSyncOperation` from using `IStagingTableManager` with `TableSpec` metadata to using `IBulkMergeHelper` with expression-based `IMergeConfiguration<T>` classes. Remove old staging table code after migration.

## Architecture Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Column metadata source | Expression lambdas | Compile-time safety, explicit configuration |
| Configuration style | Fluent `IMergeConfiguration<T>` classes | Clean Architecture - keeps domain entities clean |
| Configuration location | DataSync project | Infrastructure concern, not domain |
| Mass update support | Add to `BulkMergeHelper` | Unified bulk operations API |
| Index management | Built into `BulkMergeHelper` | Simple API, matches current behavior |
| DataReader generation | Source generator (fix first) | Best long-term solution, block until working |
| Old code removal | Full removal | Clean break, no dead code |
| DI registration | Explicit per-entity | Visibility and control |

---

## Component Design

### 1. IMergeConfiguration Interface

Location: `src/JdeScoping.DataSync/Contracts/IMergeConfiguration.cs`

```csharp
public interface IMergeConfiguration<T> where T : class
{
    /// <summary>Table name in SQL Server cache.</summary>
    string TableName { get; }

    /// <summary>Columns to match on (primary key).</summary>
    Expression<Func<T, object>> MatchOn { get; }

    /// <summary>Columns to update when matched. Null = all non-PK columns.</summary>
    Expression<Func<T, object>>? UpdateColumns { get; }

    /// <summary>Condition for when to update. Null = always update on match.</summary>
    Expression<Func<T, T, bool>>? UpdateWhen { get; }

    /// <summary>Columns to insert. Null = all columns.</summary>
    Expression<Func<T, object>>? InsertColumns { get; }
}
```

### 2. Example Merge Configuration

Location: `src/JdeScoping.DataSync/Configuration/MergeConfigurations/WorkOrderMergeConfiguration.cs`

```csharp
public class WorkOrderMergeConfiguration : IMergeConfiguration<WorkOrder>
{
    public string TableName => "WorkOrder";

    public Expression<Func<WorkOrder, object>> MatchOn =>
        x => new { x.WorkOrderNumber, x.BranchCode };

    public Expression<Func<WorkOrder, object>>? UpdateColumns =>
        x => new { x.Status, x.Quantity, x.LastUpdateDt };

    public Expression<Func<WorkOrder, WorkOrder, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<WorkOrder, object>>? InsertColumns => null; // All columns
}
```

### 3. Extended IBulkMergeHelper

Added to: `src/JdeScoping.DataSync/Contracts/IBulkMergeHelper.cs`

```csharp
/// <summary>
/// Performs a mass insert (full table refresh) with optional index management.
/// Truncates table, disables non-clustered indexes, bulk copies data, rebuilds indexes.
/// </summary>
Task<MassInsertResult> MassInsertAsync<T>(
    IAsyncEnumerable<T> data,
    string destinationTable,
    bool rebuildIndexes = true,
    int batchSize = 0,
    CancellationToken cancellationToken = default) where T : class;
```

New result type:
```csharp
public record MassInsertResult(
    long TotalRowsInserted,
    TimeSpan Elapsed,
    bool IndexesRebuilt);
```

### 4. MergeConfigurationRegistry

Location: `src/JdeScoping.DataSync/Services/MergeConfigurationRegistry.cs`

```csharp
public interface IMergeConfigurationRegistry
{
    IMergeConfiguration<T> GetConfiguration<T>() where T : class;
    bool HasConfiguration<T>() where T : class;
}

internal class MergeConfigurationRegistry : IMergeConfigurationRegistry
{
    private readonly IServiceProvider _serviceProvider;

    public MergeConfigurationRegistry(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }

    public IMergeConfiguration<T> GetConfiguration<T>() where T : class
    {
        return _serviceProvider.GetService<IMergeConfiguration<T>>()
            ?? throw new InvalidOperationException(
                $"No merge configuration registered for {typeof(T).Name}");
    }

    public bool HasConfiguration<T>() where T : class
    {
        return _serviceProvider.GetService<IMergeConfiguration<T>>() != null;
    }
}
```

### 5. Refactored TableSyncOperation

Key changes to `src/JdeScoping.DataSync/Services/TableSyncOperation.cs`:

- Remove `IStagingTableManager` dependency
- Add `IBulkMergeHelper` dependency
- Add `IMergeConfigurationRegistry` dependency
- Remove `ILotFinderRepository` dependency (no longer need `GetTableSpecAsync`)

```csharp
private async Task<long> ExecuteIncrementalUpdateAsync<T>(
    IAsyncEnumerable<T> data,
    DataUpdateTask task,
    CancellationToken ct) where T : class
{
    var config = _configRegistry.GetConfiguration<T>();

    var result = await _bulkMergeHelper.MergeAsync(
        data,
        config.TableName,
        config.MatchOn,
        config.UpdateColumns,
        config.UpdateWhen,
        config.InsertColumns,
        batchSize: _options.Value.BatchSize,
        cancellationToken: ct);

    return result.TotalRowsProcessed;
}

private async Task<long> ExecuteMassUpdateAsync<T>(
    IAsyncEnumerable<T> data,
    DataUpdateTask task,
    CancellationToken ct) where T : class
{
    var config = _configRegistry.GetConfiguration<T>();

    var result = await _bulkMergeHelper.MassInsertAsync(
        data,
        config.TableName,
        rebuildIndexes: task.ScheduleConfig.ReIndexData,
        batchSize: _options.Value.BulkCopyBatchSize,
        cancellationToken: ct);

    return result.TotalRowsInserted;
}
```

### 6. DI Registration

In `ServiceCollectionExtensions.cs`:

```csharp
// Merge configuration registry
services.AddSingleton<IMergeConfigurationRegistry, MergeConfigurationRegistry>();

// Merge configurations - explicit registration
services.AddSingleton<IMergeConfiguration<WorkOrder>, WorkOrderMergeConfiguration>();
services.AddSingleton<IMergeConfiguration<LotUsage>, LotUsageMergeConfiguration>();
services.AddSingleton<IMergeConfiguration<Item>, ItemMergeConfiguration>();
// ... one line per synced entity
```

---

## Source Generator Fix

The generator at `JdeScoping.DataSync.SourceGenerators` must produce `IDataReader` implementations.

**Verification steps:**
1. Confirm `.csproj` reference has `OutputItemType="Analyzer"` and `ReferenceOutputAssembly="false"`
2. Add diagnostic logging to generator
3. Ensure `[GenerateDataReader]` attribute is accessible
4. Generator emits `{EntityName}DataReader : IDataReader` per attributed entity

**Generated output pattern:**
```csharp
public sealed class WorkOrderDataReader : IDataReader
{
    private readonly IAsyncEnumerator<WorkOrder> _enumerator;
    private WorkOrder? _current;

    private static readonly string[] ColumnNames =
        ["WorkOrderNumber", "BranchCode", "Status", "Quantity", "LastUpdateDt"];

    public object GetValue(int i) => i switch
    {
        0 => _current!.WorkOrderNumber,
        1 => _current!.BranchCode,
        // ...
    };
}
```

---

## Code Removal (After Migration Verified)

**Delete:**
- `src/JdeScoping.DataSync/Contracts/IStagingTableManager.cs`
- `src/JdeScoping.DataSync/Services/StagingTableManager.cs`
- `tests/JdeScoping.DataSync.Tests/StagingTableManagerTests.cs`
- `tests/JdeScoping.DataSync.IntegrationTests/StagingTableManagerTests.cs`

**Update:**
- Remove `IStagingTableManager`/`StagingTableManager` from `ServiceCollectionExtensions.cs`
- Remove `GetTableSpecAsync` usage from DataSync (if no longer needed elsewhere)

---

## Testing Strategy

**Unit tests - new:**
- `MergeConfigurationRegistryTests`
- `BulkMergeHelper.MassInsertAsync` tests

**Unit tests - update:**
- `TableSyncOperationTests` - update mocks

**Integration tests:**
- `BulkMergeHelperTests` - add mass insert tests
- `TableSyncOperationTests` - end-to-end verification

**Source generator tests:**
- Verify generated `IDataReader` implementations
- Test with `SqlBulkCopy`

**Verification before old code removal:**
1. All existing tests pass
2. Integration tests verify same behavior
3. Manual smoke test if possible