jdescopingtool/PLANS/2026-01-01-tablesyncop-migration-design.md

TableSyncOperation Migration Design

Goal

Migrate TableSyncOperation from using IStagingTableManager with TableSpec metadata to using IBulkMergeHelper with expression-based IMergeConfiguration<T> classes. Remove old staging table code after migration.

Architecture Decisions

Decision	Choice	Rationale
Column metadata source	Expression lambdas	Compile-time safety, explicit configuration
Configuration style	Fluent IMergeConfiguration<T> classes	Clean Architecture - keeps domain entities clean
Configuration location	DataSync project	Infrastructure concern, not domain
Mass update support	Add to BulkMergeHelper	Unified bulk operations API
Index management	Built into BulkMergeHelper	Simple API, matches current behavior
DataReader generation	Source generator (fix first)	Best long-term solution, block until working
Old code removal	Full removal	Clean break, no dead code
DI registration	Explicit per-entity	Visibility and control

---

Component Design

1. IMergeConfiguration Interface

Location: src/JdeScoping.DataSync/Contracts/IMergeConfiguration.cs

public interface IMergeConfiguration<T> where T : class
{
    /// <summary>Table name in SQL Server cache.</summary>
    string TableName { get; }

    /// <summary>Columns to match on (primary key).</summary>
    Expression<Func<T, object>> MatchOn { get; }

    /// <summary>Columns to update when matched. Null = all non-PK columns.</summary>
    Expression<Func<T, object>>? UpdateColumns { get; }

    /// <summary>Condition for when to update. Null = always update on match.</summary>
    Expression<Func<T, T, bool>>? UpdateWhen { get; }

    /// <summary>Columns to insert. Null = all columns.</summary>
    Expression<Func<T, object>>? InsertColumns { get; }
}

2. Example Merge Configuration

Location: src/JdeScoping.DataSync/Configuration/MergeConfigurations/WorkOrderMergeConfiguration.cs

public class WorkOrderMergeConfiguration : IMergeConfiguration<WorkOrder>
{
    public string TableName => "WorkOrder";

    public Expression<Func<WorkOrder, object>> MatchOn =>
        x => new { x.WorkOrderNumber, x.BranchCode };

    public Expression<Func<WorkOrder, object>>? UpdateColumns =>
        x => new { x.Status, x.Quantity, x.LastUpdateDt };

    public Expression<Func<WorkOrder, WorkOrder, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<WorkOrder, object>>? InsertColumns => null; // All columns
}

3. Extended IBulkMergeHelper

Added to: src/JdeScoping.DataSync/Contracts/IBulkMergeHelper.cs

/// <summary>
/// Performs a mass insert (full table refresh) with optional index management.
/// Truncates table, disables non-clustered indexes, bulk copies data, rebuilds indexes.
/// </summary>
Task<MassInsertResult> MassInsertAsync<T>(
    IAsyncEnumerable<T> data,
    string destinationTable,
    bool rebuildIndexes = true,
    int batchSize = 0,
    CancellationToken cancellationToken = default) where T : class;

New result type:

public record MassInsertResult(
    long TotalRowsInserted,
    TimeSpan Elapsed,
    bool IndexesRebuilt);

4. MergeConfigurationRegistry

Location: src/JdeScoping.DataSync/Services/MergeConfigurationRegistry.cs

public interface IMergeConfigurationRegistry
{
    IMergeConfiguration<T> GetConfiguration<T>() where T : class;
    bool HasConfiguration<T>() where T : class;
}

internal class MergeConfigurationRegistry : IMergeConfigurationRegistry
{
    private readonly IServiceProvider _serviceProvider;

    public MergeConfigurationRegistry(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }

    public IMergeConfiguration<T> GetConfiguration<T>() where T : class
    {
        return _serviceProvider.GetService<IMergeConfiguration<T>>()
            ?? throw new InvalidOperationException(
                $"No merge configuration registered for {typeof(T).Name}");
    }

    public bool HasConfiguration<T>() where T : class
    {
        return _serviceProvider.GetService<IMergeConfiguration<T>>() != null;
    }
}

5. Refactored TableSyncOperation

Key changes to src/JdeScoping.DataSync/Services/TableSyncOperation.cs:

• Remove IStagingTableManager dependency
• Add IBulkMergeHelper dependency
• Add IMergeConfigurationRegistry dependency
• Remove ILotFinderRepository dependency (no longer need GetTableSpecAsync)

private async Task<long> ExecuteIncrementalUpdateAsync<T>(
    IAsyncEnumerable<T> data,
    DataUpdateTask task,
    CancellationToken ct) where T : class
{
    var config = _configRegistry.GetConfiguration<T>();

    var result = await _bulkMergeHelper.MergeAsync(
        data,
        config.TableName,
        config.MatchOn,
        config.UpdateColumns,
        config.UpdateWhen,
        config.InsertColumns,
        batchSize: _options.Value.BatchSize,
        cancellationToken: ct);

    return result.TotalRowsProcessed;
}

private async Task<long> ExecuteMassUpdateAsync<T>(
    IAsyncEnumerable<T> data,
    DataUpdateTask task,
    CancellationToken ct) where T : class
{
    var config = _configRegistry.GetConfiguration<T>();

    var result = await _bulkMergeHelper.MassInsertAsync(
        data,
        config.TableName,
        rebuildIndexes: task.ScheduleConfig.ReIndexData,
        batchSize: _options.Value.BulkCopyBatchSize,
        cancellationToken: ct);

    return result.TotalRowsInserted;
}

6. DI Registration

In ServiceCollectionExtensions.cs:

// Merge configuration registry
services.AddSingleton<IMergeConfigurationRegistry, MergeConfigurationRegistry>();

// Merge configurations - explicit registration
services.AddSingleton<IMergeConfiguration<WorkOrder>, WorkOrderMergeConfiguration>();
services.AddSingleton<IMergeConfiguration<LotUsage>, LotUsageMergeConfiguration>();
services.AddSingleton<IMergeConfiguration<Item>, ItemMergeConfiguration>();
// ... one line per synced entity

---

Source Generator Fix

The generator at JdeScoping.DataSync.SourceGenerators must produce IDataReader implementations.

Verification steps:

1. Confirm .csproj reference has OutputItemType="Analyzer" and ReferenceOutputAssembly="false"
2. Add diagnostic logging to generator
3. Ensure [GenerateDataReader] attribute is accessible
4. Generator emits {EntityName}DataReader : IDataReader per attributed entity

Generated output pattern:

public sealed class WorkOrderDataReader : IDataReader
{
    private readonly IAsyncEnumerator<WorkOrder> _enumerator;
    private WorkOrder? _current;

    private static readonly string[] ColumnNames =
        ["WorkOrderNumber", "BranchCode", "Status", "Quantity", "LastUpdateDt"];

    public object GetValue(int i) => i switch
    {
        0 => _current!.WorkOrderNumber,
        1 => _current!.BranchCode,
        // ...
    };
}

---

Code Removal (After Migration Verified)

Delete:

• src/JdeScoping.DataSync/Contracts/IStagingTableManager.cs
• src/JdeScoping.DataSync/Services/StagingTableManager.cs
• tests/JdeScoping.DataSync.Tests/StagingTableManagerTests.cs
• tests/JdeScoping.DataSync.IntegrationTests/StagingTableManagerTests.cs

Update:

• Remove IStagingTableManager/StagingTableManager from ServiceCollectionExtensions.cs
• Remove GetTableSpecAsync usage from DataSync (if no longer needed elsewhere)

---

Testing Strategy

Unit tests - new:

• MergeConfigurationRegistryTests
• BulkMergeHelper.MassInsertAsync tests

Unit tests - update:

• TableSyncOperationTests - update mocks

Integration tests:

• BulkMergeHelperTests - add mass insert tests
• TableSyncOperationTests - end-to-end verification

Source generator tests:

• Verify generated IDataReader implementations
• Test with SqlBulkCopy

Verification before old code removal:

1. All existing tests pass
2. Integration tests verify same behavior
3. Manual smoke test if possible