Files

T

Joseph Doherty 26ff8d9b4f Initial commit: JDE Scoping Tool migration project

Set up repository with legacy .NET Framework 4.8 source (OLD/),
new .NET 10 Blazor solution (NEW/), OpenSpec specifications,
documentation, and project configuration.

2026-01-02 07:43:29 -05:00

48 KiB

Raw Blame History

TableSyncOperation Migration Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Migrate TableSyncOperation from IStagingTableManager/TableSpec to IBulkMergeHelper/IMergeConfiguration, then remove old staging table code.

Architecture: Create fluent merge configurations for each synced entity type. Extend BulkMergeHelper with mass insert capability (truncate + bulk copy + index management). Refactor TableSyncOperation to use new components. Remove legacy staging table code.

Tech Stack: .NET 10, Expression trees, SqlBulkCopy, SQL Server MERGE statements

Task 1: Create IMergeConfiguration Interface

Files:

Create: src/JdeScoping.DataSync/Contracts/IMergeConfiguration.cs

Step 1: Create the interface file

using System.Linq.Expressions;

namespace JdeScoping.DataSync.Contracts;

/// <summary>
/// Defines merge configuration for an entity type.
/// </summary>
/// <typeparam name="T">The entity type.</typeparam>
public interface IMergeConfiguration<T> where T : class
{
    /// <summary>
    /// Gets the destination table name in SQL Server.
    /// </summary>
    string TableName { get; }

    /// <summary>
    /// Gets the expression defining columns to match on (primary key).
    /// </summary>
    Expression<Func<T, object>> MatchOn { get; }

    /// <summary>
    /// Gets the expression defining columns to update when matched.
    /// Null means all non-PK columns.
    /// </summary>
    Expression<Func<T, object>>? UpdateColumns { get; }

    /// <summary>
    /// Gets the condition for when to perform updates.
    /// Null means always update on match.
    /// </summary>
    Expression<Func<T, T, bool>>? UpdateWhen { get; }

    /// <summary>
    /// Gets the expression defining columns to insert.
    /// Null means all columns.
    /// </summary>
    Expression<Func<T, object>>? InsertColumns { get; }
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Contracts/IMergeConfiguration.cs
git commit -m "feat(datasync): add IMergeConfiguration interface for fluent merge config"

Task 2: Create IMergeConfigurationRegistry

Files:

Create: src/JdeScoping.DataSync/Contracts/IMergeConfigurationRegistry.cs
Create: src/JdeScoping.DataSync/Services/MergeConfigurationRegistry.cs

Step 1: Create the interface

namespace JdeScoping.DataSync.Contracts;

/// <summary>
/// Registry for looking up merge configurations by entity type.
/// </summary>
public interface IMergeConfigurationRegistry
{
    /// <summary>
    /// Gets the merge configuration for the specified entity type.
    /// </summary>
    /// <typeparam name="T">The entity type.</typeparam>
    /// <returns>The merge configuration.</returns>
    /// <exception cref="InvalidOperationException">Thrown if no configuration is registered.</exception>
    IMergeConfiguration<T> GetConfiguration<T>() where T : class;

    /// <summary>
    /// Checks if a merge configuration exists for the specified entity type.
    /// </summary>
    /// <typeparam name="T">The entity type.</typeparam>
    /// <returns>True if configuration exists.</returns>
    bool HasConfiguration<T>() where T : class;
}

Step 2: Create the implementation

namespace JdeScoping.DataSync.Services;

/// <summary>
/// Registry implementation that resolves configurations from DI.
/// </summary>
internal sealed class MergeConfigurationRegistry : IMergeConfigurationRegistry
{
    private readonly IServiceProvider _serviceProvider;

    public MergeConfigurationRegistry(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
    }

    public IMergeConfiguration<T> GetConfiguration<T>() where T : class
    {
        var config = _serviceProvider.GetService(typeof(IMergeConfiguration<T>)) as IMergeConfiguration<T>;
        return config ?? throw new InvalidOperationException(
            $"No merge configuration registered for {typeof(T).Name}. " +
            $"Register IMergeConfiguration<{typeof(T).Name}> in ServiceCollectionExtensions.");
    }

    public bool HasConfiguration<T>() where T : class
    {
        return _serviceProvider.GetService(typeof(IMergeConfiguration<T>)) != null;
    }
}

Step 3: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 4: Commit

git add src/JdeScoping.DataSync/Contracts/IMergeConfigurationRegistry.cs
git add src/JdeScoping.DataSync/Services/MergeConfigurationRegistry.cs
git commit -m "feat(datasync): add MergeConfigurationRegistry for DI-based config lookup"

Task 3: Add MassInsertResult Model

Files:

Modify: src/JdeScoping.DataSync/Models/MergeResult.cs

Step 1: Add MassInsertResult record

Add to the end of the file:

/// <summary>
/// Result of a mass insert operation.
/// </summary>
/// <param name="TotalRowsInserted">Total rows inserted.</param>
/// <param name="Elapsed">Total elapsed time.</param>
/// <param name="IndexesRebuilt">Whether indexes were rebuilt (vs just re-enabled).</param>
public record MassInsertResult(
    long TotalRowsInserted,
    TimeSpan Elapsed,
    bool IndexesRebuilt);

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Models/MergeResult.cs
git commit -m "feat(datasync): add MassInsertResult model"

Task 4: Add MassInsertAsync to IBulkMergeHelper

Files:

Modify: src/JdeScoping.DataSync/Contracts/IBulkMergeHelper.cs

Step 1: Add the method signature

Add after the existing MergeAsync method:

    /// <summary>
    /// Performs a mass insert (full table refresh) with optional index management.
    /// Truncates table, disables non-clustered indexes, bulk copies data, rebuilds indexes.
    /// </summary>
    /// <typeparam name="T">The entity type.</typeparam>
    /// <param name="data">The source data to insert.</param>
    /// <param name="destinationTable">The destination SQL table name.</param>
    /// <param name="rebuildIndexes">If true, rebuilds indexes after insert. If false, just re-enables them.</param>
    /// <param name="batchSize">Number of rows per bulk copy batch. 0 = default (10000).</param>
    /// <param name="cancellationToken">Cancellation token.</param>
    /// <returns>Result containing row count and timing.</returns>
    Task<MassInsertResult> MassInsertAsync<T>(
        IAsyncEnumerable<T> data,
        string destinationTable,
        bool rebuildIndexes = true,
        int batchSize = 0,
        CancellationToken cancellationToken = default) where T : class;

Step 2: Verify build fails (TDD)

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build fails - BulkMergeHelper doesn't implement MassInsertAsync

Step 3: Commit interface change

git add src/JdeScoping.DataSync/Contracts/IBulkMergeHelper.cs
git commit -m "feat(datasync): add MassInsertAsync to IBulkMergeHelper interface"

Task 5: Implement MassInsertAsync in BulkMergeHelper

Files:

Modify: src/JdeScoping.DataSync/Services/BulkMergeHelper.cs

Step 1: Add the implementation

Add after the existing MergeAsync method (around line 172):

    /// <inheritdoc />
    public async Task<MassInsertResult> MassInsertAsync<T>(
        IAsyncEnumerable<T> data,
        string destinationTable,
        bool rebuildIndexes = true,
        int batchSize = 0,
        CancellationToken cancellationToken = default) where T : class
    {
        ArgumentNullException.ThrowIfNull(data);
        ArgumentException.ThrowIfNullOrWhiteSpace(destinationTable);

        var stopwatch = Stopwatch.StartNew();
        var effectiveBatchSize = batchSize > 0 ? batchSize : DefaultBatchSize;

        _logger.LogInformation(
            "Starting mass insert to {DestinationTable}. BatchSize={BatchSize}, RebuildIndexes={RebuildIndexes}",
            destinationTable, effectiveBatchSize, rebuildIndexes);

        await using var connection = await _connectionFactory.CreateLotFinderConnectionAsync(cancellationToken);

        long totalRows = 0;

        try
        {
            // Step 1: Disable non-clustered indexes
            await DisableNonClusteredIndexesAsync(connection, destinationTable, cancellationToken);

            // Step 2: Truncate the table
            await connection.ExecuteAsync(
                new CommandDefinition($"TRUNCATE TABLE [{destinationTable}]", cancellationToken: cancellationToken));

            _logger.LogDebug("Truncated table {DestinationTable}", destinationTable);

            // Step 3: Bulk copy data in batches
            var batch = new List<T>(effectiveBatchSize);

            await foreach (var item in data.WithCancellation(cancellationToken))
            {
                batch.Add(item);

                if (batch.Count >= effectiveBatchSize)
                {
                    await BulkCopyDirectAsync(connection, batch, destinationTable, cancellationToken);
                    totalRows += batch.Count;
                    _logger.LogDebug("Inserted batch of {Count} rows to {Table}", batch.Count, destinationTable);
                    batch.Clear();
                }
            }

            // Insert remaining rows
            if (batch.Count > 0)
            {
                await BulkCopyDirectAsync(connection, batch, destinationTable, cancellationToken);
                totalRows += batch.Count;
                _logger.LogDebug("Inserted final batch of {Count} rows to {Table}", batch.Count, destinationTable);
            }

            // Step 4: Rebuild or re-enable indexes
            if (rebuildIndexes)
            {
                await RebuildIndexesAsync(connection, destinationTable, cancellationToken);
            }
            else
            {
                await EnableNonClusteredIndexesAsync(connection, destinationTable, cancellationToken);
            }

            stopwatch.Stop();

            _logger.LogInformation(
                "Mass insert to {DestinationTable} completed. TotalRows={TotalRows}, Elapsed={Elapsed}ms",
                destinationTable, totalRows, stopwatch.ElapsedMilliseconds);

            return new MassInsertResult(totalRows, stopwatch.Elapsed, rebuildIndexes);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            _logger.LogError(ex, "Mass insert to {DestinationTable} failed after {TotalRows} rows",
                destinationTable, totalRows);

            // Try to re-enable indexes even on failure
            try
            {
                await EnableNonClusteredIndexesAsync(connection, destinationTable, CancellationToken.None);
            }
            catch (Exception indexEx)
            {
                _logger.LogWarning(indexEx, "Failed to re-enable indexes on {Table} after error", destinationTable);
            }

            throw;
        }
    }

    private async Task BulkCopyDirectAsync<T>(
        SqlConnection connection,
        IReadOnlyList<T> batch,
        string destinationTable,
        CancellationToken cancellationToken) where T : class
    {
        using var bulkCopy = new SqlBulkCopy(connection)
        {
            DestinationTableName = $"[{destinationTable}]",
            BatchSize = batch.Count,
            BulkCopyTimeout = 3600
        };

        using var reader = _dataReaderFactory.CreateReader(batch.ToAsyncEnumerable());
        await bulkCopy.WriteToServerAsync(reader, cancellationToken);
    }

    private static async Task DisableNonClusteredIndexesAsync(
        SqlConnection connection,
        string tableName,
        CancellationToken cancellationToken)
    {
        var sql = $@"
DECLARE @sql NVARCHAR(MAX) = '';
SELECT @sql = @sql + 'ALTER INDEX [' + i.name + '] ON [{tableName}] DISABLE;' + CHAR(13)
FROM sys.indexes i
INNER JOIN sys.tables t ON i.object_id = t.object_id
WHERE t.name = @tableName
  AND i.type = 2
  AND i.is_disabled = 0;
EXEC sp_executesql @sql;";

        await connection.ExecuteAsync(
            new CommandDefinition(sql, new { tableName }, commandTimeout: 300, cancellationToken: cancellationToken));
    }

    private static async Task EnableNonClusteredIndexesAsync(
        SqlConnection connection,
        string tableName,
        CancellationToken cancellationToken)
    {
        var sql = $@"
DECLARE @sql NVARCHAR(MAX) = '';
SELECT @sql = @sql + 'ALTER INDEX [' + i.name + '] ON [{tableName}] REBUILD;' + CHAR(13)
FROM sys.indexes i
INNER JOIN sys.tables t ON i.object_id = t.object_id
WHERE t.name = @tableName
  AND i.type = 2
  AND i.is_disabled = 1;
EXEC sp_executesql @sql;";

        await connection.ExecuteAsync(
            new CommandDefinition(sql, new { tableName }, commandTimeout: 3600, cancellationToken: cancellationToken));
    }

    private static async Task RebuildIndexesAsync(
        SqlConnection connection,
        string tableName,
        CancellationToken cancellationToken)
    {
        var sql = $"ALTER INDEX ALL ON [{tableName}] REBUILD WITH (FILLFACTOR = 95)";

        await connection.ExecuteAsync(
            new CommandDefinition(sql, commandTimeout: 3600, cancellationToken: cancellationToken));
    }

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Services/BulkMergeHelper.cs
git commit -m "feat(datasync): implement MassInsertAsync with index management"

Task 6: Create Merge Configurations Directory

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/ (directory)

Step 1: Create directory

Run: mkdir -p src/JdeScoping.DataSync/Configuration/MergeConfigurations

Step 2: Commit

git add .
git commit -m "chore(datasync): add MergeConfigurations directory"

Task 7: Create WorkOrder Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/WorkOrderMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.WorkOrders;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for WorkOrder entities.
/// </summary>
public sealed class WorkOrderMergeConfiguration : IMergeConfiguration<WorkOrder>
{
    public string TableName => "WorkOrder";

    public Expression<Func<WorkOrder, object>> MatchOn =>
        x => new { x.WorkOrderNumber, x.BranchCode };

    public Expression<Func<WorkOrder, object>>? UpdateColumns =>
        x => new
        {
            x.LotNumber,
            x.ItemNumber,
            x.ShortItemNumber,
            x.ParentWorkOrderNumber,
            x.OrderQuantity,
            x.HeldQuantity,
            x.ShippedQuantity,
            x.StatusCode,
            x.StatusCodeUpdateDt,
            x.IssueDate,
            x.StartDate,
            x.RoutingType
        };

    public Expression<Func<WorkOrder, WorkOrder, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<WorkOrder, object>>? InsertColumns => null; // All columns
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/WorkOrderMergeConfiguration.cs
git commit -m "feat(datasync): add WorkOrderMergeConfiguration"

Task 8: Create Lot Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/LotMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Inventory;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for Lot entities.
/// </summary>
public sealed class LotMergeConfiguration : IMergeConfiguration<Lot>
{
    public string TableName => "Lot";

    public Expression<Func<Lot, object>> MatchOn =>
        x => new { x.LotNumber, x.BranchCode };

    public Expression<Func<Lot, object>>? UpdateColumns =>
        x => new
        {
            x.ShortItemNumber,
            x.ItemNumber,
            x.SupplierCode,
            x.StatusCode,
            x.Memo1,
            x.Memo2,
            x.Memo3
        };

    public Expression<Func<Lot, Lot, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<Lot, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/LotMergeConfiguration.cs
git commit -m "feat(datasync): add LotMergeConfiguration"

Task 9: Create LotUsage Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/LotUsageMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Inventory;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for LotUsage entities.
/// </summary>
public sealed class LotUsageMergeConfiguration : IMergeConfiguration<LotUsage>
{
    public string TableName => "LotUsage";

    public Expression<Func<LotUsage, object>> MatchOn =>
        x => x.UniqueId;

    public Expression<Func<LotUsage, object>>? UpdateColumns =>
        x => new
        {
            x.WorkOrderNumber,
            x.LotNumber,
            x.BranchCode,
            x.ShortItemNumber,
            x.Quantity
        };

    public Expression<Func<LotUsage, LotUsage, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<LotUsage, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/LotUsageMergeConfiguration.cs
git commit -m "feat(datasync): add LotUsageMergeConfiguration"

Task 10: Create Item Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/ItemMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Inventory;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for Item entities.
/// </summary>
public sealed class ItemMergeConfiguration : IMergeConfiguration<Item>
{
    public string TableName => "Item";

    public Expression<Func<Item, object>> MatchOn =>
        x => x.ShortItemNumber;

    public Expression<Func<Item, object>>? UpdateColumns =>
        x => new
        {
            x.ItemNumber,
            x.Description,
            x.PlanningFamily,
            x.StockingType
        };

    public Expression<Func<Item, Item, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<Item, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/ItemMergeConfiguration.cs
git commit -m "feat(datasync): add ItemMergeConfiguration"

Task 11: Create WorkCenter Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/WorkCenterMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Organization;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for WorkCenter entities.
/// </summary>
public sealed class WorkCenterMergeConfiguration : IMergeConfiguration<WorkCenter>
{
    public string TableName => "WorkCenter";

    public Expression<Func<WorkCenter, object>> MatchOn =>
        x => x.Code;

    public Expression<Func<WorkCenter, object>>? UpdateColumns =>
        x => x.Description;

    public Expression<Func<WorkCenter, WorkCenter, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<WorkCenter, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/WorkCenterMergeConfiguration.cs
git commit -m "feat(datasync): add WorkCenterMergeConfiguration"

Task 12: Create ProfitCenter Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/ProfitCenterMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Organization;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for ProfitCenter entities.
/// </summary>
public sealed class ProfitCenterMergeConfiguration : IMergeConfiguration<ProfitCenter>
{
    public string TableName => "ProfitCenter";

    public Expression<Func<ProfitCenter, object>> MatchOn =>
        x => x.Code;

    public Expression<Func<ProfitCenter, object>>? UpdateColumns =>
        x => x.Description;

    public Expression<Func<ProfitCenter, ProfitCenter, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<ProfitCenter, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/ProfitCenterMergeConfiguration.cs
git commit -m "feat(datasync): add ProfitCenterMergeConfiguration"

Task 13: Create JdeUser Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/JdeUserMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Organization;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for JdeUser entities.
/// </summary>
public sealed class JdeUserMergeConfiguration : IMergeConfiguration<JdeUser>
{
    public string TableName => "JdeUser";

    public Expression<Func<JdeUser, object>> MatchOn =>
        x => x.AddressNumber;

    public Expression<Func<JdeUser, object>>? UpdateColumns =>
        x => new { x.UserId, x.FullName };

    public Expression<Func<JdeUser, JdeUser, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<JdeUser, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/JdeUserMergeConfiguration.cs
git commit -m "feat(datasync): add JdeUserMergeConfiguration"

Task 14: Create Branch Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/BranchMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Organization;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for Branch entities.
/// </summary>
public sealed class BranchMergeConfiguration : IMergeConfiguration<Branch>
{
    public string TableName => "Branch";

    public Expression<Func<Branch, object>> MatchOn =>
        x => x.Code;

    public Expression<Func<Branch, object>>? UpdateColumns =>
        x => x.Description;

    public Expression<Func<Branch, Branch, bool>>? UpdateWhen =>
        (src, tgt) => src.LastUpdateDt > tgt.LastUpdateDt;

    public Expression<Func<Branch, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/BranchMergeConfiguration.cs
git commit -m "feat(datasync): add BranchMergeConfiguration"

Task 15: Create MisData Merge Configuration

Files:

Create: src/JdeScoping.DataSync/Configuration/MergeConfigurations/MisDataMergeConfiguration.cs

Step 1: Create the configuration

using System.Linq.Expressions;
using JdeScoping.Core.Models.Quality;
using JdeScoping.DataSync.Contracts;

namespace JdeScoping.DataSync.Configuration.MergeConfigurations;

/// <summary>
/// Merge configuration for MisData entities.
/// </summary>
public sealed class MisDataMergeConfiguration : IMergeConfiguration<MisData>
{
    public string TableName => "MisData";

    public Expression<Func<MisData, object>> MatchOn =>
        x => new { x.ItemNumber, x.BranchCode, x.SequenceNumber, x.MisNumber, x.CharNumber };

    public Expression<Func<MisData, object>>? UpdateColumns =>
        x => new
        {
            x.RevId,
            x.TestDescription,
            x.SamplingType,
            x.SamplingValue,
            x.ToolsGauges,
            x.WorkInstructions,
            x.Status,
            x.ReleaseDate
        };

    // MisData doesn't have LastUpdateDt, so always update on match
    public Expression<Func<MisData, MisData, bool>>? UpdateWhen => null;

    public Expression<Func<MisData, object>>? InsertColumns => null;
}

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/Configuration/MergeConfigurations/MisDataMergeConfiguration.cs
git commit -m "feat(datasync): add MisDataMergeConfiguration"

Task 16: Update DI Registration

Files:

Modify: src/JdeScoping.DataSync/DependencyInjection/ServiceCollectionExtensions.cs

Step 1: Add using statements at top of file

using JdeScoping.DataSync.Configuration.MergeConfigurations;

Step 2: Add merge configuration registrations after bulk merge services section (around line 53)

        // Register merge configuration registry
        services.AddSingleton<IMergeConfigurationRegistry, MergeConfigurationRegistry>();

        // Register merge configurations - explicit registration per entity
        services.AddSingleton<IMergeConfiguration<WorkOrder>, WorkOrderMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<Lot>, LotMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<LotUsage>, LotUsageMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<Item>, ItemMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<WorkCenter>, WorkCenterMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<ProfitCenter>, ProfitCenterMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<JdeUser>, JdeUserMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<Branch>, BranchMergeConfiguration>();
        services.AddSingleton<IMergeConfiguration<MisData>, MisDataMergeConfiguration>();

Step 3: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 4: Commit

git add src/JdeScoping.DataSync/DependencyInjection/ServiceCollectionExtensions.cs
git commit -m "feat(datasync): register merge configurations in DI"

Task 17: Add Unit Tests for MergeConfigurationRegistry

Files:

Create: tests/JdeScoping.DataSync.Tests/Services/MergeConfigurationRegistryTests.cs

Step 1: Create the test file

using JdeScoping.Core.Models.WorkOrders;
using JdeScoping.DataSync.Configuration.MergeConfigurations;
using JdeScoping.DataSync.Contracts;
using JdeScoping.DataSync.Services;
using Microsoft.Extensions.DependencyInjection;
using Shouldly;

namespace JdeScoping.DataSync.Tests.Services;

public class MergeConfigurationRegistryTests
{
    [Fact]
    public void GetConfiguration_RegisteredType_ReturnsConfiguration()
    {
        // Arrange
        var services = new ServiceCollection();
        services.AddSingleton<IMergeConfiguration<WorkOrder>, WorkOrderMergeConfiguration>();
        var provider = services.BuildServiceProvider();
        var registry = new MergeConfigurationRegistry(provider);

        // Act
        var config = registry.GetConfiguration<WorkOrder>();

        // Assert
        config.ShouldNotBeNull();
        config.TableName.ShouldBe("WorkOrder");
    }

    [Fact]
    public void GetConfiguration_UnregisteredType_ThrowsInvalidOperationException()
    {
        // Arrange
        var services = new ServiceCollection();
        var provider = services.BuildServiceProvider();
        var registry = new MergeConfigurationRegistry(provider);

        // Act & Assert
        var ex = Should.Throw<InvalidOperationException>(() => registry.GetConfiguration<UnregisteredEntity>());
        ex.Message.ShouldContain("UnregisteredEntity");
    }

    [Fact]
    public void HasConfiguration_RegisteredType_ReturnsTrue()
    {
        // Arrange
        var services = new ServiceCollection();
        services.AddSingleton<IMergeConfiguration<WorkOrder>, WorkOrderMergeConfiguration>();
        var provider = services.BuildServiceProvider();
        var registry = new MergeConfigurationRegistry(provider);

        // Act
        var result = registry.HasConfiguration<WorkOrder>();

        // Assert
        result.ShouldBeTrue();
    }

    [Fact]
    public void HasConfiguration_UnregisteredType_ReturnsFalse()
    {
        // Arrange
        var services = new ServiceCollection();
        var provider = services.BuildServiceProvider();
        var registry = new MergeConfigurationRegistry(provider);

        // Act
        var result = registry.HasConfiguration<UnregisteredEntity>();

        // Assert
        result.ShouldBeFalse();
    }

    private class UnregisteredEntity { }
}

Step 2: Verify tests pass

Run: dotnet test tests/JdeScoping.DataSync.Tests --filter "FullyQualifiedName~MergeConfigurationRegistryTests" Expected: All 4 tests pass

Step 3: Commit

git add tests/JdeScoping.DataSync.Tests/Services/MergeConfigurationRegistryTests.cs
git commit -m "test(datasync): add MergeConfigurationRegistry unit tests"

Task 18: Add Unit Tests for BulkMergeHelper.MassInsertAsync

Files:

Modify: tests/JdeScoping.DataSync.Tests/Services/BulkMergeHelperTests.cs

Step 1: Add MassInsertAsync test methods

Add these tests to the existing test class:

    #region MassInsertAsync Tests

    [Fact]
    public async Task MassInsertAsync_NullData_ThrowsArgumentNullException()
    {
        // Arrange
        var sut = CreateSut();

        // Act & Assert
        await Should.ThrowAsync<ArgumentNullException>(
            () => sut.MassInsertAsync<TestEntity>(null!, "TestTable"));
    }

    [Fact]
    public async Task MassInsertAsync_NullDestination_ThrowsArgumentException()
    {
        // Arrange
        var sut = CreateSut();
        var data = AsyncEnumerable.Empty<TestEntity>();

        // Act & Assert
        await Should.ThrowAsync<ArgumentException>(
            () => sut.MassInsertAsync(data, null!));
    }

    [Fact]
    public async Task MassInsertAsync_EmptyDestination_ThrowsArgumentException()
    {
        // Arrange
        var sut = CreateSut();
        var data = AsyncEnumerable.Empty<TestEntity>();

        // Act & Assert
        await Should.ThrowAsync<ArgumentException>(
            () => sut.MassInsertAsync(data, ""));
    }

    [Fact]
    public async Task MassInsertAsync_EmptyData_ReturnsZeroRows()
    {
        // Arrange
        SetupConnectionFactory();
        var sut = CreateSut();
        var data = AsyncEnumerable.Empty<TestEntity>();

        // Act
        var result = await sut.MassInsertAsync(data, "TestTable");

        // Assert
        result.TotalRowsInserted.ShouldBe(0);
    }

    #endregion

Step 2: Verify tests pass

Run: dotnet test tests/JdeScoping.DataSync.Tests --filter "FullyQualifiedName~BulkMergeHelperTests" Expected: All tests pass

Step 3: Commit

git add tests/JdeScoping.DataSync.Tests/Services/BulkMergeHelperTests.cs
git commit -m "test(datasync): add BulkMergeHelper.MassInsertAsync unit tests"

Task 19: Refactor TableSyncOperation - Update Dependencies

Files:

Modify: src/JdeScoping.DataSync/Services/TableSyncOperation.cs

Step 1: Update using statements and dependencies

Replace the class fields and constructor (lines 22-54) with:

public class TableSyncOperation : ITableSyncOperation
{
    private readonly IServiceProvider _serviceProvider;
    private readonly IDbConnectionFactory _connectionFactory;
    private readonly IDataUpdateRepository _updateRepository;
    private readonly IBulkMergeHelper _bulkMergeHelper;
    private readonly IMergeConfigurationRegistry _configRegistry;
    private readonly IOptions<DataSyncOptions> _options;
    private readonly ILogger<TableSyncOperation> _logger;
    private readonly DataSyncMetrics _metrics;

    /// <summary>
    /// Initializes a new instance of the <see cref="TableSyncOperation"/> class.
    /// </summary>
    public TableSyncOperation(
        IServiceProvider serviceProvider,
        IDbConnectionFactory connectionFactory,
        IDataUpdateRepository updateRepository,
        IBulkMergeHelper bulkMergeHelper,
        IMergeConfigurationRegistry configRegistry,
        IOptions<DataSyncOptions> options,
        ILogger<TableSyncOperation> logger,
        DataSyncMetrics metrics)
    {
        _serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
        _connectionFactory = connectionFactory ?? throw new ArgumentNullException(nameof(connectionFactory));
        _updateRepository = updateRepository ?? throw new ArgumentNullException(nameof(updateRepository));
        _bulkMergeHelper = bulkMergeHelper ?? throw new ArgumentNullException(nameof(bulkMergeHelper));
        _configRegistry = configRegistry ?? throw new ArgumentNullException(nameof(configRegistry));
        _options = options ?? throw new ArgumentNullException(nameof(options));
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _metrics = metrics ?? throw new ArgumentNullException(nameof(metrics));
    }

Step 2: Verify build fails

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build fails - references to removed dependencies

Step 3: Commit partial refactor

git add src/JdeScoping.DataSync/Services/TableSyncOperation.cs
git commit -m "refactor(datasync): update TableSyncOperation dependencies"

Task 20: Refactor TableSyncOperation - Update ExecuteSyncCoreAsync

Files:

Modify: src/JdeScoping.DataSync/Services/TableSyncOperation.cs

Step 1: Replace ExecuteSyncCoreAsync method

Replace the entire ExecuteSyncCoreAsync method with:

    /// <summary>
    /// Core sync logic that handles mass vs incremental updates.
    /// </summary>
    private async Task<long> ExecuteSyncCoreAsync(DataUpdateTask task, CancellationToken cancellationToken)
    {
        // Get the fetcher for this entity type
        var fetcherType = ResolveFetcherType(task.Config.FetcherTypeName);
        var fetcher = _serviceProvider.GetRequiredService(fetcherType);

        // Use reflection to call FetchAsync on the fetcher
        var fetchMethod = fetcher.GetType().GetMethod("FetchAsync")
            ?? throw new InvalidOperationException($"FetchAsync method not found on {fetcher.GetType().Name}");

        var asyncEnumerable = fetchMethod.Invoke(fetcher, [task.MinimumDt, cancellationToken]);

        // Get the element type for typed operations
        var (_, elementType) = FindGetAsyncEnumerator(asyncEnumerable!.GetType());
        if (elementType == null || !elementType.IsClass)
        {
            throw new InvalidOperationException(
                $"Fetcher element type must be a class: {asyncEnumerable.GetType().Name}");
        }

        // Handle mass update with truncation
        if (task.UpdateType == UpdateTypes.Mass && task.ScheduleConfig.PrepurgeData)
        {
            return await ExecuteMassUpdateAsync(
                asyncEnumerable!,
                task,
                elementType,
                cancellationToken);
        }

        // Handle incremental update with merge
        return await ExecuteIncrementalUpdateAsync(
            asyncEnumerable!,
            task,
            elementType,
            cancellationToken);
    }

Step 2: Commit

git add src/JdeScoping.DataSync/Services/TableSyncOperation.cs
git commit -m "refactor(datasync): update ExecuteSyncCoreAsync to use new components"

Task 21: Refactor TableSyncOperation - Replace Mass Update Methods

Files:

Modify: src/JdeScoping.DataSync/Services/TableSyncOperation.cs

Step 1: Replace ExecuteMassUpdateWithTruncateAsync and ExecuteMassUpdateTypedAsync

Replace both methods with:

    /// <summary>
    /// Executes mass update using BulkMergeHelper.
    /// </summary>
    private Task<long> ExecuteMassUpdateAsync(
        object asyncEnumerable,
        DataUpdateTask task,
        Type elementType,
        CancellationToken cancellationToken)
    {
        _logger.LogDebug("Executing mass update for {Table}", task.TableName);

        // Use typed helper to call MassInsertAsync with correct type parameter
        var helper = typeof(TableSyncOperation)
            .GetMethod(nameof(ExecuteMassUpdateTypedAsync), System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
            .MakeGenericMethod(elementType);

        return (Task<long>)helper.Invoke(this, [
            asyncEnumerable,
            task,
            cancellationToken
        ])!;
    }

    /// <summary>
    /// Typed helper for mass update.
    /// </summary>
    private async Task<long> ExecuteMassUpdateTypedAsync<T>(
        IAsyncEnumerable<T> data,
        DataUpdateTask task,
        CancellationToken cancellationToken) where T : class
    {
        var config = _configRegistry.GetConfiguration<T>();

        var result = await _bulkMergeHelper.MassInsertAsync(
            data,
            config.TableName,
            rebuildIndexes: task.ScheduleConfig.ReIndexData,
            batchSize: _options.Value.BulkCopyBatchSize,
            cancellationToken: cancellationToken);

        // Run post processor if configured
        await RunPostProcessorAsync(task, cancellationToken);

        return result.TotalRowsInserted;
    }

Step 2: Commit

git add src/JdeScoping.DataSync/Services/TableSyncOperation.cs
git commit -m "refactor(datasync): replace mass update with BulkMergeHelper.MassInsertAsync"

Task 22: Refactor TableSyncOperation - Replace Incremental Update Methods

Files:

Modify: src/JdeScoping.DataSync/Services/TableSyncOperation.cs

Step 1: Replace ExecuteIncrementalUpdateAsync and ProcessBatchAsync/ProcessBatchTypedAsync

Replace all three methods with:

    /// <summary>
    /// Executes incremental update using BulkMergeHelper.MergeAsync.
    /// </summary>
    private Task<long> ExecuteIncrementalUpdateAsync(
        object asyncEnumerable,
        DataUpdateTask task,
        Type elementType,
        CancellationToken cancellationToken)
    {
        _logger.LogDebug("Executing incremental update for {Table}", task.TableName);

        // Use typed helper to call MergeAsync with correct type parameter
        var helper = typeof(TableSyncOperation)
            .GetMethod(nameof(ExecuteIncrementalUpdateTypedAsync), System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
            .MakeGenericMethod(elementType);

        return (Task<long>)helper.Invoke(this, [
            asyncEnumerable,
            task,
            cancellationToken
        ])!;
    }

    /// <summary>
    /// Typed helper for incremental update.
    /// </summary>
    private async Task<long> ExecuteIncrementalUpdateTypedAsync<T>(
        IAsyncEnumerable<T> data,
        DataUpdateTask task,
        CancellationToken cancellationToken) where T : class
    {
        var config = _configRegistry.GetConfiguration<T>();

        var result = await _bulkMergeHelper.MergeAsync(
            data,
            config.TableName,
            config.MatchOn,
            config.UpdateColumns,
            config.UpdateWhen,
            config.InsertColumns,
            batchSize: _options.Value.BatchSize,
            cancellationToken: cancellationToken);

        // Run post processor if configured
        await RunPostProcessorAsync(task, cancellationToken);

        return result.TotalRowsProcessed;
    }

Step 2: Remove unused methods

Delete these methods (they are no longer needed):

CollectRecordsAsync

Step 3: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 4: Commit

git add src/JdeScoping.DataSync/Services/TableSyncOperation.cs
git commit -m "refactor(datasync): replace incremental update with BulkMergeHelper.MergeAsync"

Task 23: Update TableSyncOperation Unit Tests

Files:

Modify: tests/JdeScoping.DataSync.Tests/TableSyncOperationTests.cs

Step 1: Update test class dependencies

Replace the test class fields and constructor with:

public class TableSyncOperationTests
{
    private readonly IDbConnectionFactory _connectionFactory;
    private readonly IDataUpdateRepository _updateRepository;
    private readonly IBulkMergeHelper _bulkMergeHelper;
    private readonly IMergeConfigurationRegistry _configRegistry;
    private readonly IOptions<DataSyncOptions> _options;
    private readonly DataSyncMetrics _metrics;
    private readonly IServiceProvider _serviceProvider;

    public TableSyncOperationTests()
    {
        _connectionFactory = Substitute.For<IDbConnectionFactory>();
        _updateRepository = Substitute.For<IDataUpdateRepository>();
        _bulkMergeHelper = Substitute.For<IBulkMergeHelper>();
        _configRegistry = Substitute.For<IMergeConfigurationRegistry>();

        _options = Options.Create(new DataSyncOptions
        {
            BatchSize = 1000,
            BulkCopyBatchSize = 100
        });

        var services = new ServiceCollection();
        services.AddMetrics();
        var provider = services.BuildServiceProvider();
        var meterFactory = provider.GetRequiredService<IMeterFactory>();
        _metrics = new DataSyncMetrics(meterFactory);

        _serviceProvider = Substitute.For<IServiceProvider>();
    }

Step 2: Update CreateSut helper method

    private TableSyncOperation CreateSut()
    {
        return new TableSyncOperation(
            _serviceProvider,
            _connectionFactory,
            _updateRepository,
            _bulkMergeHelper,
            _configRegistry,
            _options,
            NullLogger<TableSyncOperation>.Instance,
            _metrics);
    }

Step 3: Update test methods to use new mocks

Update the mass update test to mock IBulkMergeHelper.MassInsertAsync instead of IStagingTableManager.ExecuteMassUpdateAsync.

Step 4: Verify tests pass

Run: dotnet test tests/JdeScoping.DataSync.Tests --filter "FullyQualifiedName~TableSyncOperationTests" Expected: Tests pass (some may need adjustment)

Step 5: Commit

git add tests/JdeScoping.DataSync.Tests/TableSyncOperationTests.cs
git commit -m "test(datasync): update TableSyncOperationTests for new dependencies"

Task 24: Remove IStagingTableManager from DI Registration

Files:

Modify: src/JdeScoping.DataSync/DependencyInjection/ServiceCollectionExtensions.cs

Step 1: Remove the staging manager registration

Delete this line:

services.AddScoped<IStagingTableManager, StagingTableManager>();

Step 2: Verify build

Run: dotnet build src/JdeScoping.DataSync/JdeScoping.DataSync.csproj Expected: Build succeeded

Step 3: Commit

git add src/JdeScoping.DataSync/DependencyInjection/ServiceCollectionExtensions.cs
git commit -m "refactor(datasync): remove IStagingTableManager from DI"

Task 25: Delete StagingTableManager Files

Files:

Delete: src/JdeScoping.DataSync/Contracts/IStagingTableManager.cs
Delete: src/JdeScoping.DataSync/Services/StagingTableManager.cs
Delete: tests/JdeScoping.DataSync.Tests/StagingTableManagerTests.cs
Delete: tests/JdeScoping.DataSync.IntegrationTests/StagingTableManagerTests.cs

Step 1: Delete the files

rm src/JdeScoping.DataSync/Contracts/IStagingTableManager.cs
rm src/JdeScoping.DataSync/Services/StagingTableManager.cs
rm tests/JdeScoping.DataSync.Tests/StagingTableManagerTests.cs
rm tests/JdeScoping.DataSync.IntegrationTests/StagingTableManagerTests.cs

Step 2: Verify build

Run: dotnet build Expected: Build succeeded

Step 3: Verify all tests pass

Run: dotnet test Expected: All tests pass

Step 4: Commit

git add -A
git commit -m "refactor(datasync): remove StagingTableManager and related tests"

Task 26: Final Verification

Step 1: Full solution build

Run: dotnet build Expected: Build succeeded with no warnings

Step 2: Run all tests

Run: dotnet test Expected: All tests pass

Step 3: Review changes

Run: git log --oneline -15 Verify all commits are present

Step 4: Final commit summary

git log --oneline HEAD~20..HEAD

Summary

This plan migrates TableSyncOperation from the old IStagingTableManager/TableSpec approach to the new IBulkMergeHelper/IMergeConfiguration pattern:

Tasks 1-2: Create merge configuration interfaces
Tasks 3-5: Add MassInsertAsync to BulkMergeHelper
Tasks 6-15: Create merge configurations for all 9 entity types
Task 16: Update DI registration
Tasks 17-18: Add unit tests
Tasks 19-23: Refactor TableSyncOperation
Tasks 24-25: Remove old staging table code
Task 26: Final verification

48 KiB Raw Blame History

TableSyncOperation Migration Implementation Plan

Task 1: Create IMergeConfiguration Interface

Task 2: Create IMergeConfigurationRegistry

Task 3: Add MassInsertResult Model

Task 4: Add MassInsertAsync to IBulkMergeHelper

Task 5: Implement MassInsertAsync in BulkMergeHelper

Task 6: Create Merge Configurations Directory

Task 7: Create WorkOrder Merge Configuration

Task 8: Create Lot Merge Configuration

Task 9: Create LotUsage Merge Configuration

Task 10: Create Item Merge Configuration

Task 11: Create WorkCenter Merge Configuration

Task 12: Create ProfitCenter Merge Configuration

Task 13: Create JdeUser Merge Configuration

Task 14: Create Branch Merge Configuration

Task 15: Create MisData Merge Configuration

Task 16: Update DI Registration

Task 17: Add Unit Tests for MergeConfigurationRegistry

Task 18: Add Unit Tests for BulkMergeHelper.MassInsertAsync

Task 19: Refactor TableSyncOperation - Update Dependencies

Task 20: Refactor TableSyncOperation - Update ExecuteSyncCoreAsync

Task 21: Refactor TableSyncOperation - Replace Mass Update Methods

Task 22: Refactor TableSyncOperation - Replace Incremental Update Methods

Task 23: Update TableSyncOperation Unit Tests

Task 24: Remove IStagingTableManager from DI Registration

Task 25: Delete StagingTableManager Files

Task 26: Final Verification

Summary

48 KiB

Raw Blame History