jdescopingtool/PLANS/2026-01-03-etl-performance-optimization.md

ETL Performance Optimization Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Optimize dev ETL pipelines to significantly reduce load times for large tables (currently 18+ minutes for 88M rows)

Architecture: Four-pronged optimization: SqlBulkCopy tuning, parallel table loading, Utf8JsonReader parsing, and zstd buffer optimization

Tech Stack: .NET 10, System.Text.Json (Utf8JsonReader), System.Buffers (ArrayPool), SqlBulkCopy, ZstdSharp

---

Task 1: SqlBulkCopy Performance Tuning

Files:

• Modify: NEW/src/JdeScoping.DataSync/Etl/Destinations/DbBulkImportDestination.cs

Step 1: Add SqlBulkCopyOptions.TableLock

TableLock acquires a bulk update lock on the table during the bulk copy operation, reducing logging overhead.

using var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.TableLock, null)
{
    DestinationTableName = qualifiedName,
    BatchSize = _batchSize,
    BulkCopyTimeout = _commandTimeoutSeconds,
    EnableStreaming = true
};

Step 2: Increase default batch size

Change from 10,000 to 100,000 rows per batch to reduce round-trips.

private const int DefaultBatchSize = 100000;

Step 3: Reduce NotifyAfter frequency

Currently fires 8,800+ times for 88M rows. Reduce to every 10 batches.

bulkCopy.NotifyAfter = _batchSize * 10;

Step 4: Add option for infinite timeout

Add a constant for large table loads.

private const int InfiniteTimeout = 0;

Step 5: Verify changes compile

Run: dotnet build NEW/src/JdeScoping.DataSync/ Expected: Build succeeded

---

Task 2: Parallel Loading in DevEtlRegistry

Files:

• Modify: NEW/src/JdeScoping.DataSync/DevEtl/DevEtlRegistry.cs

Step 1: Add RunAllParallelAsync method

public async Task<IReadOnlyList<PipelineResult>> RunAllParallelAsync(
    int maxDegreeOfParallelism = 4,
    CancellationToken cancellationToken = default)
{
    var results = new ConcurrentBag<PipelineResult>();
    var semaphore = new SemaphoreSlim(maxDegreeOfParallelism);

    // Separate tables by size - run very large ones sequentially at the end
    var smallMediumTables = GetAvailableTables()
        .Where(t => !IsVeryLargeTable(t))
        .ToList();
    var veryLargeTables = GetAvailableTables()
        .Where(IsVeryLargeTable)
        .ToList();

    // Run small/medium tables in parallel
    var tasks = smallMediumTables.Select(async tableName =>
    {
        await semaphore.WaitAsync(cancellationToken);
        try
        {
            var result = await RunAsync(tableName, cancellationToken);
            results.Add(result);
        }
        finally
        {
            semaphore.Release();
        }
    });

    await Task.WhenAll(tasks);

    // Run very large tables sequentially (IO-bound, would contend)
    foreach (var tableName in veryLargeTables)
    {
        cancellationToken.ThrowIfCancellationRequested();
        var result = await RunAsync(tableName, cancellationToken);
        results.Add(result);
    }

    return results.ToList();
}

private static bool IsVeryLargeTable(string tableName) =>
    tableName.Contains("WorkOrderTime", StringComparison.OrdinalIgnoreCase) ||
    tableName.Contains("WorkOrderStep", StringComparison.OrdinalIgnoreCase) ||
    tableName.Contains("WorkOrderRouting", StringComparison.OrdinalIgnoreCase);

Step 2: Add required using statements

using System.Collections.Concurrent;

Step 3: Verify changes compile

Run: dotnet build NEW/src/JdeScoping.DataSync/ Expected: Build succeeded

---

Task 3: Utf8JsonReader-Based Streaming Parser

Files:

• Create: NEW/src/JdeScoping.DataSync/Etl/Sources/Utf8JsonStreamingDataReader.cs
• Modify: NEW/src/JdeScoping.DataSync/Etl/Sources/JsonZstdFileSource.cs

Step 1: Create Utf8JsonStreamingDataReader

This replaces the current char-by-char parsing with efficient Utf8JsonReader.

using System.Buffers;
using System.Data;
using System.Text;
using System.Text.Json;
using JdeScoping.DataSync.Etl.Models;

namespace JdeScoping.DataSync.Etl.Sources;

/// <summary>
/// High-performance streaming JSON array reader using Utf8JsonReader.
/// </summary>
internal sealed class Utf8JsonStreamingDataReader : IDataReader
{
    private const int DefaultBufferSize = 256 * 1024; // 256 KB

    private readonly Stream _stream;
    private readonly JsonColumnSchema[] _schema;
    private readonly Dictionary<string, int> _nameToOrdinal;
    private readonly byte[][] _encodedColumnNames;
    private byte[] _buffer;
    private int _bytesInBuffer;
    private int _bytesConsumed;
    private JsonReaderState _readerState;
    private object?[] _currentRow;
    private bool _disposed;
    private bool _started;
    private bool _finished;

    public Utf8JsonStreamingDataReader(Stream stream, JsonColumnSchema[] schema)
    {
        _stream = stream ?? throw new ArgumentNullException(nameof(stream));
        _schema = schema ?? throw new ArgumentNullException(nameof(schema));
        _buffer = ArrayPool<byte>.Shared.Rent(DefaultBufferSize);
        _currentRow = new object?[schema.Length];
        _readerState = new JsonReaderState();

        _nameToOrdinal = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
        _encodedColumnNames = new byte[schema.Length][];
        for (int i = 0; i < schema.Length; i++)
        {
            _nameToOrdinal[schema[i].Name] = i;
            _encodedColumnNames[i] = Encoding.UTF8.GetBytes(schema[i].Name);
        }
    }

    // ... IDataReader implementation
}

Step 2: Implement Read() with Utf8JsonReader

Key difference: Parse directly from byte buffer, no string allocation per object.

public bool Read()
{
    if (_disposed || _finished) return false;

    try
    {
        while (true)
        {
            var reader = new Utf8JsonReader(
                _buffer.AsSpan(_bytesConsumed, _bytesInBuffer - _bytesConsumed),
                isFinalBlock: false,
                _readerState);

            if (TryReadNextObject(ref reader))
            {
                _bytesConsumed += (int)reader.BytesConsumed;
                _readerState = reader.CurrentState;
                return true;
            }

            // Need more data
            if (!RefillBuffer())
            {
                // Final block
                reader = new Utf8JsonReader(
                    _buffer.AsSpan(_bytesConsumed, _bytesInBuffer - _bytesConsumed),
                    isFinalBlock: true,
                    _readerState);

                if (TryReadNextObject(ref reader))
                {
                    _bytesConsumed += (int)reader.BytesConsumed;
                    return true;
                }

                _finished = true;
                return false;
            }
        }
    }
    catch (JsonException ex)
    {
        throw new InvalidDataException($"Failed to parse JSON: {ex.Message}", ex);
    }
}

Step 3: Implement TryReadNextObject with ValueTextEquals

Use pre-encoded column names to avoid string allocations.

private bool TryReadNextObject(ref Utf8JsonReader reader)
{
    if (!_started)
    {
        if (!reader.Read() || reader.TokenType != JsonTokenType.StartArray)
            throw new InvalidDataException("Expected JSON array.");
        _started = true;
    }

    if (!reader.Read())
        return false;

    if (reader.TokenType == JsonTokenType.EndArray)
    {
        _finished = true;
        return false;
    }

    if (reader.TokenType != JsonTokenType.StartObject)
        throw new InvalidDataException($"Expected object, got {reader.TokenType}");

    Array.Clear(_currentRow);

    while (reader.Read() && reader.TokenType != JsonTokenType.EndObject)
    {
        if (reader.TokenType != JsonTokenType.PropertyName)
            continue;

        // Find matching column using pre-encoded names
        int ordinal = -1;
        for (int i = 0; i < _encodedColumnNames.Length; i++)
        {
            if (reader.ValueTextEquals(_encodedColumnNames[i]))
            {
                ordinal = i;
                break;
            }
        }

        if (!reader.Read())
            return false;

        if (ordinal >= 0)
        {
            _currentRow[ordinal] = ParseValue(ref reader, _schema[ordinal].ClrType);
        }
    }

    return reader.TokenType == JsonTokenType.EndObject;
}

Step 4: Implement RefillBuffer

private bool RefillBuffer()
{
    // Move unconsumed data to start of buffer
    var remaining = _bytesInBuffer - _bytesConsumed;
    if (remaining > 0)
    {
        Buffer.BlockCopy(_buffer, _bytesConsumed, _buffer, 0, remaining);
    }
    _bytesInBuffer = remaining;
    _bytesConsumed = 0;

    // Read more data
    var bytesRead = _stream.Read(_buffer, _bytesInBuffer, _buffer.Length - _bytesInBuffer);
    if (bytesRead == 0)
        return false;

    _bytesInBuffer += bytesRead;
    return true;
}

Step 5: Update JsonZstdFileSource to use new reader

Add constructor parameter to select reader implementation.

public JsonZstdFileSource(string filePath, JsonColumnSchema[] schema, bool useHighPerformanceReader = true)
{
    // ... existing validation ...
    _useHighPerformanceReader = useHighPerformanceReader;
}

public Task<IDataReader> ReadDataAsync(CancellationToken cancellationToken = default)
{
    // ... existing setup ...

    _reader = _useHighPerformanceReader
        ? new Utf8JsonStreamingDataReader(bufferedStream, _schema)
        : new JsonStreamingDataReader(bufferedStream, _schema);

    return Task.FromResult(_reader);
}

Step 6: Verify changes compile

Run: dotnet build NEW/src/JdeScoping.DataSync/ Expected: Build succeeded

---

Task 4: Zstd Buffer Optimizations

Files:

• Modify: NEW/src/JdeScoping.DataSync/Etl/Sources/JsonZstdFileSource.cs

Step 1: Add SequentialScan and sync IO

_fileStream = new FileStream(
    _filePath,
    FileMode.Open,
    FileAccess.Read,
    FileShare.Read,
    bufferSize: 256 * 1024,  // 256 KB
    FileOptions.SequentialScan);  // Hint for OS read-ahead

Step 2: Wrap DecompressionStream in BufferedStream

_decompressionStream = new DecompressionStream(_fileStream);
var bufferedStream = new BufferedStream(_decompressionStream, 256 * 1024);
_reader = new Utf8JsonStreamingDataReader(bufferedStream, _schema);

Step 3: Verify changes compile

Run: dotnet build NEW/src/JdeScoping.DataSync/ Expected: Build succeeded

---

Task 5: Integration Testing

Files:

• Test: NEW/tests/JdeScoping.DataSync.Tests/DevEtl/

Step 1: Run existing tests to ensure no regressions

Run: dotnet test NEW/tests/JdeScoping.DataSync.Tests/ --filter "DevEtl" Expected: All tests pass

Step 2: Test parallel loading

Create a simple console app to test:

var registry = new DevEtlRegistry(factory, cacheDir);
var sw = Stopwatch.StartNew();
var results = await registry.RunAllParallelAsync(maxDegreeOfParallelism: 4);
Console.WriteLine($"Total time: {sw.Elapsed}");
foreach (var r in results.OrderByDescending(r => r.TotalRows))
    Console.WriteLine($"{r.SourceName}: {r.TotalRows:N0} rows in {r.Elapsed}");

---

Expected Performance Improvements

Optimization	Expected Impact
TableLock	20-40% faster for large tables (reduced logging)
Batch size 100k	10-20% faster (fewer round-trips)
Utf8JsonReader	30-50% faster parsing (zero-alloc)
Parallel loading	2-3x faster for full load (4 parallel)
Buffer optimizations	5-10% faster IO

Combined estimate: Full load should drop from ~45 minutes to ~15-20 minutes.