jdescopingtool/openspec/changes/archive/2026-01-01-implement-excel-export/design.md

Excel Export Design

Overview

This document describes the architecture and implementation approach for the Excel export subsystem, including workbook generation, sheet generators, formatting patterns, and memory management.

Architecture

Component Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                        Search Processing                             │
│  (SearchProcessor produces SearchModel with populated results)       │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     IExcelExportService                              │
│  Task<byte[]> GenerateAsync(SearchModel, CancellationToken)          │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      ExcelExportService                              │
├─────────────────────────────────────────────────────────────────────┤
│  - ILogger<ExcelExportService>                                       │
│  - IOptions<ExcelExportOptions>                                      │
├─────────────────────────────────────────────────────────────────────┤
│  Uses:                                                               │
│  - CriteriaSheetGenerator                                            │
│  - AttributeTableWriter                                              │
│  - WorksheetProtector                                                │
│  - HeaderFormatter                                                   │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         ClosedXML                                    │
│  XLWorkbook → IXLWorksheet → IXLCell/IXLRange                        │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
                            byte[] output

Project Structure

NEW/src/JdeScoping.ExcelExport/
├── Attributes/
│   ├── OutputColumnAttribute.cs
│   └── OutputTableAttribute.cs
├── Configuration/
│   └── ExcelExportOptions.cs
├── Generators/
│   ├── CriteriaSheetGenerator.cs      # Search Criteria tab
│   ├── AttributeTableWriter.cs         # Generic attribute-driven table writer
│   └── DataEntryTemplateGenerator.cs   # Bulk upload templates
├── Formatting/
│   ├── HeaderFormatter.cs              # Header cell formatting
│   ├── ColumnFormatter.cs              # Column width and number format
│   └── WorksheetProtector.cs           # Password protection
├── Helpers/
│   └── OutputColumnCache.cs            # Cached reflection for column metadata
├── Models/
│   └── OutputColumn.cs                 # Column metadata model
├── Interfaces/
│   └── IExcelExportService.cs
├── ExcelExportService.cs
├── ServiceCollectionExtensions.cs
└── JdeScoping.ExcelExport.csproj

ClosedXML Workbook Generation

Library Selection: ClosedXML

Why ClosedXML over EPPlus:

Aspect	EPPlus v7+	ClosedXML
License	Commercial (Polyform NC)	MIT (fully free)
API Similarity	N/A (legacy v4 was LGPL)	Very similar to EPPlus v4
Maintenance	Active (commercial)	Active (community)
.NET 10 Support	Yes	Yes
NuGet Downloads	High	High

NuGet Package: ClosedXML (version 0.104.* or later)

API Migration Guide: EPPlus to ClosedXML

EPPlus (Legacy)	ClosedXML (New)
ExcelPackage	XLWorkbook
ExcelWorkbook	XLWorkbook (same object)
ExcelWorksheet	IXLWorksheet
ExcelRange	IXLRange or IXLCell
ExcelTable	IXLTable
worksheet.Cells[row, col]	worksheet.Cell(row, col)
worksheet.Tables.Add(range, name)	worksheet.Range(...).CreateTable(name) or AsTable()
TableStyles.Light18	XLTableTheme.TableStyleLight18
Color.Gainsboro	XLColor.Gainsboro
range.Style.Fill.BackgroundColor.SetColor(...)	range.Style.Fill.BackgroundColor = XLColor.X
worksheet.Column(col).AutoFit()	worksheet.Column(col).AdjustToContents()
worksheet.Protection.SetPassword(pwd)	worksheet.Protect(pwd)
worksheet.ProtectedRanges.Add(...)	Not needed (use cell unlock instead)

Workbook Generation Flow

public async Task<byte[]> GenerateAsync(
    SearchModel search,
    CancellationToken cancellationToken = default)
{
    using var scope = _logger.BeginScope(new Dictionary<string, object>
    {
        ["SearchId"] = search.Id,
        ["SearchName"] = search.Name
    });

    _logger.LogInformation("Starting Excel export generation");

    // ClosedXML operations are synchronous, wrap in Task.Run for non-blocking
    return await Task.Run(() =>
    {
        cancellationToken.ThrowIfCancellationRequested();

        using var workbook = new XLWorkbook();

        // 1. Always generate Search Criteria sheet (first tab)
        GenerateCriteriaSheet(workbook, search);

        // 2. Always generate Search Results sheet (second tab)
        GenerateResultsSheet(workbook, search.Results);

        // 3. Conditionally generate MIS Info sheet
        if (search.ExtractMisData && search.MisResults != null)
        {
            GenerateMisInfoSheet(workbook, search.MisResults);
        }

        // 4. Conditionally generate Investigation sheet
        if (search.ExtractMisData && search.MisNonMatchResults != null)
        {
            GenerateInvestigationSheet(workbook, search.MisNonMatchResults);
        }

        cancellationToken.ThrowIfCancellationRequested();

        // Save to byte array
        using var stream = new MemoryStream();
        workbook.SaveAs(stream);
        return stream.ToArray();

    }, cancellationToken);
}

Sheet Generators

Criteria Sheet Generator

The Search Criteria sheet documents all search parameters and execution metadata.

Structure:

Row 1:  [Search Name]     [value]
Row 2:  [User Name]       [value]
Row 3:  (blank)
Row 4:  [Submit timestamp] [MMM dd, yyyy hh:mm:ss tt EST]
Row 5:  [Start timestamp]  [MMM dd, yyyy hh:mm:ss tt EST]
Row 6:  [Completed timestamp] [MMM dd, yyyy hh:mm:ss tt EST]
Row 7:  (blank)
Row 8+: [Timespan Filter Table]
        (2 blank rows)
        [Work Order Filter Table]
        (2 blank rows)
        [Item Number Filter Table]
        ...
        [Extract MIS data?] [YES/NO]

Filter Table Order:

1. Timespan Filter
2. Work Order Filter
3. Item Number Filter
4. Profit Center Filter
5. Work Center Filter
6. Component Lot Filter
7. Operator Filter
8. Item/Operation/MIS Filter

Implementation Pattern:

private void GenerateCriteriaSheet(XLWorkbook workbook, SearchModel search)
{
    var worksheet = workbook.Worksheets.Add("Search Criteria");
    var row = 1;

    // Header rows
    ApplyHeaderFormat(worksheet.Cell(row, 1), "Search Name");
    worksheet.Cell(row, 2).Value = search.Name;

    ApplyHeaderFormat(worksheet.Cell(++row, 1), "User Name");
    worksheet.Cell(row, 2).Value = search.UserName;

    row++; // blank

    // Timestamps
    ApplyHeaderFormat(worksheet.Cell(++row, 1), "Submit timestamp");
    worksheet.Cell(row, 2).Value = $"{search.SubmitDT:MMM dd, yyyy hh:mm:ss tt} EST";

    // ... more timestamps ...

    // Filter tables
    row = WriteFilterTable(worksheet, ++row, CreateTimespanFilter(search));
    row = WriteFilterTable(worksheet, row + 3, search.WorkOrderFilter);
    row = WriteFilterTable(worksheet, row + 3, search.ItemNumberFilter);
    // ... more filter tables ...

    // Extract MIS data indicator
    var headerRange = worksheet.Range(row, 1, row, 2);
    ApplyHeaderFormat(headerRange, "Extract MIS data?", merge: true);
    worksheet.Cell(++row, 1).Value = search.ExtractMisData ? "YES" : "NO";

    // Auto-fit with 15% padding
    for (int col = 1; col <= 4; col++)
    {
        worksheet.Column(col).AdjustToContents();
        worksheet.Column(col).Width *= 1.15;
    }

    // Protection
    worksheet.Protect(_options.Value.CriteriaSheetPassword);
}

Attribute-Driven Table Writer

The AttributeTableWriter generates Excel tables from model collections using reflection on OutputColumnAttribute and OutputTableAttribute.

Column Ordering:

1. Sort by OutputColumnAttribute.Order (ascending)
2. Break ties alphabetically by property name

Implementation Pattern:

public class AttributeTableWriter
{
    private readonly OutputColumnCache _cache;

    public IXLTable WriteTable<T>(
        IXLWorksheet worksheet,
        int startRow,
        int startCol,
        IEnumerable<T> data,
        string? tableNameOverride = null)
    {
        var tableAttr = typeof(T).GetCustomAttribute<OutputTableAttribute>();
        var columns = _cache.GetColumns<T>();
        var tableName = tableNameOverride ?? tableAttr?.TableName ?? typeof(T).Name;

        // Write header row
        var col = startCol;
        foreach (var column in columns)
        {
            var cell = worksheet.Cell(startRow, col);
            cell.Value = column.Attribute.HeaderText;
            ApplyHeaderFormat(cell);
            col++;
        }

        // Write data rows
        var dataList = data.ToList();
        var row = startRow + 1;
        foreach (var item in dataList)
        {
            col = startCol;
            foreach (var column in columns)
            {
                var value = column.Property.GetValue(item);
                worksheet.Cell(row, col).Value = XLCellValue.FromObject(value);
                col++;
            }
            row++;
        }

        // Create table
        var dataRange = worksheet.Range(
            startRow, startCol,
            startRow + dataList.Count, startCol + columns.Count - 1);
        var table = dataRange.CreateTable(tableName);
        table.Theme = XLTableTheme.TableStyleLight18;
        table.ShowTotalsRow = false;

        // Apply column formatting
        col = startCol;
        foreach (var column in columns)
        {
            ApplyColumnFormat(worksheet.Column(col), column.Attribute);
            col++;
        }

        return table;
    }
}

Data Sheet Generators (Results, MIS Info, Investigation)

These sheets use the AttributeTableWriter with model-specific configuration.

Search Results Sheet:

• 19 columns per SearchResult model
• Auto-fit with 30% padding
• No protection (legacy behavior via attribute-driven path)
• Light18 table style

MIS Info Sheet:

• 19 columns per MisSearchResult model
• Three columns with wrapped text (fixed 65-char width):
  • Test Description
  • Tools & Gauges
  • Work Instructions
• Other columns: auto-fit with 30% padding

Investigation Sheet:

• 12 columns per MisNonMatchSearchResult model
• Date columns use [$-409]MM/dd/yyyy;@ format
• Auto-fit with 30% padding

Column Formatting Patterns

Format Constants

public static class ExcelFormats
{
    public const string STD_FORMAT = "@";  // Text
    public const string DATE_FORMAT = "[$-409]MM/dd/yyyy;@";
    public const string TIMESTAMP_FORMAT = "[$-409]m/d/yy h:mm AM/PM;@";
    public const double WRAPPED_COLUMN_WIDTH = 65;
    public const double CRITERIA_PADDING_FACTOR = 1.15;
    public const double DATA_PADDING_FACTOR = 1.30;
}

Auto-Fit with Padding

private void ApplyColumnFormat(IXLColumn column, OutputColumnAttribute attr)
{
    // Set number format
    column.Style.NumberFormat.Format = attr.Format;

    if (attr.WrapText)
    {
        column.Style.Alignment.WrapText = true;
    }

    if (attr.AutoWidth)
    {
        column.AdjustToContents();
        column.Width *= ExcelFormats.DATA_PADDING_FACTOR;
    }
    else
    {
        column.Width = attr.Width;
    }
}

Wrapped Text Columns

Columns marked with WrapText = true and AutoWidth = false skip auto-fit:

if (attr.WrapText && !attr.AutoWidth)
{
    column.Width = attr.Width; // Usually 65
    column.Style.Alignment.WrapText = true;
    // Do NOT call AdjustToContents()
}

Worksheet Protection

Protection Configuration

public class ExcelExportOptions
{
    public const string SectionName = "ExcelExport";

    public string CriteriaSheetPassword { get; set; } = "JDE_SCOPING_TOOL_PASS";
    public string DataSheetPassword { get; set; } = "JDESCOPINGTOOL";
}

Allowed Operations

Protected worksheets allow specific operations:

private void ApplyProtection(IXLWorksheet worksheet, string password)
{
    var protection = worksheet.Protect(password);

    // Allow these operations
    protection.AllowElement(XLSheetProtectionElements.DeleteColumns);
    protection.AllowElement(XLSheetProtectionElements.AutoFilter);
    protection.AllowElement(XLSheetProtectionElements.FormatCells);
    protection.AllowElement(XLSheetProtectionElements.FormatColumns);
    protection.AllowElement(XLSheetProtectionElements.FormatRows);
    protection.AllowElement(XLSheetProtectionElements.SelectLockedCells);
    protection.AllowElement(XLSheetProtectionElements.SelectUnlockedCells);
    protection.AllowElement(XLSheetProtectionElements.EditObjects);
    protection.AllowElement(XLSheetProtectionElements.Sort);

    // DeleteRows is NOT allowed (not in AllowElement call)
}

Editable Extension Area

Legacy code unlocked an area beyond the data for user additions. In ClosedXML:

// Unlock cells beyond data range for user editing
var lastDataRow = table.RangeAddress.LastAddress.RowNumber;
var lastDataCol = table.RangeAddress.LastAddress.ColumnNumber;

// Unlock 1000 rows and columns beyond data
var extensionRange = worksheet.Range(
    1, lastDataCol + 1,
    lastDataRow + 1000, lastDataCol + 1000);
extensionRange.Style.Protection.Locked = false;

Note: The legacy code applies protection to data sheets (Search Results, MIS Info, Investigation) via the ApplySecurity method. The criteria sheet uses a different password.

Memory Management for Large Exports

Current Approach: In-Memory

The current implementation loads all data into memory and generates the workbook synchronously:

// All results loaded
var results = search.Results; // List<SearchResult>

// Workbook built in memory
using var workbook = new XLWorkbook();
// ... add sheets ...

// Serialize to byte[]
using var stream = new MemoryStream();
workbook.SaveAs(stream);
return stream.ToArray();

Memory Considerations

Export Size	Rows	Estimated Memory	Approach
Small	<10K	<50 MB	In-memory (current)
Medium	10K-100K	50-500 MB	In-memory (current)
Large	>100K	>500 MB	Consider streaming (future)

Future Optimization: Streaming

For very large exports, ClosedXML supports SAX-based streaming via XLWorkbook.SaveAsAsync(). This would require:

1. Writing sheets incrementally
2. Using IAsyncEnumerable<T> for data sources
3. Streaming directly to response or file

This is deferred to a future phase if memory pressure becomes an issue.

Temp File Handling

No Temp Files Required

The current implementation writes directly to MemoryStream and returns byte[]. The result is stored in the Search.Results column (VARBINARY) in the database.

If debugging is enabled (legacy behavior), a copy may be written to disk:

if (_options.Value.DebugWriteToFile)
{
    var debugPath = Path.Combine(
        _options.Value.DebugOutputDirectory,
        $"Search_{search.Id}_{DateTime.Now:yyyyMMdd_HHmmss}.xlsx");
    await File.WriteAllBytesAsync(debugPath, result, cancellationToken);
}

This is optional and controlled by configuration.

Header Formatting

Standard Header Style

private void ApplyHeaderFormat(IXLCell cell, string? text = null)
{
    cell.Style.Alignment.Horizontal = XLAlignmentHorizontalValues.Center;
    cell.Style.Font.Bold = true;
    cell.Style.Fill.BackgroundColor = XLColor.Gainsboro;

    if (!string.IsNullOrEmpty(text))
    {
        cell.Value = text;
    }
}

private void ApplyHeaderFormat(IXLRange range, string? text = null, bool merge = false)
{
    range.Style.Alignment.Horizontal = XLAlignmentHorizontalValues.Center;
    range.Style.Font.Bold = true;
    range.Style.Fill.BackgroundColor = XLColor.Gainsboro;

    if (merge)
    {
        range.Merge();
    }

    if (!string.IsNullOrEmpty(text))
    {
        range.FirstCell().Value = text;
    }
}

Data Entry Template Generator

For bulk data entry via the UI:

public class DataEntryTemplateGenerator
{
    public byte[] Generate<T>(IEnumerable<T>? sourceData, string headerText)
    {
        using var workbook = new XLWorkbook();
        var worksheet = workbook.Worksheets.Add("Data Entry Template");

        // Header
        var headerCell = worksheet.Cell(1, 1);
        ApplyHeaderFormat(headerCell, headerText);
        worksheet.Column(1).Width = 45;

        // Data (if provided)
        if (sourceData != null)
        {
            var row = 2;
            foreach (var item in sourceData)
            {
                worksheet.Cell(row++, 1).Value = XLCellValue.FromObject(item);
            }
        }

        // All cells as text
        worksheet.Column(1).Style.NumberFormat.Format = "@";

        using var stream = new MemoryStream();
        workbook.SaveAs(stream);
        return stream.ToArray();
    }

    public byte[] Generate(object[][]? sourceData, string[] headers)
    {
        using var workbook = new XLWorkbook();
        var worksheet = workbook.Worksheets.Add("Data Entry Template");

        // Headers
        for (int col = 0; col < headers.Length; col++)
        {
            ApplyHeaderFormat(worksheet.Cell(1, col + 1), headers[col]);
            worksheet.Column(col + 1).Width = 65;
            worksheet.Column(col + 1).Style.NumberFormat.Format = "@";
        }

        // Data
        if (sourceData != null)
        {
            for (int row = 0; row < sourceData.Length; row++)
            {
                for (int col = 0; col < sourceData[row].Length; col++)
                {
                    worksheet.Cell(row + 2, col + 1).Value =
                        XLCellValue.FromObject(sourceData[row][col]);
                }
            }
        }

        using var stream = new MemoryStream();
        workbook.SaveAs(stream);
        return stream.ToArray();
    }
}

Service Registration

public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddExcelExport(
        this IServiceCollection services,
        IConfiguration configuration)
    {
        // Bind options
        services.Configure<ExcelExportOptions>(
            configuration.GetSection(ExcelExportOptions.SectionName));

        // Register main service (scoped - per request)
        services.AddScoped<IExcelExportService, ExcelExportService>();

        // Register helpers (singleton - stateless)
        services.AddSingleton<OutputColumnCache>();
        services.AddSingleton<AttributeTableWriter>();
        services.AddSingleton<DataEntryTemplateGenerator>();

        return services;
    }
}

NuGet Dependencies

<ItemGroup>
  <PackageReference Include="ClosedXML" Version="0.104.*" />
  <PackageReference Include="Microsoft.Extensions.Options" Version="9.0.*" />
  <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.*" />
  <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.*" />
</ItemGroup>

Testing Strategy

Unit Tests

• Mock search models with various filter combinations
• Verify sheet generation logic without file I/O
• Test column ordering and formatting
• Test inclusion reason calculation
• Test null handling for optional sheets

Integration Tests

• Generate actual .xlsx files
• Open with ClosedXML to verify structure
• Compare against legacy output samples
• Test protection passwords
• Verify table styles applied correctly

Test Data

public static class ExcelExportTestData
{
    public static SearchModel CreateMinimalSearch() => new()
    {
        Id = 1,
        Name = "Test Search",
        UserName = "testuser",
        SubmitDT = DateTime.Now.AddHours(-1),
        StartDT = DateTime.Now.AddMinutes(-30),
        EndDT = DateTime.Now,
        ExtractMisData = false,
        Results = new List<SearchResult>
        {
            CreateSearchResult(12345, "ITEM-001")
        }
    };

    public static SearchModel CreateFullSearch() => new()
    {
        // ... includes MisResults and MisNonMatchResults
    };
}