# Excel Export Design

## Overview

This document describes the architecture and implementation approach for the Excel export subsystem, including workbook generation, sheet generators, formatting patterns, and memory management.

## Architecture

### Component Diagram

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Search Processing                             │
│  (SearchProcessor produces SearchModel with populated results)       │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     IExcelExportService                              │
│  Task<byte[]> GenerateAsync(SearchModel, CancellationToken)          │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      ExcelExportService                              │
├─────────────────────────────────────────────────────────────────────┤
│  - ILogger<ExcelExportService>                                       │
│  - IOptions<ExcelExportOptions>                                      │
├─────────────────────────────────────────────────────────────────────┤
│  Uses:                                                               │
│  - CriteriaSheetGenerator                                            │
│  - AttributeTableWriter                                              │
│  - WorksheetProtector                                                │
│  - HeaderFormatter                                                   │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         ClosedXML                                    │
│  XLWorkbook → IXLWorksheet → IXLCell/IXLRange                        │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
                            byte[] output
```

### Project Structure

```
NEW/src/JdeScoping.ExcelExport/
├── Attributes/
│   ├── OutputColumnAttribute.cs
│   └── OutputTableAttribute.cs
├── Configuration/
│   └── ExcelExportOptions.cs
├── Generators/
│   ├── CriteriaSheetGenerator.cs      # Search Criteria tab
│   ├── AttributeTableWriter.cs         # Generic attribute-driven table writer
│   └── DataEntryTemplateGenerator.cs   # Bulk upload templates
├── Formatting/
│   ├── HeaderFormatter.cs              # Header cell formatting
│   ├── ColumnFormatter.cs              # Column width and number format
│   └── WorksheetProtector.cs           # Password protection
├── Helpers/
│   └── OutputColumnCache.cs            # Cached reflection for column metadata
├── Models/
│   └── OutputColumn.cs                 # Column metadata model
├── Interfaces/
│   └── IExcelExportService.cs
├── ExcelExportService.cs
├── ServiceCollectionExtensions.cs
└── JdeScoping.ExcelExport.csproj
```

## ClosedXML Workbook Generation

### Library Selection: ClosedXML

**Why ClosedXML over EPPlus:**

| Aspect | EPPlus v7+ | ClosedXML |
|--------|------------|-----------|
| License | Commercial (Polyform NC) | MIT (fully free) |
| API Similarity | N/A (legacy v4 was LGPL) | Very similar to EPPlus v4 |
| Maintenance | Active (commercial) | Active (community) |
| .NET 10 Support | Yes | Yes |
| NuGet Downloads | High | High |

**NuGet Package:** `ClosedXML` (version 0.104.* or later)

### API Migration Guide: EPPlus to ClosedXML

| EPPlus (Legacy) | ClosedXML (New) |
|-----------------|-----------------|
| `ExcelPackage` | `XLWorkbook` |
| `ExcelWorkbook` | `XLWorkbook` (same object) |
| `ExcelWorksheet` | `IXLWorksheet` |
| `ExcelRange` | `IXLRange` or `IXLCell` |
| `ExcelTable` | `IXLTable` |
| `worksheet.Cells[row, col]` | `worksheet.Cell(row, col)` |
| `worksheet.Tables.Add(range, name)` | `worksheet.Range(...).CreateTable(name)` or `AsTable()` |
| `TableStyles.Light18` | `XLTableTheme.TableStyleLight18` |
| `Color.Gainsboro` | `XLColor.Gainsboro` |
| `range.Style.Fill.BackgroundColor.SetColor(...)` | `range.Style.Fill.BackgroundColor = XLColor.X` |
| `worksheet.Column(col).AutoFit()` | `worksheet.Column(col).AdjustToContents()` |
| `worksheet.Protection.SetPassword(pwd)` | `worksheet.Protect(pwd)` |
| `worksheet.ProtectedRanges.Add(...)` | Not needed (use cell unlock instead) |

### Workbook Generation Flow

```csharp
public async Task<byte[]> GenerateAsync(
    SearchModel search,
    CancellationToken cancellationToken = default)
{
    using var scope = _logger.BeginScope(new Dictionary<string, object>
    {
        ["SearchId"] = search.Id,
        ["SearchName"] = search.Name
    });

    _logger.LogInformation("Starting Excel export generation");

    // ClosedXML operations are synchronous, wrap in Task.Run for non-blocking
    return await Task.Run(() =>
    {
        cancellationToken.ThrowIfCancellationRequested();

        using var workbook = new XLWorkbook();

        // 1. Always generate Search Criteria sheet (first tab)
        GenerateCriteriaSheet(workbook, search);

        // 2. Always generate Search Results sheet (second tab)
        GenerateResultsSheet(workbook, search.Results);

        // 3. Conditionally generate MIS Info sheet
        if (search.ExtractMisData && search.MisResults != null)
        {
            GenerateMisInfoSheet(workbook, search.MisResults);
        }

        // 4. Conditionally generate Investigation sheet
        if (search.ExtractMisData && search.MisNonMatchResults != null)
        {
            GenerateInvestigationSheet(workbook, search.MisNonMatchResults);
        }

        cancellationToken.ThrowIfCancellationRequested();

        // Save to byte array
        using var stream = new MemoryStream();
        workbook.SaveAs(stream);
        return stream.ToArray();

    }, cancellationToken);
}
```

## Sheet Generators

### Criteria Sheet Generator

The Search Criteria sheet documents all search parameters and execution metadata.

**Structure:**

```
Row 1:  [Search Name]     [value]
Row 2:  [User Name]       [value]
Row 3:  (blank)
Row 4:  [Submit timestamp] [MMM dd, yyyy hh:mm:ss tt EST]
Row 5:  [Start timestamp]  [MMM dd, yyyy hh:mm:ss tt EST]
Row 6:  [Completed timestamp] [MMM dd, yyyy hh:mm:ss tt EST]
Row 7:  (blank)
Row 8+: [Timespan Filter Table]
        (2 blank rows)
        [Work Order Filter Table]
        (2 blank rows)
        [Item Number Filter Table]
        ...
        [Extract MIS data?] [YES/NO]
```

**Filter Table Order:**
1. Timespan Filter
2. Work Order Filter
3. Item Number Filter
4. Profit Center Filter
5. Work Center Filter
6. Component Lot Filter
7. Operator Filter
8. Item/Operation/MIS Filter

**Implementation Pattern:**

```csharp
private void GenerateCriteriaSheet(XLWorkbook workbook, SearchModel search)
{
    var worksheet = workbook.Worksheets.Add("Search Criteria");
    var row = 1;

    // Header rows
    ApplyHeaderFormat(worksheet.Cell(row, 1), "Search Name");
    worksheet.Cell(row, 2).Value = search.Name;

    ApplyHeaderFormat(worksheet.Cell(++row, 1), "User Name");
    worksheet.Cell(row, 2).Value = search.UserName;

    row++; // blank

    // Timestamps
    ApplyHeaderFormat(worksheet.Cell(++row, 1), "Submit timestamp");
    worksheet.Cell(row, 2).Value = $"{search.SubmitDT:MMM dd, yyyy hh:mm:ss tt} EST";

    // ... more timestamps ...

    // Filter tables
    row = WriteFilterTable(worksheet, ++row, CreateTimespanFilter(search));
    row = WriteFilterTable(worksheet, row + 3, search.WorkOrderFilter);
    row = WriteFilterTable(worksheet, row + 3, search.ItemNumberFilter);
    // ... more filter tables ...

    // Extract MIS data indicator
    var headerRange = worksheet.Range(row, 1, row, 2);
    ApplyHeaderFormat(headerRange, "Extract MIS data?", merge: true);
    worksheet.Cell(++row, 1).Value = search.ExtractMisData ? "YES" : "NO";

    // Auto-fit with 15% padding
    for (int col = 1; col <= 4; col++)
    {
        worksheet.Column(col).AdjustToContents();
        worksheet.Column(col).Width *= 1.15;
    }

    // Protection
    worksheet.Protect(_options.Value.CriteriaSheetPassword);
}
```

### Attribute-Driven Table Writer

The `AttributeTableWriter` generates Excel tables from model collections using reflection on `OutputColumnAttribute` and `OutputTableAttribute`.

**Column Ordering:**
1. Sort by `OutputColumnAttribute.Order` (ascending)
2. Break ties alphabetically by property name

**Implementation Pattern:**

```csharp
public class AttributeTableWriter
{
    private readonly OutputColumnCache _cache;

    public IXLTable WriteTable<T>(
        IXLWorksheet worksheet,
        int startRow,
        int startCol,
        IEnumerable<T> data,
        string? tableNameOverride = null)
    {
        var tableAttr = typeof(T).GetCustomAttribute<OutputTableAttribute>();
        var columns = _cache.GetColumns<T>();
        var tableName = tableNameOverride ?? tableAttr?.TableName ?? typeof(T).Name;

        // Write header row
        var col = startCol;
        foreach (var column in columns)
        {
            var cell = worksheet.Cell(startRow, col);
            cell.Value = column.Attribute.HeaderText;
            ApplyHeaderFormat(cell);
            col++;
        }

        // Write data rows
        var dataList = data.ToList();
        var row = startRow + 1;
        foreach (var item in dataList)
        {
            col = startCol;
            foreach (var column in columns)
            {
                var value = column.Property.GetValue(item);
                worksheet.Cell(row, col).Value = XLCellValue.FromObject(value);
                col++;
            }
            row++;
        }

        // Create table
        var dataRange = worksheet.Range(
            startRow, startCol,
            startRow + dataList.Count, startCol + columns.Count - 1);
        var table = dataRange.CreateTable(tableName);
        table.Theme = XLTableTheme.TableStyleLight18;
        table.ShowTotalsRow = false;

        // Apply column formatting
        col = startCol;
        foreach (var column in columns)
        {
            ApplyColumnFormat(worksheet.Column(col), column.Attribute);
            col++;
        }

        return table;
    }
}
```

### Data Sheet Generators (Results, MIS Info, Investigation)

These sheets use the `AttributeTableWriter` with model-specific configuration.

**Search Results Sheet:**
- 19 columns per SearchResult model
- Auto-fit with 30% padding
- No protection (legacy behavior via attribute-driven path)
- Light18 table style

**MIS Info Sheet:**
- 19 columns per MisSearchResult model
- Three columns with wrapped text (fixed 65-char width):
  - Test Description
  - Tools & Gauges
  - Work Instructions
- Other columns: auto-fit with 30% padding

**Investigation Sheet:**
- 12 columns per MisNonMatchSearchResult model
- Date columns use `[$-409]MM/dd/yyyy;@` format
- Auto-fit with 30% padding

## Column Formatting Patterns

### Format Constants

```csharp
public static class ExcelFormats
{
    public const string STD_FORMAT = "@";  // Text
    public const string DATE_FORMAT = "[$-409]MM/dd/yyyy;@";
    public const string TIMESTAMP_FORMAT = "[$-409]m/d/yy h:mm AM/PM;@";
    public const double WRAPPED_COLUMN_WIDTH = 65;
    public const double CRITERIA_PADDING_FACTOR = 1.15;
    public const double DATA_PADDING_FACTOR = 1.30;
}
```

### Auto-Fit with Padding

```csharp
private void ApplyColumnFormat(IXLColumn column, OutputColumnAttribute attr)
{
    // Set number format
    column.Style.NumberFormat.Format = attr.Format;

    if (attr.WrapText)
    {
        column.Style.Alignment.WrapText = true;
    }

    if (attr.AutoWidth)
    {
        column.AdjustToContents();
        column.Width *= ExcelFormats.DATA_PADDING_FACTOR;
    }
    else
    {
        column.Width = attr.Width;
    }
}
```

### Wrapped Text Columns

Columns marked with `WrapText = true` and `AutoWidth = false` skip auto-fit:

```csharp
if (attr.WrapText && !attr.AutoWidth)
{
    column.Width = attr.Width; // Usually 65
    column.Style.Alignment.WrapText = true;
    // Do NOT call AdjustToContents()
}
```

## Worksheet Protection

### Protection Configuration

```csharp
public class ExcelExportOptions
{
    public const string SectionName = "ExcelExport";

    public string CriteriaSheetPassword { get; set; } = "JDE_SCOPING_TOOL_PASS";
    public string DataSheetPassword { get; set; } = "JDESCOPINGTOOL";
}
```

### Allowed Operations

Protected worksheets allow specific operations:

```csharp
private void ApplyProtection(IXLWorksheet worksheet, string password)
{
    var protection = worksheet.Protect(password);

    // Allow these operations
    protection.AllowElement(XLSheetProtectionElements.DeleteColumns);
    protection.AllowElement(XLSheetProtectionElements.AutoFilter);
    protection.AllowElement(XLSheetProtectionElements.FormatCells);
    protection.AllowElement(XLSheetProtectionElements.FormatColumns);
    protection.AllowElement(XLSheetProtectionElements.FormatRows);
    protection.AllowElement(XLSheetProtectionElements.SelectLockedCells);
    protection.AllowElement(XLSheetProtectionElements.SelectUnlockedCells);
    protection.AllowElement(XLSheetProtectionElements.EditObjects);
    protection.AllowElement(XLSheetProtectionElements.Sort);

    // DeleteRows is NOT allowed (not in AllowElement call)
}
```

### Editable Extension Area

Legacy code unlocked an area beyond the data for user additions. In ClosedXML:

```csharp
// Unlock cells beyond data range for user editing
var lastDataRow = table.RangeAddress.LastAddress.RowNumber;
var lastDataCol = table.RangeAddress.LastAddress.ColumnNumber;

// Unlock 1000 rows and columns beyond data
var extensionRange = worksheet.Range(
    1, lastDataCol + 1,
    lastDataRow + 1000, lastDataCol + 1000);
extensionRange.Style.Protection.Locked = false;
```

**Note:** The legacy code applies protection to data sheets (Search Results, MIS Info, Investigation) via the `ApplySecurity` method. The criteria sheet uses a different password.

## Memory Management for Large Exports

### Current Approach: In-Memory

The current implementation loads all data into memory and generates the workbook synchronously:

```csharp
// All results loaded
var results = search.Results; // List<SearchResult>

// Workbook built in memory
using var workbook = new XLWorkbook();
// ... add sheets ...

// Serialize to byte[]
using var stream = new MemoryStream();
workbook.SaveAs(stream);
return stream.ToArray();
```

### Memory Considerations

| Export Size | Rows | Estimated Memory | Approach |
|-------------|------|------------------|----------|
| Small | <10K | <50 MB | In-memory (current) |
| Medium | 10K-100K | 50-500 MB | In-memory (current) |
| Large | >100K | >500 MB | Consider streaming (future) |

### Future Optimization: Streaming

For very large exports, ClosedXML supports SAX-based streaming via `XLWorkbook.SaveAsAsync()`. This would require:

1. Writing sheets incrementally
2. Using `IAsyncEnumerable<T>` for data sources
3. Streaming directly to response or file

This is deferred to a future phase if memory pressure becomes an issue.

## Temp File Handling

### No Temp Files Required

The current implementation writes directly to `MemoryStream` and returns `byte[]`. The result is stored in the `Search.Results` column (VARBINARY) in the database.

If debugging is enabled (legacy behavior), a copy may be written to disk:

```csharp
if (_options.Value.DebugWriteToFile)
{
    var debugPath = Path.Combine(
        _options.Value.DebugOutputDirectory,
        $"Search_{search.Id}_{DateTime.Now:yyyyMMdd_HHmmss}.xlsx");
    await File.WriteAllBytesAsync(debugPath, result, cancellationToken);
}
```

This is optional and controlled by configuration.

## Header Formatting

### Standard Header Style

```csharp
private void ApplyHeaderFormat(IXLCell cell, string? text = null)
{
    cell.Style.Alignment.Horizontal = XLAlignmentHorizontalValues.Center;
    cell.Style.Font.Bold = true;
    cell.Style.Fill.BackgroundColor = XLColor.Gainsboro;

    if (!string.IsNullOrEmpty(text))
    {
        cell.Value = text;
    }
}

private void ApplyHeaderFormat(IXLRange range, string? text = null, bool merge = false)
{
    range.Style.Alignment.Horizontal = XLAlignmentHorizontalValues.Center;
    range.Style.Font.Bold = true;
    range.Style.Fill.BackgroundColor = XLColor.Gainsboro;

    if (merge)
    {
        range.Merge();
    }

    if (!string.IsNullOrEmpty(text))
    {
        range.FirstCell().Value = text;
    }
}
```

## Data Entry Template Generator

For bulk data entry via the UI:

```csharp
public class DataEntryTemplateGenerator
{
    public byte[] Generate<T>(IEnumerable<T>? sourceData, string headerText)
    {
        using var workbook = new XLWorkbook();
        var worksheet = workbook.Worksheets.Add("Data Entry Template");

        // Header
        var headerCell = worksheet.Cell(1, 1);
        ApplyHeaderFormat(headerCell, headerText);
        worksheet.Column(1).Width = 45;

        // Data (if provided)
        if (sourceData != null)
        {
            var row = 2;
            foreach (var item in sourceData)
            {
                worksheet.Cell(row++, 1).Value = XLCellValue.FromObject(item);
            }
        }

        // All cells as text
        worksheet.Column(1).Style.NumberFormat.Format = "@";

        using var stream = new MemoryStream();
        workbook.SaveAs(stream);
        return stream.ToArray();
    }

    public byte[] Generate(object[][]? sourceData, string[] headers)
    {
        using var workbook = new XLWorkbook();
        var worksheet = workbook.Worksheets.Add("Data Entry Template");

        // Headers
        for (int col = 0; col < headers.Length; col++)
        {
            ApplyHeaderFormat(worksheet.Cell(1, col + 1), headers[col]);
            worksheet.Column(col + 1).Width = 65;
            worksheet.Column(col + 1).Style.NumberFormat.Format = "@";
        }

        // Data
        if (sourceData != null)
        {
            for (int row = 0; row < sourceData.Length; row++)
            {
                for (int col = 0; col < sourceData[row].Length; col++)
                {
                    worksheet.Cell(row + 2, col + 1).Value =
                        XLCellValue.FromObject(sourceData[row][col]);
                }
            }
        }

        using var stream = new MemoryStream();
        workbook.SaveAs(stream);
        return stream.ToArray();
    }
}
```

## Service Registration

```csharp
public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddExcelExport(
        this IServiceCollection services,
        IConfiguration configuration)
    {
        // Bind options
        services.Configure<ExcelExportOptions>(
            configuration.GetSection(ExcelExportOptions.SectionName));

        // Register main service (scoped - per request)
        services.AddScoped<IExcelExportService, ExcelExportService>();

        // Register helpers (singleton - stateless)
        services.AddSingleton<OutputColumnCache>();
        services.AddSingleton<AttributeTableWriter>();
        services.AddSingleton<DataEntryTemplateGenerator>();

        return services;
    }
}
```

## NuGet Dependencies

```xml
<ItemGroup>
  <PackageReference Include="ClosedXML" Version="0.104.*" />
  <PackageReference Include="Microsoft.Extensions.Options" Version="9.0.*" />
  <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.*" />
  <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.*" />
</ItemGroup>
```

## Testing Strategy

### Unit Tests

- Mock search models with various filter combinations
- Verify sheet generation logic without file I/O
- Test column ordering and formatting
- Test inclusion reason calculation
- Test null handling for optional sheets

### Integration Tests

- Generate actual .xlsx files
- Open with ClosedXML to verify structure
- Compare against legacy output samples
- Test protection passwords
- Verify table styles applied correctly

### Test Data

```csharp
public static class ExcelExportTestData
{
    public static SearchModel CreateMinimalSearch() => new()
    {
        Id = 1,
        Name = "Test Search",
        UserName = "testuser",
        SubmitDT = DateTime.Now.AddHours(-1),
        StartDT = DateTime.Now.AddMinutes(-30),
        EndDT = DateTime.Now,
        ExtractMisData = false,
        Results = new List<SearchResult>
        {
            CreateSearchResult(12345, "ITEM-001")
        }
    };

    public static SearchModel CreateFullSearch() => new()
    {
        // ... includes MisResults and MisNonMatchResults
    };
}
```