Files
jdescopingtool/openspec/changes/archive/2026-01-01-implement-excel-export/design.md
T
Joseph Doherty 26ff8d9b4f Initial commit: JDE Scoping Tool migration project
Set up repository with legacy .NET Framework 4.8 source (OLD/),
new .NET 10 Blazor solution (NEW/), OpenSpec specifications,
documentation, and project configuration.
2026-01-02 07:43:29 -05:00

688 lines
23 KiB
Markdown

# Excel Export Design
## Overview
This document describes the architecture and implementation approach for the Excel export subsystem, including workbook generation, sheet generators, formatting patterns, and memory management.
## Architecture
### Component Diagram
```
┌─────────────────────────────────────────────────────────────────────┐
│ Search Processing │
│ (SearchProcessor produces SearchModel with populated results) │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ IExcelExportService │
│ Task<byte[]> GenerateAsync(SearchModel, CancellationToken) │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ ExcelExportService │
├─────────────────────────────────────────────────────────────────────┤
│ - ILogger<ExcelExportService> │
│ - IOptions<ExcelExportOptions> │
├─────────────────────────────────────────────────────────────────────┤
│ Uses: │
│ - CriteriaSheetGenerator │
│ - AttributeTableWriter │
│ - WorksheetProtector │
│ - HeaderFormatter │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ ClosedXML │
│ XLWorkbook → IXLWorksheet → IXLCell/IXLRange │
└─────────────────────────────────────────────────────────────────────┘
byte[] output
```
### Project Structure
```
NEW/src/JdeScoping.ExcelExport/
├── Attributes/
│ ├── OutputColumnAttribute.cs
│ └── OutputTableAttribute.cs
├── Configuration/
│ └── ExcelExportOptions.cs
├── Generators/
│ ├── CriteriaSheetGenerator.cs # Search Criteria tab
│ ├── AttributeTableWriter.cs # Generic attribute-driven table writer
│ └── DataEntryTemplateGenerator.cs # Bulk upload templates
├── Formatting/
│ ├── HeaderFormatter.cs # Header cell formatting
│ ├── ColumnFormatter.cs # Column width and number format
│ └── WorksheetProtector.cs # Password protection
├── Helpers/
│ └── OutputColumnCache.cs # Cached reflection for column metadata
├── Models/
│ └── OutputColumn.cs # Column metadata model
├── Interfaces/
│ └── IExcelExportService.cs
├── ExcelExportService.cs
├── ServiceCollectionExtensions.cs
└── JdeScoping.ExcelExport.csproj
```
## ClosedXML Workbook Generation
### Library Selection: ClosedXML
**Why ClosedXML over EPPlus:**
| Aspect | EPPlus v7+ | ClosedXML |
|--------|------------|-----------|
| License | Commercial (Polyform NC) | MIT (fully free) |
| API Similarity | N/A (legacy v4 was LGPL) | Very similar to EPPlus v4 |
| Maintenance | Active (commercial) | Active (community) |
| .NET 10 Support | Yes | Yes |
| NuGet Downloads | High | High |
**NuGet Package:** `ClosedXML` (version 0.104.* or later)
### API Migration Guide: EPPlus to ClosedXML
| EPPlus (Legacy) | ClosedXML (New) |
|-----------------|-----------------|
| `ExcelPackage` | `XLWorkbook` |
| `ExcelWorkbook` | `XLWorkbook` (same object) |
| `ExcelWorksheet` | `IXLWorksheet` |
| `ExcelRange` | `IXLRange` or `IXLCell` |
| `ExcelTable` | `IXLTable` |
| `worksheet.Cells[row, col]` | `worksheet.Cell(row, col)` |
| `worksheet.Tables.Add(range, name)` | `worksheet.Range(...).CreateTable(name)` or `AsTable()` |
| `TableStyles.Light18` | `XLTableTheme.TableStyleLight18` |
| `Color.Gainsboro` | `XLColor.Gainsboro` |
| `range.Style.Fill.BackgroundColor.SetColor(...)` | `range.Style.Fill.BackgroundColor = XLColor.X` |
| `worksheet.Column(col).AutoFit()` | `worksheet.Column(col).AdjustToContents()` |
| `worksheet.Protection.SetPassword(pwd)` | `worksheet.Protect(pwd)` |
| `worksheet.ProtectedRanges.Add(...)` | Not needed (use cell unlock instead) |
### Workbook Generation Flow
```csharp
public async Task<byte[]> GenerateAsync(
SearchModel search,
CancellationToken cancellationToken = default)
{
using var scope = _logger.BeginScope(new Dictionary<string, object>
{
["SearchId"] = search.Id,
["SearchName"] = search.Name
});
_logger.LogInformation("Starting Excel export generation");
// ClosedXML operations are synchronous, wrap in Task.Run for non-blocking
return await Task.Run(() =>
{
cancellationToken.ThrowIfCancellationRequested();
using var workbook = new XLWorkbook();
// 1. Always generate Search Criteria sheet (first tab)
GenerateCriteriaSheet(workbook, search);
// 2. Always generate Search Results sheet (second tab)
GenerateResultsSheet(workbook, search.Results);
// 3. Conditionally generate MIS Info sheet
if (search.ExtractMisData && search.MisResults != null)
{
GenerateMisInfoSheet(workbook, search.MisResults);
}
// 4. Conditionally generate Investigation sheet
if (search.ExtractMisData && search.MisNonMatchResults != null)
{
GenerateInvestigationSheet(workbook, search.MisNonMatchResults);
}
cancellationToken.ThrowIfCancellationRequested();
// Save to byte array
using var stream = new MemoryStream();
workbook.SaveAs(stream);
return stream.ToArray();
}, cancellationToken);
}
```
## Sheet Generators
### Criteria Sheet Generator
The Search Criteria sheet documents all search parameters and execution metadata.
**Structure:**
```
Row 1: [Search Name] [value]
Row 2: [User Name] [value]
Row 3: (blank)
Row 4: [Submit timestamp] [MMM dd, yyyy hh:mm:ss tt EST]
Row 5: [Start timestamp] [MMM dd, yyyy hh:mm:ss tt EST]
Row 6: [Completed timestamp] [MMM dd, yyyy hh:mm:ss tt EST]
Row 7: (blank)
Row 8+: [Timespan Filter Table]
(2 blank rows)
[Work Order Filter Table]
(2 blank rows)
[Item Number Filter Table]
...
[Extract MIS data?] [YES/NO]
```
**Filter Table Order:**
1. Timespan Filter
2. Work Order Filter
3. Item Number Filter
4. Profit Center Filter
5. Work Center Filter
6. Component Lot Filter
7. Operator Filter
8. Item/Operation/MIS Filter
**Implementation Pattern:**
```csharp
private void GenerateCriteriaSheet(XLWorkbook workbook, SearchModel search)
{
var worksheet = workbook.Worksheets.Add("Search Criteria");
var row = 1;
// Header rows
ApplyHeaderFormat(worksheet.Cell(row, 1), "Search Name");
worksheet.Cell(row, 2).Value = search.Name;
ApplyHeaderFormat(worksheet.Cell(++row, 1), "User Name");
worksheet.Cell(row, 2).Value = search.UserName;
row++; // blank
// Timestamps
ApplyHeaderFormat(worksheet.Cell(++row, 1), "Submit timestamp");
worksheet.Cell(row, 2).Value = $"{search.SubmitDT:MMM dd, yyyy hh:mm:ss tt} EST";
// ... more timestamps ...
// Filter tables
row = WriteFilterTable(worksheet, ++row, CreateTimespanFilter(search));
row = WriteFilterTable(worksheet, row + 3, search.WorkOrderFilter);
row = WriteFilterTable(worksheet, row + 3, search.ItemNumberFilter);
// ... more filter tables ...
// Extract MIS data indicator
var headerRange = worksheet.Range(row, 1, row, 2);
ApplyHeaderFormat(headerRange, "Extract MIS data?", merge: true);
worksheet.Cell(++row, 1).Value = search.ExtractMisData ? "YES" : "NO";
// Auto-fit with 15% padding
for (int col = 1; col <= 4; col++)
{
worksheet.Column(col).AdjustToContents();
worksheet.Column(col).Width *= 1.15;
}
// Protection
worksheet.Protect(_options.Value.CriteriaSheetPassword);
}
```
### Attribute-Driven Table Writer
The `AttributeTableWriter` generates Excel tables from model collections using reflection on `OutputColumnAttribute` and `OutputTableAttribute`.
**Column Ordering:**
1. Sort by `OutputColumnAttribute.Order` (ascending)
2. Break ties alphabetically by property name
**Implementation Pattern:**
```csharp
public class AttributeTableWriter
{
private readonly OutputColumnCache _cache;
public IXLTable WriteTable<T>(
IXLWorksheet worksheet,
int startRow,
int startCol,
IEnumerable<T> data,
string? tableNameOverride = null)
{
var tableAttr = typeof(T).GetCustomAttribute<OutputTableAttribute>();
var columns = _cache.GetColumns<T>();
var tableName = tableNameOverride ?? tableAttr?.TableName ?? typeof(T).Name;
// Write header row
var col = startCol;
foreach (var column in columns)
{
var cell = worksheet.Cell(startRow, col);
cell.Value = column.Attribute.HeaderText;
ApplyHeaderFormat(cell);
col++;
}
// Write data rows
var dataList = data.ToList();
var row = startRow + 1;
foreach (var item in dataList)
{
col = startCol;
foreach (var column in columns)
{
var value = column.Property.GetValue(item);
worksheet.Cell(row, col).Value = XLCellValue.FromObject(value);
col++;
}
row++;
}
// Create table
var dataRange = worksheet.Range(
startRow, startCol,
startRow + dataList.Count, startCol + columns.Count - 1);
var table = dataRange.CreateTable(tableName);
table.Theme = XLTableTheme.TableStyleLight18;
table.ShowTotalsRow = false;
// Apply column formatting
col = startCol;
foreach (var column in columns)
{
ApplyColumnFormat(worksheet.Column(col), column.Attribute);
col++;
}
return table;
}
}
```
### Data Sheet Generators (Results, MIS Info, Investigation)
These sheets use the `AttributeTableWriter` with model-specific configuration.
**Search Results Sheet:**
- 19 columns per SearchResult model
- Auto-fit with 30% padding
- No protection (legacy behavior via attribute-driven path)
- Light18 table style
**MIS Info Sheet:**
- 19 columns per MisSearchResult model
- Three columns with wrapped text (fixed 65-char width):
- Test Description
- Tools & Gauges
- Work Instructions
- Other columns: auto-fit with 30% padding
**Investigation Sheet:**
- 12 columns per MisNonMatchSearchResult model
- Date columns use `[$-409]MM/dd/yyyy;@` format
- Auto-fit with 30% padding
## Column Formatting Patterns
### Format Constants
```csharp
public static class ExcelFormats
{
public const string STD_FORMAT = "@"; // Text
public const string DATE_FORMAT = "[$-409]MM/dd/yyyy;@";
public const string TIMESTAMP_FORMAT = "[$-409]m/d/yy h:mm AM/PM;@";
public const double WRAPPED_COLUMN_WIDTH = 65;
public const double CRITERIA_PADDING_FACTOR = 1.15;
public const double DATA_PADDING_FACTOR = 1.30;
}
```
### Auto-Fit with Padding
```csharp
private void ApplyColumnFormat(IXLColumn column, OutputColumnAttribute attr)
{
// Set number format
column.Style.NumberFormat.Format = attr.Format;
if (attr.WrapText)
{
column.Style.Alignment.WrapText = true;
}
if (attr.AutoWidth)
{
column.AdjustToContents();
column.Width *= ExcelFormats.DATA_PADDING_FACTOR;
}
else
{
column.Width = attr.Width;
}
}
```
### Wrapped Text Columns
Columns marked with `WrapText = true` and `AutoWidth = false` skip auto-fit:
```csharp
if (attr.WrapText && !attr.AutoWidth)
{
column.Width = attr.Width; // Usually 65
column.Style.Alignment.WrapText = true;
// Do NOT call AdjustToContents()
}
```
## Worksheet Protection
### Protection Configuration
```csharp
public class ExcelExportOptions
{
public const string SectionName = "ExcelExport";
public string CriteriaSheetPassword { get; set; } = "JDE_SCOPING_TOOL_PASS";
public string DataSheetPassword { get; set; } = "JDESCOPINGTOOL";
}
```
### Allowed Operations
Protected worksheets allow specific operations:
```csharp
private void ApplyProtection(IXLWorksheet worksheet, string password)
{
var protection = worksheet.Protect(password);
// Allow these operations
protection.AllowElement(XLSheetProtectionElements.DeleteColumns);
protection.AllowElement(XLSheetProtectionElements.AutoFilter);
protection.AllowElement(XLSheetProtectionElements.FormatCells);
protection.AllowElement(XLSheetProtectionElements.FormatColumns);
protection.AllowElement(XLSheetProtectionElements.FormatRows);
protection.AllowElement(XLSheetProtectionElements.SelectLockedCells);
protection.AllowElement(XLSheetProtectionElements.SelectUnlockedCells);
protection.AllowElement(XLSheetProtectionElements.EditObjects);
protection.AllowElement(XLSheetProtectionElements.Sort);
// DeleteRows is NOT allowed (not in AllowElement call)
}
```
### Editable Extension Area
Legacy code unlocked an area beyond the data for user additions. In ClosedXML:
```csharp
// Unlock cells beyond data range for user editing
var lastDataRow = table.RangeAddress.LastAddress.RowNumber;
var lastDataCol = table.RangeAddress.LastAddress.ColumnNumber;
// Unlock 1000 rows and columns beyond data
var extensionRange = worksheet.Range(
1, lastDataCol + 1,
lastDataRow + 1000, lastDataCol + 1000);
extensionRange.Style.Protection.Locked = false;
```
**Note:** The legacy code applies protection to data sheets (Search Results, MIS Info, Investigation) via the `ApplySecurity` method. The criteria sheet uses a different password.
## Memory Management for Large Exports
### Current Approach: In-Memory
The current implementation loads all data into memory and generates the workbook synchronously:
```csharp
// All results loaded
var results = search.Results; // List<SearchResult>
// Workbook built in memory
using var workbook = new XLWorkbook();
// ... add sheets ...
// Serialize to byte[]
using var stream = new MemoryStream();
workbook.SaveAs(stream);
return stream.ToArray();
```
### Memory Considerations
| Export Size | Rows | Estimated Memory | Approach |
|-------------|------|------------------|----------|
| Small | <10K | <50 MB | In-memory (current) |
| Medium | 10K-100K | 50-500 MB | In-memory (current) |
| Large | >100K | >500 MB | Consider streaming (future) |
### Future Optimization: Streaming
For very large exports, ClosedXML supports SAX-based streaming via `XLWorkbook.SaveAsAsync()`. This would require:
1. Writing sheets incrementally
2. Using `IAsyncEnumerable<T>` for data sources
3. Streaming directly to response or file
This is deferred to a future phase if memory pressure becomes an issue.
## Temp File Handling
### No Temp Files Required
The current implementation writes directly to `MemoryStream` and returns `byte[]`. The result is stored in the `Search.Results` column (VARBINARY) in the database.
If debugging is enabled (legacy behavior), a copy may be written to disk:
```csharp
if (_options.Value.DebugWriteToFile)
{
var debugPath = Path.Combine(
_options.Value.DebugOutputDirectory,
$"Search_{search.Id}_{DateTime.Now:yyyyMMdd_HHmmss}.xlsx");
await File.WriteAllBytesAsync(debugPath, result, cancellationToken);
}
```
This is optional and controlled by configuration.
## Header Formatting
### Standard Header Style
```csharp
private void ApplyHeaderFormat(IXLCell cell, string? text = null)
{
cell.Style.Alignment.Horizontal = XLAlignmentHorizontalValues.Center;
cell.Style.Font.Bold = true;
cell.Style.Fill.BackgroundColor = XLColor.Gainsboro;
if (!string.IsNullOrEmpty(text))
{
cell.Value = text;
}
}
private void ApplyHeaderFormat(IXLRange range, string? text = null, bool merge = false)
{
range.Style.Alignment.Horizontal = XLAlignmentHorizontalValues.Center;
range.Style.Font.Bold = true;
range.Style.Fill.BackgroundColor = XLColor.Gainsboro;
if (merge)
{
range.Merge();
}
if (!string.IsNullOrEmpty(text))
{
range.FirstCell().Value = text;
}
}
```
## Data Entry Template Generator
For bulk data entry via the UI:
```csharp
public class DataEntryTemplateGenerator
{
public byte[] Generate<T>(IEnumerable<T>? sourceData, string headerText)
{
using var workbook = new XLWorkbook();
var worksheet = workbook.Worksheets.Add("Data Entry Template");
// Header
var headerCell = worksheet.Cell(1, 1);
ApplyHeaderFormat(headerCell, headerText);
worksheet.Column(1).Width = 45;
// Data (if provided)
if (sourceData != null)
{
var row = 2;
foreach (var item in sourceData)
{
worksheet.Cell(row++, 1).Value = XLCellValue.FromObject(item);
}
}
// All cells as text
worksheet.Column(1).Style.NumberFormat.Format = "@";
using var stream = new MemoryStream();
workbook.SaveAs(stream);
return stream.ToArray();
}
public byte[] Generate(object[][]? sourceData, string[] headers)
{
using var workbook = new XLWorkbook();
var worksheet = workbook.Worksheets.Add("Data Entry Template");
// Headers
for (int col = 0; col < headers.Length; col++)
{
ApplyHeaderFormat(worksheet.Cell(1, col + 1), headers[col]);
worksheet.Column(col + 1).Width = 65;
worksheet.Column(col + 1).Style.NumberFormat.Format = "@";
}
// Data
if (sourceData != null)
{
for (int row = 0; row < sourceData.Length; row++)
{
for (int col = 0; col < sourceData[row].Length; col++)
{
worksheet.Cell(row + 2, col + 1).Value =
XLCellValue.FromObject(sourceData[row][col]);
}
}
}
using var stream = new MemoryStream();
workbook.SaveAs(stream);
return stream.ToArray();
}
}
```
## Service Registration
```csharp
public static class ServiceCollectionExtensions
{
public static IServiceCollection AddExcelExport(
this IServiceCollection services,
IConfiguration configuration)
{
// Bind options
services.Configure<ExcelExportOptions>(
configuration.GetSection(ExcelExportOptions.SectionName));
// Register main service (scoped - per request)
services.AddScoped<IExcelExportService, ExcelExportService>();
// Register helpers (singleton - stateless)
services.AddSingleton<OutputColumnCache>();
services.AddSingleton<AttributeTableWriter>();
services.AddSingleton<DataEntryTemplateGenerator>();
return services;
}
}
```
## NuGet Dependencies
```xml
<ItemGroup>
<PackageReference Include="ClosedXML" Version="0.104.*" />
<PackageReference Include="Microsoft.Extensions.Options" Version="9.0.*" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="9.0.*" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.*" />
</ItemGroup>
```
## Testing Strategy
### Unit Tests
- Mock search models with various filter combinations
- Verify sheet generation logic without file I/O
- Test column ordering and formatting
- Test inclusion reason calculation
- Test null handling for optional sheets
### Integration Tests
- Generate actual .xlsx files
- Open with ClosedXML to verify structure
- Compare against legacy output samples
- Test protection passwords
- Verify table styles applied correctly
### Test Data
```csharp
public static class ExcelExportTestData
{
public static SearchModel CreateMinimalSearch() => new()
{
Id = 1,
Name = "Test Search",
UserName = "testuser",
SubmitDT = DateTime.Now.AddHours(-1),
StartDT = DateTime.Now.AddMinutes(-30),
EndDT = DateTime.Now,
ExtractMisData = false,
Results = new List<SearchResult>
{
CreateSearchResult(12345, "ITEM-001")
}
};
public static SearchModel CreateFullSearch() => new()
{
// ... includes MisResults and MisNonMatchResults
};
}
```