jdescopingtool/PLANS/2025-01-22-regex-transformer-design.md

# Regex Transformer Design

**Date:** 2025-01-22
**Status:** Approved
**Author:** Claude + User collaboration

## Overview

Add a new `RegexTransformer` to the DataSync ETL pipeline that transforms string values in a column using regular expressions. Includes a custom editor for the ConfigManager with a live test/preview feature.

## Requirements

1. **Two transformation modes:**
   - **Find & Replace** - Replace matched text with replacement string
   - **Match & Extract** - Extract first capture group from pattern

2. **Single column per transformer** - Each transformer operates on one column; add multiple transformers for multiple columns

3. **Configurable non-match behavior:**
   - Keep original value (default)
   - Return null
   - Return empty string

4. **Case-insensitive option** - Optional flag for case-insensitive matching

5. **Test/preview in editor** - Users can test their regex pattern against sample input before saving

## Architecture

### Core Transformer

**File:** `NEW/src/JdeScoping.DataSync/Etl/Transformers/RegexTransformer.cs`

```csharp
public enum NonMatchBehavior
{
    KeepOriginal,
    ReturnNull,
    ReturnEmpty
}

public class RegexTransformer : DataTransformerBase
{
    public RegexTransformer(
        string columnName,
        string pattern,
        string? replacement = null,           // null = Match & Extract mode
        bool ignoreCase = false,
        NonMatchBehavior nonMatchBehavior = NonMatchBehavior.KeepOriginal)
}
```

**Behavior:**
- Extends `DataTransformerBase`
- In `OnInitialize()`: finds column ordinal, compiles regex
- Overrides `GetValue()`: transforms target column, passes through others
- Mode determined by `replacement` parameter: non-null = Find & Replace, null = Match & Extract

**Transformation logic:**
1. If not target column → pass through
2. If null/DBNull → pass through
3. Find & Replace mode: `regex.Replace(value, replacement)`
4. Match & Extract mode: return `match.Groups[1].Value` if match, else apply `NonMatchBehavior`

### Configuration Model

**File:** `NEW/src/Utils/JdeScoping.ConfigManager/Models/PipelineModel.cs`

Add enum:
```csharp
[JsonConverter(typeof(JsonStringEnumConverter))]
public enum NonMatchBehavior
{
    KeepOriginal,
    ReturnNull,
    ReturnEmpty
}
```

Add properties to `TransformerModel`:
```csharp
public string? ColumnName { get; set; }
public string? Pattern { get; set; }
public string? Replacement { get; set; }
public bool IgnoreCase { get; set; }
public NonMatchBehavior NonMatchBehavior { get; set; } = NonMatchBehavior.KeepOriginal;
```

**JSON example:**
```json
{
  "Type": "Regex",
  "ColumnName": "BatchID",
  "Pattern": "^IIS_",
  "Replacement": "",
  "IgnoreCase": true,
  "NonMatchBehavior": "KeepOriginal"
}
```

### ViewModel

**File:** `NEW/src/Utils/JdeScoping.ConfigManager/ViewModels/PipelineSteps/TransformerStepViewModels.cs`

Add `RegexTransformerViewModel` class:

**Core properties (bound to config):**
- `ColumnName` (string)
- `Pattern` (string)
- `Replacement` (string)
- `IgnoreCase` (bool)
- `NonMatchBehavior` (enum)

**Mode properties (computed):**
- `IsFindReplaceMode` / `IsMatchExtractMode` - for radio button binding
- `PatternHelpText` - changes based on mode

**Test feature properties:**
- `TestInput` (string)
- `TestResultValue` (string)
- `TestResultLabel` (string) - "Output" or "No Match"
- `TestResultIcon` (string) - "✓" or "—"
- `TestResultBackground` (string) - green or orange
- `HasTestResult` / `HasTestError` (bool)
- `TestErrorMessage` (string)

**Command:**
- `TestPatternCommand` - executes regex test

### Editor View

**File:** `NEW/src/Utils/JdeScoping.ConfigManager/Views/Editors/RegexEditorView.axaml`

Layout sections:
1. **Header** - Title "Regex Transformer" + description
2. **Column Name** - Text input with required indicator
3. **Mode Selection** - Radio buttons: "Find & Replace" / "Match & Extract"
4. **Pattern** - Monospace text input with dynamic help text
5. **Replacement** - Monospace text input (visible only in Find & Replace mode)
6. **Options Row** - Case Insensitive checkbox + Non-Match Behavior dropdown
7. **Test Section** - Bordered area with:
   - Sample input textbox
   - Test button (blue)
   - Result display with status icon (green checkmark / orange dash)
   - Error display (red-tinted) for invalid patterns
8. **Help Box** - Pattern examples

**Design tokens (matching existing editors):**
- Background: `#0D0F12`, `#151920`, `#232A35`
- Text: `#E6EDF5` (bright), `#9BA8B8` (labels), `#5C6A7A` (dim)
- Accent: `#3B82F6` (blue button), `#22C55E` (success), `#F59E0B` (warning), `#FF6B6B` (error)
- Font: JetBrains Mono for code fields
- Spacing: 16px between sections, 4px within field groups

### Registration

**TransformerFactory updates:**
```csharp
// Create() switch:
"regex" => new RegexTransformerViewModel(model, onChanged),

// CreateNew() switch:
"regex" => new RegexTransformerViewModel(onChanged),

// AvailableTypes:
["ColumnDrop", "ColumnRename", "JdeDate", "Regex"]
```

**MainWindow.axaml DataTemplate:**
```xml
<DataTemplate DataType="{x:Type steps:RegexTransformerViewModel}">
    <editors:RegexEditorView/>
</DataTemplate>
```

## Testing Strategy

### RegexTransformer Unit Tests

| Test Case | Description |
|-----------|-------------|
| `FindReplace_RemovesPrefix` | `^IIS_` + `""` → `IIS_12345` becomes `12345` |
| `FindReplace_ReplacesWithText` | `foo` + `bar` → `foo123` becomes `bar123` |
| `FindReplace_UseCaptureGroups` | `(\d+)-(\d+)` + `$2-$1` swaps groups |
| `MatchExtract_ExtractsFirstGroup` | `ID_(\d+)` extracts `12345` from `ID_12345` |
| `MatchExtract_NoMatch_KeepOriginal` | Returns original when no match |
| `MatchExtract_NoMatch_ReturnNull` | Returns DBNull when configured |
| `MatchExtract_NoMatch_ReturnEmpty` | Returns `""` when configured |
| `IgnoreCase_MatchesDifferentCase` | `^iis_` matches `IIS_12345` |
| `NullValue_PassesThrough` | DBNull input returns DBNull |
| `NonTargetColumn_Unchanged` | Other columns pass through |
| `InvalidRegex_ThrowsOnInitialize` | Bad pattern throws meaningful exception |

### RegexTransformerViewModel Unit Tests

| Test Case | Description |
|-----------|-------------|
| `TestPattern_ValidRegex_ShowsResult` | Test button displays transformed output |
| `TestPattern_InvalidRegex_ShowsError` | Bad pattern shows error message |
| `ModeSwitch_UpdatesHelpText` | Help text changes with mode |
| `ToModel_SerializesCorrectly` | ViewModel produces valid TransformerModel |
| `FromModel_LoadsAllProperties` | Constructor populates from existing config |

## Files Summary

**Create:**
- `NEW/src/JdeScoping.DataSync/Etl/Transformers/RegexTransformer.cs`
- `NEW/src/Utils/JdeScoping.ConfigManager/Views/Editors/RegexEditorView.axaml`
- `NEW/src/Utils/JdeScoping.ConfigManager/Views/Editors/RegexEditorView.axaml.cs`
- `NEW/tests/JdeScoping.DataSync.Tests/Etl/Transformers/RegexTransformerTests.cs`
- `NEW/tests/JdeScoping.ConfigManager.Tests/ViewModels/RegexTransformerViewModelTests.cs`

**Modify:**
- `NEW/src/Utils/JdeScoping.ConfigManager/Models/PipelineModel.cs` - Add enum + properties
- `NEW/src/Utils/JdeScoping.ConfigManager/ViewModels/PipelineSteps/TransformerStepViewModels.cs` - Add ViewModel + factory
- `NEW/src/Utils/JdeScoping.ConfigManager/Views/MainWindow.axaml` - Add DataTemplate