jdescopingtool/PLANS/2025-01-22-regex-transformer-design.md

Regex Transformer Design

Date: 2025-01-22 Status: Approved Author: Claude + User collaboration

Overview

Add a new RegexTransformer to the DataSync ETL pipeline that transforms string values in a column using regular expressions. Includes a custom editor for the ConfigManager with a live test/preview feature.

Requirements

1. Two transformation modes:

   • Find & Replace - Replace matched text with replacement string
   • Match & Extract - Extract first capture group from pattern

2. Single column per transformer - Each transformer operates on one column; add multiple transformers for multiple columns

3. Configurable non-match behavior:

   • Keep original value (default)
   • Return null
   • Return empty string

4. Case-insensitive option - Optional flag for case-insensitive matching

5. Test/preview in editor - Users can test their regex pattern against sample input before saving

Architecture

Core Transformer

File: NEW/src/JdeScoping.DataSync/Etl/Transformers/RegexTransformer.cs

public enum NonMatchBehavior
{
    KeepOriginal,
    ReturnNull,
    ReturnEmpty
}

public class RegexTransformer : DataTransformerBase
{
    public RegexTransformer(
        string columnName,
        string pattern,
        string? replacement = null,           // null = Match & Extract mode
        bool ignoreCase = false,
        NonMatchBehavior nonMatchBehavior = NonMatchBehavior.KeepOriginal)
}

Behavior:

• Extends DataTransformerBase
• In OnInitialize(): finds column ordinal, compiles regex
• Overrides GetValue(): transforms target column, passes through others
• Mode determined by replacement parameter: non-null = Find & Replace, null = Match & Extract

Transformation logic:

1. If not target column → pass through
2. If null/DBNull → pass through
3. Find & Replace mode: regex.Replace(value, replacement)
4. Match & Extract mode: return match.Groups[1].Value if match, else apply NonMatchBehavior

Configuration Model

File: NEW/src/Utils/JdeScoping.ConfigManager/Models/PipelineModel.cs

Add enum:

[JsonConverter(typeof(JsonStringEnumConverter))]
public enum NonMatchBehavior
{
    KeepOriginal,
    ReturnNull,
    ReturnEmpty
}

Add properties to TransformerModel:

public string? ColumnName { get; set; }
public string? Pattern { get; set; }
public string? Replacement { get; set; }
public bool IgnoreCase { get; set; }
public NonMatchBehavior NonMatchBehavior { get; set; } = NonMatchBehavior.KeepOriginal;

JSON example:

{
  "Type": "Regex",
  "ColumnName": "BatchID",
  "Pattern": "^IIS_",
  "Replacement": "",
  "IgnoreCase": true,
  "NonMatchBehavior": "KeepOriginal"
}

ViewModel

File: NEW/src/Utils/JdeScoping.ConfigManager/ViewModels/PipelineSteps/TransformerStepViewModels.cs

Add RegexTransformerViewModel class:

Core properties (bound to config):

• ColumnName (string)
• Pattern (string)
• Replacement (string)
• IgnoreCase (bool)
• NonMatchBehavior (enum)

Mode properties (computed):

• IsFindReplaceMode / IsMatchExtractMode - for radio button binding
• PatternHelpText - changes based on mode

Test feature properties:

• TestInput (string)
• TestResultValue (string)
• TestResultLabel (string) - "Output" or "No Match"
• TestResultIcon (string) - "✓" or "—"
• TestResultBackground (string) - green or orange
• HasTestResult / HasTestError (bool)
• TestErrorMessage (string)

Command:

• TestPatternCommand - executes regex test

Editor View

File: NEW/src/Utils/JdeScoping.ConfigManager/Views/Editors/RegexEditorView.axaml

Layout sections:

1. Header - Title "Regex Transformer" + description
2. Column Name - Text input with required indicator
3. Mode Selection - Radio buttons: "Find & Replace" / "Match & Extract"
4. Pattern - Monospace text input with dynamic help text
5. Replacement - Monospace text input (visible only in Find & Replace mode)
6. Options Row - Case Insensitive checkbox + Non-Match Behavior dropdown
7. Test Section - Bordered area with:
   • Sample input textbox
   • Test button (blue)
   • Result display with status icon (green checkmark / orange dash)
   • Error display (red-tinted) for invalid patterns
8. Help Box - Pattern examples

Design tokens (matching existing editors):

• Background: #0D0F12, #151920, #232A35
• Text: #E6EDF5 (bright), #9BA8B8 (labels), #5C6A7A (dim)
• Accent: #3B82F6 (blue button), #22C55E (success), #F59E0B (warning), #FF6B6B (error)
• Font: JetBrains Mono for code fields
• Spacing: 16px between sections, 4px within field groups

Registration

TransformerFactory updates:

// Create() switch:
"regex" => new RegexTransformerViewModel(model, onChanged),

// CreateNew() switch:
"regex" => new RegexTransformerViewModel(onChanged),

// AvailableTypes:
["ColumnDrop", "ColumnRename", "JdeDate", "Regex"]

MainWindow.axaml DataTemplate:

<DataTemplate DataType="{x:Type steps:RegexTransformerViewModel}">
    <editors:RegexEditorView/>
</DataTemplate>

Testing Strategy

RegexTransformer Unit Tests

Test Case	Description
FindReplace_RemovesPrefix	^IIS_ + "" → IIS_12345 becomes 12345
FindReplace_ReplacesWithText	foo + bar → foo123 becomes bar123
FindReplace_UseCaptureGroups	(\d+)-(\d+) + $2-$1 swaps groups
MatchExtract_ExtractsFirstGroup	ID_(\d+) extracts 12345 from ID_12345
MatchExtract_NoMatch_KeepOriginal	Returns original when no match
MatchExtract_NoMatch_ReturnNull	Returns DBNull when configured
MatchExtract_NoMatch_ReturnEmpty	Returns "" when configured
IgnoreCase_MatchesDifferentCase	^iis_ matches IIS_12345
NullValue_PassesThrough	DBNull input returns DBNull
NonTargetColumn_Unchanged	Other columns pass through
InvalidRegex_ThrowsOnInitialize	Bad pattern throws meaningful exception

RegexTransformerViewModel Unit Tests

Test Case	Description
TestPattern_ValidRegex_ShowsResult	Test button displays transformed output
TestPattern_InvalidRegex_ShowsError	Bad pattern shows error message
ModeSwitch_UpdatesHelpText	Help text changes with mode
ToModel_SerializesCorrectly	ViewModel produces valid TransformerModel
FromModel_LoadsAllProperties	Constructor populates from existing config

Files Summary

Create:

• NEW/src/JdeScoping.DataSync/Etl/Transformers/RegexTransformer.cs
• NEW/src/Utils/JdeScoping.ConfigManager/Views/Editors/RegexEditorView.axaml
• NEW/src/Utils/JdeScoping.ConfigManager/Views/Editors/RegexEditorView.axaml.cs
• NEW/tests/JdeScoping.DataSync.Tests/Etl/Transformers/RegexTransformerTests.cs
• NEW/tests/JdeScoping.ConfigManager.Tests/ViewModels/RegexTransformerViewModelTests.cs

Modify:

• NEW/src/Utils/JdeScoping.ConfigManager/Models/PipelineModel.cs - Add enum + properties
• NEW/src/Utils/JdeScoping.ConfigManager/ViewModels/PipelineSteps/TransformerStepViewModels.cs - Add ViewModel + factory
• NEW/src/Utils/JdeScoping.ConfigManager/Views/MainWindow.axaml - Add DataTemplate