- Filter MERGE SQL columns to only include columns that exist in destination
(allColumns and updateColumns were using unfiltered source columns)
- Fix schema-qualified table names to use proper [schema].[table] format
instead of wrapping entire name in single brackets
- Add empty column mapping validation to throw early if no columns intersect
- Add JdeDateTransformer output column collision detection in OnInitialize
- Add TODO comment for WithCommandTimeout (stored but not yet passed to
destinations)
- Add tests for FormatQualifiedTableName and output column collision
- Add DefaultInvalidDateSentinel (1900-01-01) for invalid date handling
- Add optional invalidDateSentinel constructor parameter
- Add MapOrdinal override returning -1 for computed DateTime column
- Add GetDataTypeName override returning "datetime" for computed column
- Update ParseJdeDateTime with comprehensive validation:
- Validate date is positive
- Validate century (0 or 1)
- Validate year (0-99)
- Validate day of year (1-366 and respects leap year)
- Validate time components (hours 0-23, minutes/seconds 0-59)
- Add tests for all new functionality
Add virtual methods to DataTransformerBase for GetBytes, GetChars,
GetData, and GetDataTypeName that properly handle computed columns
by throwing NotSupportedException when MapOrdinal returns -1.
Update TransformingDataReader to delegate these methods to the
transformer instead of directly to the source reader.
Add MapOrdinal method to the IDataTransformer interface and provide
a default implementation in DataTransformerBase. This enables
transformers to report the mapping between transformed ordinals and
source ordinals, supporting use cases like computed columns which
return -1 to indicate no source ordinal.
- Add MapOrdinal(int, IDataReader) to IDataTransformer interface
- Add virtual MapOrdinal implementation in DataTransformerBase
- Add DataTransformerBaseTests with test for default behavior
Add transformer that combines JDE Julian date (CYYDDD) and time (HHMMSS)
columns into a single DateTime column. Includes static ParseJdeDateTime
method for direct date conversion.
Add a data transformer that removes specified columns from the data stream.
Columns are matched by name (case-insensitive) and multiple columns can be
dropped in a single transformer. Includes comprehensive tests using NSubstitute
for mock IDataReader.
Add core transformer infrastructure for the ETL pipeline:
- DataTransformerBase: abstract base class with virtual methods for
field count, names, types, values, ordinals, and null checking
- TransformingDataReader: IDataReader wrapper that delegates to
transformer, enabling on-the-fly data transformations
Joseph Doherty