Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
- Filter MERGE SQL columns to only include columns that exist in destination
(allColumns and updateColumns were using unfiltered source columns)
- Fix schema-qualified table names to use proper [schema].[table] format
instead of wrapping entire name in single brackets
- Add empty column mapping validation to throw early if no columns intersect
- Add JdeDateTransformer output column collision detection in OnInitialize
- Add TODO comment for WithCommandTimeout (stored but not yet passed to
destinations)
- Add tests for FormatQualifiedTableName and output column collision
- Add ParseTableName method to parse table names with optional schema
- Supports: "Table", "dbo.Table", "[dbo].[Table]"
- Returns (schema, table) tuple, defaults to "dbo" schema
- Update DisableIndexes, RebuildIndexes, UpdateStatistics to:
- Use QUOTENAME() for SQL injection protection
- Pass schema and table as parameters via SqlScriptRunner
- Support non-dbo schemas
- Update CustomSql to accept optional parameters and timeout
- Add comprehensive tests for ParseTableName with various formats
- Add DefaultInvalidDateSentinel (1900-01-01) for invalid date handling
- Add optional invalidDateSentinel constructor parameter
- Add MapOrdinal override returning -1 for computed DateTime column
- Add GetDataTypeName override returning "datetime" for computed column
- Update ParseJdeDateTime with comprehensive validation:
- Validate date is positive
- Validate century (0 or 1)
- Validate year (0-99)
- Validate day of year (1-366 and respects leap year)
- Validate time components (hours 0-23, minutes/seconds 0-59)
- Add tests for all new functionality
Add virtual methods to DataTransformerBase for GetBytes, GetChars,
GetData, and GetDataTypeName that properly handle computed columns
by throwing NotSupportedException when MapOrdinal returns -1.
Update TransformingDataReader to delegate these methods to the
transformer instead of directly to the source reader.
Add MapOrdinal method to the IDataTransformer interface and provide
a default implementation in DataTransformerBase. This enables
transformers to report the mapping between transformed ordinals and
source ordinals, supporting use cases like computed columns which
return -1 to indicate no source ordinal.
- Add MapOrdinal(int, IDataReader) to IDataTransformer interface
- Add virtual MapOrdinal implementation in DataTransformerBase
- Add DataTransformerBaseTests with test for default behavior
Adds DbQuerySource, an IImportSource implementation that executes SQL
queries against the LotFinder database. Supports parameterized queries
using anonymous objects and configurable command timeouts.
Add transformer that combines JDE Julian date (CYYDDD) and time (HHMMSS)
columns into a single DateTime column. Includes static ParseJdeDateTime
method for direct date conversion.
Add a data transformer that removes specified columns from the data stream.
Columns are matched by name (case-insensitive) and multiple columns can be
dropped in a single transformer. Includes comprehensive tests using NSubstitute
for mock IDataReader.
Add core transformer infrastructure for the ETL pipeline:
- DataTransformerBase: abstract base class with virtual methods for
field count, names, types, values, ordinals, and null checking
- TransformingDataReader: IDataReader wrapper that delegates to
transformer, enabling on-the-fly data transformations
Add SqlScriptRunner class that implements IScriptRunner for executing
SQL scripts against the LotFinderDB cache database. Includes constructor
validation and configurable timeout support (default 1 hour).
Joseph Doherty