- Add using statement for JdeScoping.Client.Http namespace
- Register AuthRedirectHandler as a scoped service
- Replace simple HttpClient registration with handler pipeline
- AuthRedirectHandler intercepts 401 responses and redirects to login
- Keep all existing service registrations for backward compatibility
Add integration tests for the Branch development ETL pipeline:
- BranchDevEtlTests with tests for pipeline creation and execution
- Tests verify pipeline creates correctly and loads data from cache files
- Added appsettings.json with local connection string and cache directory
- Added necessary packages (Configuration.Json, SqlClient, Dapper)
Tests require local SQL Server and CACHED_DB_FILES directory with
branch.json.zstd to pass; tests silently skip if resources unavailable.
8-task plan to create comprehensive ETL pipeline documentation:
- Overview, Sources, Transformers, Destinations, Configuration, Troubleshooting
- Update ComponentMap with ETL source paths
- Final verification of links and commits
- Filter MERGE SQL columns to only include columns that exist in destination
(allColumns and updateColumns were using unfiltered source columns)
- Fix schema-qualified table names to use proper [schema].[table] format
instead of wrapping entire name in single brackets
- Add empty column mapping validation to throw early if no columns intersect
- Add JdeDateTransformer output column collision detection in OnInitialize
- Add TODO comment for WithCommandTimeout (stored but not yet passed to
destinations)
- Add tests for FormatQualifiedTableName and output column collision
- Add ParseTableName method to parse table names with optional schema
- Supports: "Table", "dbo.Table", "[dbo].[Table]"
- Returns (schema, table) tuple, defaults to "dbo" schema
- Update DisableIndexes, RebuildIndexes, UpdateStatistics to:
- Use QUOTENAME() for SQL injection protection
- Pass schema and table as parameters via SqlScriptRunner
- Support non-dbo schemas
- Update CustomSql to accept optional parameters and timeout
- Add comprehensive tests for ParseTableName with various formats
- Add DefaultInvalidDateSentinel (1900-01-01) for invalid date handling
- Add optional invalidDateSentinel constructor parameter
- Add MapOrdinal override returning -1 for computed DateTime column
- Add GetDataTypeName override returning "datetime" for computed column
- Update ParseJdeDateTime with comprehensive validation:
- Validate date is positive
- Validate century (0 or 1)
- Validate year (0-99)
- Validate day of year (1-366 and respects leap year)
- Validate time components (hours 0-23, minutes/seconds 0-59)
- Add tests for all new functionality
Add virtual methods to DataTransformerBase for GetBytes, GetChars,
GetData, and GetDataTypeName that properly handle computed columns
by throwing NotSupportedException when MapOrdinal returns -1.
Update TransformingDataReader to delegate these methods to the
transformer instead of directly to the source reader.
Add MapOrdinal method to the IDataTransformer interface and provide
a default implementation in DataTransformerBase. This enables
transformers to report the mapping between transformed ordinals and
source ordinals, supporting use cases like computed columns which
return -1 to indicate no source ordinal.
- Add MapOrdinal(int, IDataReader) to IDataTransformer interface
- Add virtual MapOrdinal implementation in DataTransformerBase
- Add DataTransformerBaseTests with test for default behavior
12-task TDD implementation plan covering:
- MapOrdinal interface and binary method overrides
- ColumnDropTransformer and JdeDateTransformer ordinal mapping
- JDE date validation with 1900-01-01 sentinel
- Column rename collision detection
- SqlScriptRunner parameters support
- CommonScripts with ParseTableName and QUOTENAME
- Destination command timeouts
- Column mapping with destination schema intersection
- EtlPipelineBuilder.WithCommandTimeout validation
- Full test suite verification
- Codex MCP final review
Adds DbQuerySource, an IImportSource implementation that executes SQL
queries against the LotFinder database. Supports parameterized queries
using anonymous objects and configurable command timeouts.
Add transformer that combines JDE Julian date (CYYDDD) and time (HHMMSS)
columns into a single DateTime column. Includes static ParseJdeDateTime
method for direct date conversion.
Add a data transformer that removes specified columns from the data stream.
Columns are matched by name (case-insensitive) and multiple columns can be
dropped in a single transformer. Includes comprehensive tests using NSubstitute
for mock IDataReader.
Add core transformer infrastructure for the ETL pipeline:
- DataTransformerBase: abstract base class with virtual methods for
field count, names, types, values, ordinals, and null checking
- TransformingDataReader: IDataReader wrapper that delegates to
transformer, enabling on-the-fly data transformations
Add SqlScriptRunner class that implements IScriptRunner for executing
SQL scripts against the LotFinderDB cache database. Includes constructor
validation and configurable timeout support (default 1 hour).
Add the four core interfaces for the ETL pipeline:
- IImportSource: defines data sources for reading
- IDataTransformer: defines data transformation layer
- IImportDestination: defines data write destinations
- IScriptRunner: defines script execution capability
Move configuration options from Core/DataAccess/DataSync/ExcelIO to
dedicated Options folders within each project for better organization.
Update all references and tests accordingly.
Joseph Doherty