- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Add comprehensive tests for the 5 simple table extraction functions:
- fn_GetSearchWorkOrders
- fn_GetSearchItemNumbers
- fn_GetSearchProfitCenters
- fn_GetSearchWorkCenters
- fn_GetSearchOperatorIDs
Each function tested for:
- Valid array returns all values
- Empty array returns empty
- Missing property returns empty
- Search not found returns empty
- NULL criteria returns empty
- Invalid JSON returns empty
- Bad type values (nulls) filtered out
Additional edge case tests for large arrays and long values.
Total: 38 tests (7 per function + 2 additional edge cases).
Add DatabaseTestBase.cs with xUnit Collection for test isolation:
- DatabaseTestCollection disables parallel execution
- DatabaseTestFixture verifies database connectivity
- DatabaseTestBase provides connection and cleanup helpers
- InsertTestSearchAsync for creating test data with SearchCriteria
- InsertTestSearchWithRawCriteriaAsync for testing invalid JSON scenarios
Also adds required packages: Dapper, FluentAssertions, Microsoft.Data.SqlClient
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
Add integration tests for SearchApiClient covering:
- GetUserSearchesAsync with authentication returns success
- GetUserSearchesAsync without authentication returns unauthorized
- GetSearchAsync with non-existent ID returns not found
Add comprehensive unit tests for FileApiClient covering:
- Route verification tests for all 4 download endpoints (POST method)
- Route verification tests for 2 upload endpoints (POST method)
- Success tests for downloads returning byte arrays
- Success tests verifying existing data can be passed to downloads
- Success tests for uploads returning IReadOnlyList<T>
- Multipart content-type and filename verification test
- Error tests (404 for download, 400 validation error for upload)
Add unit tests for AuthApiClient covering:
- Route verification tests for GetPublicKey, Login, Logout, GetCurrentUser
- Success tests for all 4 methods
- Proper handling of 204 NoContent for Logout (returns Unit)
- Route verification tests for all 4 methods (FindItems, FindProfitCenters,
FindWorkCenters, FindOperators)
- Query string encoding tests verifying special characters (&, =) and spaces
are properly URL-encoded
- Success tests for all lookup methods verifying deserialization
- Representative error test for 500 status code
Add comprehensive unit tests for SearchApiClient covering:
- Route verification tests for all 6 methods (GetUserSearches, GetQueuedSearches,
GetSearch, CopySearch, CreateSearch, GetResults)
- Success tests verifying deserialization for all methods
- Representative error tests (404 for GetSearch, 401 for GetUserSearches)
Uses MockHttpMessageHandler to verify correct API routes are called
and ApiResult<T> pattern maps responses correctly.
Add 7 edge case tests for ApiClientBase:
- GetAsync_Returns200_EmptyBody_MapsToApiError
- GetAsync_Returns200_InvalidJson_MapsToApiError
- GetAsync_Returns204_ForUnitType_MapsToSuccess
- GetAsync_Returns204_ForNonUnitType_MapsToApiError
- GetAsync_NetworkException_MapsToApiError
- GetAsync_Timeout_MapsToApiError
- GetAsync_Returns400_WithoutValidationFormat_MapsToApiError
Also fix bug in ApiClientBase where 204 NoContent for Unit type
failed due to incorrect type casting. The implicit conversion from
Unit to ApiResult<Unit> must be applied before the runtime cast.
- Add GetAsync_Returns404_MapsToNotFound test
- Add GetAsync_Returns400_WithValidationErrors_MapsToValidationError test
- Add GetAsync_Returns401_MapsToUnauthorized test
- Add GetAsync_Returns403_MapsToForbidden test
- Add GetAsync_Returns500_MapsToApiError test
- Add FrameworkReference to Microsoft.AspNetCore.App for ValidationProblemDetails
Added using statement for JdeScoping.Core.ApiContracts. The existing
routes (/api/file/...) are left unchanged as they test legacy download
by cache key endpoints which differ from the new FileIO routes.
- Create JdeScoping.DataSync.Dev for sandbox testing ETL code
- Create JdeScoping.DataSync.Dev.Tests for associated tests
- Move 22 source files and 8 test files
- Update namespaces from DevEtl to Dev
- Add both projects to solution
Add integration tests for the Branch development ETL pipeline:
- BranchDevEtlTests with tests for pipeline creation and execution
- Tests verify pipeline creates correctly and loads data from cache files
- Added appsettings.json with local connection string and cache directory
- Added necessary packages (Configuration.Json, SqlClient, Dapper)
Tests require local SQL Server and CACHED_DB_FILES directory with
branch.json.zstd to pass; tests silently skip if resources unavailable.
- Filter MERGE SQL columns to only include columns that exist in destination
(allColumns and updateColumns were using unfiltered source columns)
- Fix schema-qualified table names to use proper [schema].[table] format
instead of wrapping entire name in single brackets
- Add empty column mapping validation to throw early if no columns intersect
- Add JdeDateTransformer output column collision detection in OnInitialize
- Add TODO comment for WithCommandTimeout (stored but not yet passed to
destinations)
- Add tests for FormatQualifiedTableName and output column collision
- Add ParseTableName method to parse table names with optional schema
- Supports: "Table", "dbo.Table", "[dbo].[Table]"
- Returns (schema, table) tuple, defaults to "dbo" schema
- Update DisableIndexes, RebuildIndexes, UpdateStatistics to:
- Use QUOTENAME() for SQL injection protection
- Pass schema and table as parameters via SqlScriptRunner
- Support non-dbo schemas
- Update CustomSql to accept optional parameters and timeout
- Add comprehensive tests for ParseTableName with various formats
- Add DefaultInvalidDateSentinel (1900-01-01) for invalid date handling
- Add optional invalidDateSentinel constructor parameter
- Add MapOrdinal override returning -1 for computed DateTime column
- Add GetDataTypeName override returning "datetime" for computed column
- Update ParseJdeDateTime with comprehensive validation:
- Validate date is positive
- Validate century (0 or 1)
- Validate year (0-99)
- Validate day of year (1-366 and respects leap year)
- Validate time components (hours 0-23, minutes/seconds 0-59)
- Add tests for all new functionality
Add virtual methods to DataTransformerBase for GetBytes, GetChars,
GetData, and GetDataTypeName that properly handle computed columns
by throwing NotSupportedException when MapOrdinal returns -1.
Update TransformingDataReader to delegate these methods to the
transformer instead of directly to the source reader.
Add MapOrdinal method to the IDataTransformer interface and provide
a default implementation in DataTransformerBase. This enables
transformers to report the mapping between transformed ordinals and
source ordinals, supporting use cases like computed columns which
return -1 to indicate no source ordinal.
- Add MapOrdinal(int, IDataReader) to IDataTransformer interface
- Add virtual MapOrdinal implementation in DataTransformerBase
- Add DataTransformerBaseTests with test for default behavior
Adds DbQuerySource, an IImportSource implementation that executes SQL
queries against the LotFinder database. Supports parameterized queries
using anonymous objects and configurable command timeouts.
Add transformer that combines JDE Julian date (CYYDDD) and time (HHMMSS)
columns into a single DateTime column. Includes static ParseJdeDateTime
method for direct date conversion.
Add a data transformer that removes specified columns from the data stream.
Columns are matched by name (case-insensitive) and multiple columns can be
dropped in a single transformer. Includes comprehensive tests using NSubstitute
for mock IDataReader.
Add core transformer infrastructure for the ETL pipeline:
- DataTransformerBase: abstract base class with virtual methods for
field count, names, types, values, ordinals, and null checking
- TransformingDataReader: IDataReader wrapper that delegates to
transformer, enabling on-the-fly data transformations
Add SqlScriptRunner class that implements IScriptRunner for executing
SQL scripts against the LotFinderDB cache database. Includes constructor
validation and configurable timeout support (default 1 hour).
Joseph Doherty