Remove legacy ETL properties from DataSources configuration:
- FetcherTypeName (9 entries)
- PostProcessorTypeName (1 entry - MisData)
- PrepurgeData from MassConfig (9 entries)
- ReIndexData from MassConfig (9 entries)
These properties were used by the old fetcher-based ETL system
and are no longer needed with the new EtlPipeline architecture.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
- Change interface methods to accept int searchId instead of SearchModel
- Update SqlKataSearchQueryBuilder to generate SQL using extraction functions
- SQL now calls dbo.fn_GetSearchWorkOrders(@SearchId) etc instead of TVPs
- Update SearchProcessor to pass model.Id to query builder
- Update tests for new method signatures
Replace the old sync logic (fetchers, merge configurations, bulk merge
helper, post processors) with the new ETL pipeline factory.
Changes:
- Inject IEtlPipelineFactory instead of old dependencies
- Remove IServiceProvider, IDbConnectionFactory, IBulkMergeHelper,
IMergeConfigurationRegistry dependencies
- Simplify ExecuteSyncCoreAsync to build and execute pipeline
- Keep DataUpdateRepository calls for tracking sync timestamps
- Determine SyncMode from UpdateType (Mass vs Incremental)
- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Configure ETL pipelines for all 9 sync tables:
- WorkOrder_Curr, Lot, LotUsage_Curr (JDE)
- Item, WorkCenter, ProfitCenter (JDE)
- JdeUser, Branch (JDE)
- MisData (CMS) with postScript for ProcessedFlag update
Each pipeline includes:
- Source query with JDE Julian date/time parameters
- Sync modes for mass (365 day lookback, prePurge, reIndex)
and incremental (7 day lookback) operations
- Destination table with match columns and updateWhen conditions
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Add comprehensive tests for the 5 simple table extraction functions:
- fn_GetSearchWorkOrders
- fn_GetSearchItemNumbers
- fn_GetSearchProfitCenters
- fn_GetSearchWorkCenters
- fn_GetSearchOperatorIDs
Each function tested for:
- Valid array returns all values
- Empty array returns empty
- Missing property returns empty
- Search not found returns empty
- NULL criteria returns empty
- Invalid JSON returns empty
- Bad type values (nulls) filtered out
Additional edge case tests for large arrays and long values.
Total: 38 tests (7 per function + 2 additional edge cases).
Add DatabaseTestBase.cs with xUnit Collection for test isolation:
- DatabaseTestCollection disables parallel execution
- DatabaseTestFixture verifies database connectivity
- DatabaseTestBase provides connection and cleanup helpers
- InsertTestSearchAsync for creating test data with SearchCriteria
- InsertTestSearchWithRawCriteriaAsync for testing invalid JSON scenarios
Also adds required packages: Dapper, FluentAssertions, Microsoft.Data.SqlClient
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
Key changes from Codex feedback:
- THROW cannot be used in UDFs (SQL Server restriction)
- Multi-statement TVFs have poor performance (no stats)
- Changed to inline TVFs for extraction (optimal performance)
- Added validation stored procedure for strict error handling
- Use TRY_CONVERT for safe type conversion
- Use OPENJSON...WITH for complex object extraction
- Replace IsApiError with IsError
- Replace ApiError. with Error.
- Replace ValidationError.Errors with ValidationError.FieldErrors
These match the actual ApiResult<T> implementation. The tests
themselves were already correct - subagents adapted during implementation.
Add integration tests for SearchApiClient covering:
- GetUserSearchesAsync with authentication returns success
- GetUserSearchesAsync without authentication returns unauthorized
- GetSearchAsync with non-existent ID returns not found
Add comprehensive unit tests for FileApiClient covering:
- Route verification tests for all 4 download endpoints (POST method)
- Route verification tests for 2 upload endpoints (POST method)
- Success tests for downloads returning byte arrays
- Success tests verifying existing data can be passed to downloads
- Success tests for uploads returning IReadOnlyList<T>
- Multipart content-type and filename verification test
- Error tests (404 for download, 400 validation error for upload)
Add unit tests for AuthApiClient covering:
- Route verification tests for GetPublicKey, Login, Logout, GetCurrentUser
- Success tests for all 4 methods
- Proper handling of 204 NoContent for Logout (returns Unit)
- Route verification tests for all 4 methods (FindItems, FindProfitCenters,
FindWorkCenters, FindOperators)
- Query string encoding tests verifying special characters (&, =) and spaces
are properly URL-encoded
- Success tests for all lookup methods verifying deserialization
- Representative error test for 500 status code
Add comprehensive unit tests for SearchApiClient covering:
- Route verification tests for all 6 methods (GetUserSearches, GetQueuedSearches,
GetSearch, CopySearch, CreateSearch, GetResults)
- Success tests verifying deserialization for all methods
- Representative error tests (404 for GetSearch, 401 for GetUserSearches)
Uses MockHttpMessageHandler to verify correct API routes are called
and ApiResult<T> pattern maps responses correctly.
Add 7 edge case tests for ApiClientBase:
- GetAsync_Returns200_EmptyBody_MapsToApiError
- GetAsync_Returns200_InvalidJson_MapsToApiError
- GetAsync_Returns204_ForUnitType_MapsToSuccess
- GetAsync_Returns204_ForNonUnitType_MapsToApiError
- GetAsync_NetworkException_MapsToApiError
- GetAsync_Timeout_MapsToApiError
- GetAsync_Returns400_WithoutValidationFormat_MapsToApiError
Also fix bug in ApiClientBase where 204 NoContent for Unit type
failed due to incorrect type casting. The implicit conversion from
Unit to ApiResult<Unit> must be applied before the runtime cast.
- Add GetAsync_Returns404_MapsToNotFound test
- Add GetAsync_Returns400_WithValidationErrors_MapsToValidationError test
- Add GetAsync_Returns401_MapsToUnauthorized test
- Add GetAsync_Returns403_MapsToForbidden test
- Add GetAsync_Returns500_MapsToApiError test
- Add FrameworkReference to Microsoft.AspNetCore.App for ValidationProblemDetails
Joseph Doherty