- Change interface methods to accept int searchId instead of SearchModel
- Update SqlKataSearchQueryBuilder to generate SQL using extraction functions
- SQL now calls dbo.fn_GetSearchWorkOrders(@SearchId) etc instead of TVPs
- Update SearchProcessor to pass model.Id to query builder
- Update tests for new method signatures
Replace the old sync logic (fetchers, merge configurations, bulk merge
helper, post processors) with the new ETL pipeline factory.
Changes:
- Inject IEtlPipelineFactory instead of old dependencies
- Remove IServiceProvider, IDbConnectionFactory, IBulkMergeHelper,
IMergeConfigurationRegistry dependencies
- Simplify ExecuteSyncCoreAsync to build and execute pipeline
- Keep DataUpdateRepository calls for tracking sync timestamps
- Determine SyncMode from UpdateType (Mass vs Incremental)
- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Configure ETL pipelines for all 9 sync tables:
- WorkOrder_Curr, Lot, LotUsage_Curr (JDE)
- Item, WorkCenter, ProfitCenter (JDE)
- JdeUser, Branch (JDE)
- MisData (CMS) with postScript for ProcessedFlag update
Each pipeline includes:
- Source query with JDE Julian date/time parameters
- Sync modes for mass (365 day lookback, prePurge, reIndex)
and incremental (7 day lookback) operations
- Destination table with match columns and updateWhen conditions
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Add comprehensive tests for the 5 simple table extraction functions:
- fn_GetSearchWorkOrders
- fn_GetSearchItemNumbers
- fn_GetSearchProfitCenters
- fn_GetSearchWorkCenters
- fn_GetSearchOperatorIDs
Each function tested for:
- Valid array returns all values
- Empty array returns empty
- Missing property returns empty
- Search not found returns empty
- NULL criteria returns empty
- Invalid JSON returns empty
- Bad type values (nulls) filtered out
Additional edge case tests for large arrays and long values.
Total: 38 tests (7 per function + 2 additional edge cases).
Add DatabaseTestBase.cs with xUnit Collection for test isolation:
- DatabaseTestCollection disables parallel execution
- DatabaseTestFixture verifies database connectivity
- DatabaseTestBase provides connection and cleanup helpers
- InsertTestSearchAsync for creating test data with SearchCriteria
- InsertTestSearchWithRawCriteriaAsync for testing invalid JSON scenarios
Also adds required packages: Dapper, FluentAssertions, Microsoft.Data.SqlClient
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
Key changes from Codex feedback:
- THROW cannot be used in UDFs (SQL Server restriction)
- Multi-statement TVFs have poor performance (no stats)
- Changed to inline TVFs for extraction (optimal performance)
- Added validation stored procedure for strict error handling
- Use TRY_CONVERT for safe type conversion
- Use OPENJSON...WITH for complex object extraction
- Replace IsApiError with IsError
- Replace ApiError. with Error.
- Replace ValidationError.Errors with ValidationError.FieldErrors
These match the actual ApiResult<T> implementation. The tests
themselves were already correct - subagents adapted during implementation.
Add integration tests for SearchApiClient covering:
- GetUserSearchesAsync with authentication returns success
- GetUserSearchesAsync without authentication returns unauthorized
- GetSearchAsync with non-existent ID returns not found
Add comprehensive unit tests for FileApiClient covering:
- Route verification tests for all 4 download endpoints (POST method)
- Route verification tests for 2 upload endpoints (POST method)
- Success tests for downloads returning byte arrays
- Success tests verifying existing data can be passed to downloads
- Success tests for uploads returning IReadOnlyList<T>
- Multipart content-type and filename verification test
- Error tests (404 for download, 400 validation error for upload)
Add unit tests for AuthApiClient covering:
- Route verification tests for GetPublicKey, Login, Logout, GetCurrentUser
- Success tests for all 4 methods
- Proper handling of 204 NoContent for Logout (returns Unit)
- Route verification tests for all 4 methods (FindItems, FindProfitCenters,
FindWorkCenters, FindOperators)
- Query string encoding tests verifying special characters (&, =) and spaces
are properly URL-encoded
- Success tests for all lookup methods verifying deserialization
- Representative error test for 500 status code
Add comprehensive unit tests for SearchApiClient covering:
- Route verification tests for all 6 methods (GetUserSearches, GetQueuedSearches,
GetSearch, CopySearch, CreateSearch, GetResults)
- Success tests verifying deserialization for all methods
- Representative error tests (404 for GetSearch, 401 for GetUserSearches)
Uses MockHttpMessageHandler to verify correct API routes are called
and ApiResult<T> pattern maps responses correctly.
Add 7 edge case tests for ApiClientBase:
- GetAsync_Returns200_EmptyBody_MapsToApiError
- GetAsync_Returns200_InvalidJson_MapsToApiError
- GetAsync_Returns204_ForUnitType_MapsToSuccess
- GetAsync_Returns204_ForNonUnitType_MapsToApiError
- GetAsync_NetworkException_MapsToApiError
- GetAsync_Timeout_MapsToApiError
- GetAsync_Returns400_WithoutValidationFormat_MapsToApiError
Also fix bug in ApiClientBase where 204 NoContent for Unit type
failed due to incorrect type casting. The implicit conversion from
Unit to ApiResult<Unit> must be applied before the runtime cast.
- Add GetAsync_Returns404_MapsToNotFound test
- Add GetAsync_Returns400_WithValidationErrors_MapsToValidationError test
- Add GetAsync_Returns401_MapsToUnauthorized test
- Add GetAsync_Returns403_MapsToForbidden test
- Add GetAsync_Returns500_MapsToApiError test
- Add FrameworkReference to Microsoft.AspNetCore.App for ValidationProblemDetails
Added using statement for JdeScoping.Core.ApiContracts. The existing
routes (/api/file/...) are left unchanged as they test legacy download
by cache key endpoints which differ from the new FileIO routes.
Add @using JdeScoping.Client.Extensions to _Imports.razor so that
extension methods (like ViewModelMappingExtensions) are available
globally in Razor components without explicit using statements.
Delete old service interfaces and implementations that have been
replaced by the new typed API clients:
- ISearchService/SearchService -> ISearchApiClient/SearchApiClient
- ILookupService/LookupService -> ILookupApiClient/LookupApiClient
- IFileService/FileService -> IFileApiClient/FileApiClient
Also remove unused IFileService injection from SearchEdit.razor
(download functionality now uses SearchApi.GetResultsAsync).
Remove ISearchService, ILookupService, and IFileService registrations
from Program.cs as these are now replaced by the typed API clients
(ISearchApiClient, ILookupApiClient, IFileApiClient).
Joseph Doherty