- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Configure ETL pipelines for all 9 sync tables:
- WorkOrder_Curr, Lot, LotUsage_Curr (JDE)
- Item, WorkCenter, ProfitCenter (JDE)
- JdeUser, Branch (JDE)
- MisData (CMS) with postScript for ProcessedFlag update
Each pipeline includes:
- Source query with JDE Julian date/time parameters
- Sync modes for mass (365 day lookback, prePurge, reIndex)
and incremental (7 day lookback) operations
- Destination table with match columns and updateWhen conditions
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
Add 7 edge case tests for ApiClientBase:
- GetAsync_Returns200_EmptyBody_MapsToApiError
- GetAsync_Returns200_InvalidJson_MapsToApiError
- GetAsync_Returns204_ForUnitType_MapsToSuccess
- GetAsync_Returns204_ForNonUnitType_MapsToApiError
- GetAsync_NetworkException_MapsToApiError
- GetAsync_Timeout_MapsToApiError
- GetAsync_Returns400_WithoutValidationFormat_MapsToApiError
Also fix bug in ApiClientBase where 204 NoContent for Unit type
failed due to incorrect type casting. The implicit conversion from
Unit to ApiResult<Unit> must be applied before the runtime cast.
Add @using JdeScoping.Client.Extensions to _Imports.razor so that
extension methods (like ViewModelMappingExtensions) are available
globally in Razor components without explicit using statements.
Delete old service interfaces and implementations that have been
replaced by the new typed API clients:
- ISearchService/SearchService -> ISearchApiClient/SearchApiClient
- ILookupService/LookupService -> ILookupApiClient/LookupApiClient
- IFileService/FileService -> IFileApiClient/FileApiClient
Also remove unused IFileService injection from SearchEdit.razor
(download functionality now uses SearchApi.GetResultsAsync).
Remove ISearchService, ILookupService, and IFileService registrations
from Program.cs as these are now replaced by the typed API clients
(ISearchApiClient, ILookupApiClient, IFileApiClient).
Update ItemNumberFilterPanel.razor to use ILookupApiClient and IFileApiClient
instead of legacy ILookupService and IFileService interfaces. All methods now
use the result.Switch() pattern for proper ApiResult error handling.
- Replace IAuthService with IAuthApiClient for logout functionality
- Add @using JdeScoping.Core.ApiContracts
- LogoutAsync now calls AuthApi.LogoutAsync() and navigates to login
regardless of API result
- Replace IAuthService injection with IAuthApiClient
- Keep ICryptoService for credential encryption
- Add AuthStateProvider injection to notify authentication state
- Use result.Switch() pattern for ApiResult<LoginResultModel> handling
- Properly handle ValidationError with FieldErrors dictionary
Move dev-only JSON reading infrastructure from DataSync to DataSync.Dev:
- JsonColumnSchema (Models/)
- JsonZstdFileSource (Sources/)
- JsonStreamingDataReader (Sources/)
- Utf8JsonStreamingDataReader (Sources/)
Update namespaces and using statements in all DevEtl files.
- Replace ISearchService with ISearchApiClient
- Add @using for JdeScoping.Core.ApiContracts and JdeScoping.Client.Extensions
- Update LoadSearchAsync to use result.Switch() pattern with ApiResult<T>
- Handle CopySearchId, Id, and new search cases with proper error handling
- Use ToClient() extension method to convert Core to Client SearchViewModel
- Add _errorMessage field and display error alert in UI
- Update SubmitSearchInternalAsync and DownloadResultsAsync for consistency
- Add FormatValidationErrors helper for ValidationError.FieldErrors
- Replace ISearchService injection with ISearchApiClient
- Add @using for JdeScoping.Core.ApiContracts and JdeScoping.Client.Extensions
- Update LoadQueueAsync to use result.Switch() pattern with ApiResult<T>
- Add _errorMessage field for error display
- Add RadzenAlert for error message display in UI
- Replace ISearchService injection with ISearchApiClient
- Add @using for JdeScoping.Core.ApiContracts and JdeScoping.Client.Extensions
- Update LoadSearchesAsync to use ApiResult<T>.Switch() pattern
- Add _errorMessage field for error state
- Display RadzenAlert for error conditions
- Use ToClientList() extension method to convert Core->Client view models
- Create JdeScoping.DataSync.Dev for sandbox testing ETL code
- Create JdeScoping.DataSync.Dev.Tests for associated tests
- Move 22 source files and 8 test files
- Update namespaces from DevEtl to Dev
- Add both projects to solution
- Add using statement for JdeScoping.Client.Http namespace
- Register AuthRedirectHandler as a scoped service
- Replace simple HttpClient registration with handler pipeline
- AuthRedirectHandler intercepts 401 responses and redirects to login
- Keep all existing service registrations for backward compatibility
- Filter MERGE SQL columns to only include columns that exist in destination
(allColumns and updateColumns were using unfiltered source columns)
- Fix schema-qualified table names to use proper [schema].[table] format
instead of wrapping entire name in single brackets
- Add empty column mapping validation to throw early if no columns intersect
- Add JdeDateTransformer output column collision detection in OnInitialize
- Add TODO comment for WithCommandTimeout (stored but not yet passed to
destinations)
- Add tests for FormatQualifiedTableName and output column collision
- Add ParseTableName method to parse table names with optional schema
- Supports: "Table", "dbo.Table", "[dbo].[Table]"
- Returns (schema, table) tuple, defaults to "dbo" schema
- Update DisableIndexes, RebuildIndexes, UpdateStatistics to:
- Use QUOTENAME() for SQL injection protection
- Pass schema and table as parameters via SqlScriptRunner
- Support non-dbo schemas
- Update CustomSql to accept optional parameters and timeout
- Add comprehensive tests for ParseTableName with various formats
Joseph Doherty