Add ConfigurationValidationRunner with IConfigurationValidator interface for
validating required settings at startup. Includes SecureStore and LDAP validators.
Expand ConfigManager with pipeline editing UI, dialogs, and step editors.
Update documentation with config validation guidance.
Add comprehensive XML documentation (param/returns tags) across 132 source
files to improve IntelliSense and API discoverability. Include ConfigManager
design documents and implementation plans for phases 1-9.
Implement deferred code review findings:
- Add IDialogService/IClipboardService interfaces for testable platform operations
- Create AvaloniaDialogService and AvaloniaClipboardService implementations
- Extract dialog strings and file extensions to centralized Constants classes
- Refactor ViewModels to use DI instead of event delegates
- Update tests to use mock services
Document the MisData_Curr/Hist table split in architecture docs, add
post-sync archival scripts to move BackLevel records to history table,
and generate comprehensive unit test report (856 tests, 100% pass rate).
Split MisData table into MisData_Curr (Status='Current') and
MisData_Hist (Status='BackLevel') following existing archival pattern.
- Add MisData_Curr and MisData_Hist table scripts
- Create MisData view (UNION ALL) for backward compatibility
- Update production and DEV pipeline configurations
- Update unit tests for new table count (22 tables)
- Update database documentation
- Add GetAvailableTables, GetPipelineConfig, GetScheduleDefaults to IEtlPipelineFactory
- Implement new methods in EtlPipelineFactory
- Create PipelineController with endpoints:
- GET /api/pipelines - list all pipeline names
- GET /api/pipelines/{name} - get pipeline configuration
- GET /api/pipelines/{name}/status - get schedule status
- GET /api/pipelines/{name}/executions - get execution history
- Add JdeScoping.DataSync reference to JdeScoping.Api
- GetRecentUpdatesAsync: Returns last N DataUpdate records for a table,
optionally filtered by UpdateType, ordered by StartDt descending
- GetLastRunsAsync: Returns the most recent run (successful or not) for
each UpdateType for a given table using a CTE with ROW_NUMBER
- Fix hourly lookback bug: use hourly timestamp/interval (not daily)
- Update DI registrations across DataSync, DataAccess, Api layers
- Add WorkProcessor config to appsettings.json
- Remove deprecated DataSyncService (replaced by WorkProcessor)
All 340 DataSync tests pass. Legacy bug from OLD solution now fixed.
- SearchRepository: Search table operations with Dapper
- SearchExecutionService: Search pipeline with proper cancellation handling
- WorkProcessor: Unified BackgroundService for syncs and searches
- SearchNotificationService: SignalR notifications in Api layer
All 45 new tests pass. Proper shutdown vs timeout distinction
prevents marking searches as error on host shutdown.
- Delete SyncMode.cs enum file
- Remove SyncModes property from PipelineConfig
- Remove SyncModeConfig and DestinationOverride records
- Remove WithMode(SyncMode) from IEtlPipelineBuilder
- Remove BuildWithSyncModes() and related methods from EtlPipelineFactory
- Remove syncModes sections from all pipelines in pipelines.json
- Update tests to use schedules-only configuration
All pipelines now require 'schedules' format (mass/daily/hourly).
WithUpdateType(UpdateTypes) is the only way to set update type.
- Add scheduleDefaults with mass/daily/hourly defaults
- Add massQuery (unfiltered) to each pipeline for Mass schedule
- Add schedules section to each pipeline inheriting defaults
- Keep syncModes for backward compatibility
Add optional customIntervals parameter to GetSyncStatusAsync to allow
per-pipeline interval overrides instead of hardcoded defaults. This
enables tables like MisData to use longer sync intervals (e.g., 70 days)
while other tables use standard intervals.
Key changes:
- IDataUpdateRepository.GetSyncStatusAsync now accepts an optional
Dictionary<string, int> for custom intervals keyed by "TableName_UpdateType"
- GetExpectedInterval and IsOverdue made public static for testing and reuse
- Added GetDefaultInterval method for accessing default values
- Updated DataSyncHealthCheck to use new signature
- Added comprehensive unit tests for custom interval behavior
Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
Add configuration models for the new schedule-based pipeline configuration:
- ScheduleConfig: Per-schedule configuration (Enabled, IntervalMinutes, PrePurge, ReIndex, UpdateWhen)
- ScheduleDefaults: Default configurations for Mass (weekly), Daily, and Hourly schedules
- PipelineSchedules: Per-pipeline schedule overrides
- MergeWith method for merging pipeline overrides with defaults
Part of Task 1 from pipeline-schedule-alignment implementation plan.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
Replace the old sync logic (fetchers, merge configurations, bulk merge
helper, post processors) with the new ETL pipeline factory.
Changes:
- Inject IEtlPipelineFactory instead of old dependencies
- Remove IServiceProvider, IDbConnectionFactory, IBulkMergeHelper,
IMergeConfigurationRegistry dependencies
- Simplify ExecuteSyncCoreAsync to build and execute pipeline
- Keep DataUpdateRepository calls for tracking sync timestamps
- Determine SyncMode from UpdateType (Mass vs Incremental)
- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Configure ETL pipelines for all 9 sync tables:
- WorkOrder_Curr, Lot, LotUsage_Curr (JDE)
- Item, WorkCenter, ProfitCenter (JDE)
- JdeUser, Branch (JDE)
- MisData (CMS) with postScript for ProcessedFlag update
Each pipeline includes:
- Source query with JDE Julian date/time parameters
- Sync modes for mass (365 day lookback, prePurge, reIndex)
and incremental (7 day lookback) operations
- Destination table with match columns and updateWhen conditions
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
Move dev-only JSON reading infrastructure from DataSync to DataSync.Dev:
- JsonColumnSchema (Models/)
- JsonZstdFileSource (Sources/)
- JsonStreamingDataReader (Sources/)
- Utf8JsonStreamingDataReader (Sources/)
Update namespaces and using statements in all DevEtl files.
- Create JdeScoping.DataSync.Dev for sandbox testing ETL code
- Create JdeScoping.DataSync.Dev.Tests for associated tests
- Move 22 source files and 8 test files
- Update namespaces from DevEtl to Dev
- Add both projects to solution
Joseph Doherty