Add ConfigurationValidationRunner with IConfigurationValidator interface for
validating required settings at startup. Includes SecureStore and LDAP validators.
Expand ConfigManager with pipeline editing UI, dialogs, and step editors.
Update documentation with config validation guidance.
Remove 9 unused types from Core (duplicate extension classes, TableSpec, ColumnSpec, LotLocation), move ComponentLotViewModel and OperatorViewModel from Client to Core, and refactor DataSync.Dev to use pipeline-based configuration. Fix Login.razor to use UserInfoDto directly.
- SearchRepository: Search table operations with Dapper
- SearchExecutionService: Search pipeline with proper cancellation handling
- WorkProcessor: Unified BackgroundService for syncs and searches
- SearchNotificationService: SignalR notifications in Api layer
All 45 new tests pass. Proper shutdown vs timeout distinction
prevents marking searches as error on host shutdown.
- Delete SyncMode.cs enum file
- Remove SyncModes property from PipelineConfig
- Remove SyncModeConfig and DestinationOverride records
- Remove WithMode(SyncMode) from IEtlPipelineBuilder
- Remove BuildWithSyncModes() and related methods from EtlPipelineFactory
- Remove syncModes sections from all pipelines in pipelines.json
- Update tests to use schedules-only configuration
All pipelines now require 'schedules' format (mass/daily/hourly).
WithUpdateType(UpdateTypes) is the only way to set update type.
Add optional customIntervals parameter to GetSyncStatusAsync to allow
per-pipeline interval overrides instead of hardcoded defaults. This
enables tables like MisData to use longer sync intervals (e.g., 70 days)
while other tables use standard intervals.
Key changes:
- IDataUpdateRepository.GetSyncStatusAsync now accepts an optional
Dictionary<string, int> for custom intervals keyed by "TableName_UpdateType"
- GetExpectedInterval and IsOverdue made public static for testing and reuse
- Added GetDefaultInterval method for accessing default values
- Updated DataSyncHealthCheck to use new signature
- Added comprehensive unit tests for custom interval behavior
Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Set up repository with legacy .NET Framework 4.8 source (OLD/),
new .NET 10 Blazor solution (NEW/), OpenSpec specifications,
documentation, and project configuration.
Joseph Doherty