Add ConfigurationValidationRunner with IConfigurationValidator interface for
validating required settings at startup. Includes SecureStore and LDAP validators.
Expand ConfigManager with pipeline editing UI, dialogs, and step editors.
Update documentation with config validation guidance.
Implement deferred code review findings:
- Add IDialogService/IClipboardService interfaces for testable platform operations
- Create AvaloniaDialogService and AvaloniaClipboardService implementations
- Extract dialog strings and file extensions to centralized Constants classes
- Refactor ViewModels to use DI instead of event delegates
- Update tests to use mock services
Remove 9 unused types from Core (duplicate extension classes, TableSpec, ColumnSpec, LotLocation), move ComponentLotViewModel and OperatorViewModel from Client to Core, and refactor DataSync.Dev to use pipeline-based configuration. Fix Login.razor to use UserInfoDto directly.
- Fix hourly lookback bug: use hourly timestamp/interval (not daily)
- Update DI registrations across DataSync, DataAccess, Api layers
- Add WorkProcessor config to appsettings.json
- Remove deprecated DataSyncService (replaced by WorkProcessor)
All 340 DataSync tests pass. Legacy bug from OLD solution now fixed.
- SearchRepository: Search table operations with Dapper
- SearchExecutionService: Search pipeline with proper cancellation handling
- WorkProcessor: Unified BackgroundService for syncs and searches
- SearchNotificationService: SignalR notifications in Api layer
All 45 new tests pass. Proper shutdown vs timeout distinction
prevents marking searches as error on host shutdown.
- Delete SyncMode.cs enum file
- Remove SyncModes property from PipelineConfig
- Remove SyncModeConfig and DestinationOverride records
- Remove WithMode(SyncMode) from IEtlPipelineBuilder
- Remove BuildWithSyncModes() and related methods from EtlPipelineFactory
- Remove syncModes sections from all pipelines in pipelines.json
- Update tests to use schedules-only configuration
All pipelines now require 'schedules' format (mass/daily/hourly).
WithUpdateType(UpdateTypes) is the only way to set update type.
Add optional customIntervals parameter to GetSyncStatusAsync to allow
per-pipeline interval overrides instead of hardcoded defaults. This
enables tables like MisData to use longer sync intervals (e.g., 70 days)
while other tables use standard intervals.
Key changes:
- IDataUpdateRepository.GetSyncStatusAsync now accepts an optional
Dictionary<string, int> for custom intervals keyed by "TableName_UpdateType"
- GetExpectedInterval and IsOverdue made public static for testing and reuse
- Added GetDefaultInterval method for accessing default values
- Updated DataSyncHealthCheck to use new signature
- Added comprehensive unit tests for custom interval behavior
Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
Add configuration models for the new schedule-based pipeline configuration:
- ScheduleConfig: Per-schedule configuration (Enabled, IntervalMinutes, PrePurge, ReIndex, UpdateWhen)
- ScheduleDefaults: Default configurations for Mass (weekly), Daily, and Hourly schedules
- PipelineSchedules: Per-pipeline schedule overrides
- MergeWith method for merging pipeline overrides with defaults
Part of Task 1 from pipeline-schedule-alignment implementation plan.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Extend DbQuerySource to support multiple connection types:
- Add connectionType parameter ("jde", "cms", "lotfinder")
- Use appropriate IDbConnectionFactory method for each type
- Support Dictionary<string, object> parameters
- Use DbConnection/DbCommand for cross-database compatibility
- Create JdeScoping.DataSync.Dev for sandbox testing ETL code
- Create JdeScoping.DataSync.Dev.Tests for associated tests
- Move 22 source files and 8 test files
- Update namespaces from DevEtl to Dev
- Add both projects to solution
Add integration tests for the Branch development ETL pipeline:
- BranchDevEtlTests with tests for pipeline creation and execution
- Tests verify pipeline creates correctly and loads data from cache files
- Added appsettings.json with local connection string and cache directory
- Added necessary packages (Configuration.Json, SqlClient, Dapper)
Tests require local SQL Server and CACHED_DB_FILES directory with
branch.json.zstd to pass; tests silently skip if resources unavailable.
- Filter MERGE SQL columns to only include columns that exist in destination
(allColumns and updateColumns were using unfiltered source columns)
- Fix schema-qualified table names to use proper [schema].[table] format
instead of wrapping entire name in single brackets
- Add empty column mapping validation to throw early if no columns intersect
- Add JdeDateTransformer output column collision detection in OnInitialize
- Add TODO comment for WithCommandTimeout (stored but not yet passed to
destinations)
- Add tests for FormatQualifiedTableName and output column collision
- Add ParseTableName method to parse table names with optional schema
- Supports: "Table", "dbo.Table", "[dbo].[Table]"
- Returns (schema, table) tuple, defaults to "dbo" schema
- Update DisableIndexes, RebuildIndexes, UpdateStatistics to:
- Use QUOTENAME() for SQL injection protection
- Pass schema and table as parameters via SqlScriptRunner
- Support non-dbo schemas
- Update CustomSql to accept optional parameters and timeout
- Add comprehensive tests for ParseTableName with various formats
- Add DefaultInvalidDateSentinel (1900-01-01) for invalid date handling
- Add optional invalidDateSentinel constructor parameter
- Add MapOrdinal override returning -1 for computed DateTime column
- Add GetDataTypeName override returning "datetime" for computed column
- Update ParseJdeDateTime with comprehensive validation:
- Validate date is positive
- Validate century (0 or 1)
- Validate year (0-99)
- Validate day of year (1-366 and respects leap year)
- Validate time components (hours 0-23, minutes/seconds 0-59)
- Add tests for all new functionality
Add virtual methods to DataTransformerBase for GetBytes, GetChars,
GetData, and GetDataTypeName that properly handle computed columns
by throwing NotSupportedException when MapOrdinal returns -1.
Update TransformingDataReader to delegate these methods to the
transformer instead of directly to the source reader.
Add MapOrdinal method to the IDataTransformer interface and provide
a default implementation in DataTransformerBase. This enables
transformers to report the mapping between transformed ordinals and
source ordinals, supporting use cases like computed columns which
return -1 to indicate no source ordinal.
- Add MapOrdinal(int, IDataReader) to IDataTransformer interface
- Add virtual MapOrdinal implementation in DataTransformerBase
- Add DataTransformerBaseTests with test for default behavior
Adds DbQuerySource, an IImportSource implementation that executes SQL
queries against the LotFinder database. Supports parameterized queries
using anonymous objects and configurable command timeouts.
Add transformer that combines JDE Julian date (CYYDDD) and time (HHMMSS)
columns into a single DateTime column. Includes static ParseJdeDateTime
method for direct date conversion.
Add a data transformer that removes specified columns from the data stream.
Columns are matched by name (case-insensitive) and multiple columns can be
dropped in a single transformer. Includes comprehensive tests using NSubstitute
for mock IDataReader.
Add core transformer infrastructure for the ETL pipeline:
- DataTransformerBase: abstract base class with virtual methods for
field count, names, types, values, ordinals, and null checking
- TransformingDataReader: IDataReader wrapper that delegates to
transformer, enabling on-the-fly data transformations
Add SqlScriptRunner class that implements IScriptRunner for executing
SQL scripts against the LotFinderDB cache database. Includes constructor
validation and configurable timeout support (default 1 hour).
Move configuration options from Core/DataAccess/DataSync/ExcelIO to
dedicated Options folders within each project for better organization.
Update all references and tests accordingly.
Set up repository with legacy .NET Framework 4.8 source (OLD/),
new .NET 10 Blazor solution (NEW/), OpenSpec specifications,
documentation, and project configuration.
Joseph Doherty