- Add scheduleDefaults with mass/daily/hourly defaults
- Add massQuery (unfiltered) to each pipeline for Mass schedule
- Add schedules section to each pipeline inheriting defaults
- Keep syncModes for backward compatibility
Add optional customIntervals parameter to GetSyncStatusAsync to allow
per-pipeline interval overrides instead of hardcoded defaults. This
enables tables like MisData to use longer sync intervals (e.g., 70 days)
while other tables use standard intervals.
Key changes:
- IDataUpdateRepository.GetSyncStatusAsync now accepts an optional
Dictionary<string, int> for custom intervals keyed by "TableName_UpdateType"
- GetExpectedInterval and IsOverdue made public static for testing and reuse
- Added GetDefaultInterval method for accessing default values
- Updated DataSyncHealthCheck to use new signature
- Added comprehensive unit tests for custom interval behavior
Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add CreateGiwConnectionAsync to IDbConnectionFactory and DbConnectionFactory
for connecting to the GIW Oracle database. This connection is needed for the
StatusCode data sync pipeline.
- Reuses existing CreateOracleConnectionAsync helper with "GIW" data source
- Follows same pattern as JDE, JDEStage, and CMS connections
- Includes 4 unit tests covering missing/empty/invalid connection strings
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
Add configuration models for the new schedule-based pipeline configuration:
- ScheduleConfig: Per-schedule configuration (Enabled, IntervalMinutes, PrePurge, ReIndex, UpdateWhen)
- ScheduleDefaults: Default configurations for Mass (weekly), Daily, and Hourly schedules
- PipelineSchedules: Per-pipeline schedule overrides
- MergeWith method for merging pipeline overrides with defaults
Part of Task 1 from pipeline-schedule-alignment implementation plan.
- Remove OutputColumnAttribute, OutputTableAttribute, OutputColumnCache
- Remove AttributeTableWriter and ColumnFormatter
- Remove duplicate ExcelFormats from Mapping (use Formatting version)
- Remove OutputColumn model
- Add FilterEntryMaps for criteria sheet filter models
- Update CriteriaSheetGenerator to use FluentTableWriter
- Remove attributes from filter entry models (now use fluent maps)
- Update DI to register filter entry maps and remove old services
- Update tests to use new fluent infrastructure
- Delete obsolete test files for removed infrastructure
Task 16 of fluent-excel-mapping-implementation plan.
- Replace AttributeTableWriter with FluentTableWriter in ExcelExportService
- Inject ExcelMapRegistry for fluent map lookups
- Use registry.GetMap<T>().TabName instead of reflection-based attribute reading
- Update ExcelIO SearchModel to reference Core result types via aliases
- Remove System.Reflection import (no longer needed)
- Add JdeScoping.Core.Models.SearchResults import for result types
Replace JsonZstdFileSource with ProtobufZstdFileSource across all 20
DevEtl pipeline files. This removes the need for manually defined
JsonColumnSchema arrays as protobuf files embed their schema.
Changes per file:
- Remove using JdeScoping.DataSync.Dev.Models
- Remove private JsonColumnSchema[] Schema array
- Change CacheFileName from .json.zstd to .pb.zstd
- Change WithSource from JsonZstdFileSource to ProtobufZstdFileSource
Add centralized SqlObjects class with constants for stored procedure
and function names using usp_/fn_ prefixes. Update LotFinderRepository
and MisQueryBuilder to use the new naming convention.
Remove dependency on deleted SQL Server Table-Valued Parameter types
by refactoring lookup methods to use built-in SQL Server functions:
- Simple single-value lookups (Items, WorkOrders, WorkCenters,
ProfitCenters, Users) now use STRING_SPLIT with comma-separated
strings from C#
- Complex multi-column lookup (Lots with LotNumber + ItemNumber)
now uses OPENJSON with JSON string from C#
This eliminates the need for TVP type definitions (scripts 033-039)
while maintaining equivalent functionality.
- Remove all List<*FilterEntry> properties and *FilterEnabled computed properties from SearchModel
- Delete TableValuedParameterExtensions.cs
- Delete entire FilterEntries folder and all filter entry model classes
- Delete FilterHandlers folder and all filter handler classes
- Delete IFilterHandler interface and FilterResult model
- Update MisQueryBuilder to use SQL extraction functions instead of model properties
- Update SearchProcessor to get ExtractMisData from database using fn_GetSearchExtractMisData
- Update DependencyInjection to remove filter handler registrations
- Delete obsolete tests for TVP extensions and filter handlers
Filter criteria are now stored as JSON in Search.Criteria column and extracted using SQL functions (fn_GetSearch*) during query execution.
Remove legacy ETL properties from DataSources configuration:
- FetcherTypeName (9 entries)
- PostProcessorTypeName (1 entry - MisData)
- PrepurgeData from MassConfig (9 entries)
- ReIndexData from MassConfig (9 entries)
These properties were used by the old fetcher-based ETL system
and are no longer needed with the new EtlPipeline architecture.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
- Change interface methods to accept int searchId instead of SearchModel
- Update SqlKataSearchQueryBuilder to generate SQL using extraction functions
- SQL now calls dbo.fn_GetSearchWorkOrders(@SearchId) etc instead of TVPs
- Update SearchProcessor to pass model.Id to query builder
- Update tests for new method signatures
Joseph Doherty