Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add CreateGiwConnectionAsync to IDbConnectionFactory and DbConnectionFactory
for connecting to the GIW Oracle database. This connection is needed for the
StatusCode data sync pipeline.
- Reuses existing CreateOracleConnectionAsync helper with "GIW" data source
- Follows same pattern as JDE, JDEStage, and CMS connections
- Includes 4 unit tests covering missing/empty/invalid connection strings
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
Add configuration models for the new schedule-based pipeline configuration:
- ScheduleConfig: Per-schedule configuration (Enabled, IntervalMinutes, PrePurge, ReIndex, UpdateWhen)
- ScheduleDefaults: Default configurations for Mass (weekly), Daily, and Hourly schedules
- PipelineSchedules: Per-pipeline schedule overrides
- MergeWith method for merging pipeline overrides with defaults
Part of Task 1 from pipeline-schedule-alignment implementation plan.
- Remove OutputColumnAttribute, OutputTableAttribute, OutputColumnCache
- Remove AttributeTableWriter and ColumnFormatter
- Remove duplicate ExcelFormats from Mapping (use Formatting version)
- Remove OutputColumn model
- Add FilterEntryMaps for criteria sheet filter models
- Update CriteriaSheetGenerator to use FluentTableWriter
- Remove attributes from filter entry models (now use fluent maps)
- Update DI to register filter entry maps and remove old services
- Update tests to use new fluent infrastructure
- Delete obsolete test files for removed infrastructure
Task 16 of fluent-excel-mapping-implementation plan.
- Replace AttributeTableWriter with FluentTableWriter in ExcelExportService
- Inject ExcelMapRegistry for fluent map lookups
- Use registry.GetMap<T>().TabName instead of reflection-based attribute reading
- Update ExcelIO SearchModel to reference Core result types via aliases
- Remove System.Reflection import (no longer needed)
- Add JdeScoping.Core.Models.SearchResults import for result types
Replace JsonZstdFileSource with ProtobufZstdFileSource across all 20
DevEtl pipeline files. This removes the need for manually defined
JsonColumnSchema arrays as protobuf files embed their schema.
Changes per file:
- Remove using JdeScoping.DataSync.Dev.Models
- Remove private JsonColumnSchema[] Schema array
- Change CacheFileName from .json.zstd to .pb.zstd
- Change WithSource from JsonZstdFileSource to ProtobufZstdFileSource
Add centralized SqlObjects class with constants for stored procedure
and function names using usp_/fn_ prefixes. Update LotFinderRepository
and MisQueryBuilder to use the new naming convention.
Remove dependency on deleted SQL Server Table-Valued Parameter types
by refactoring lookup methods to use built-in SQL Server functions:
- Simple single-value lookups (Items, WorkOrders, WorkCenters,
ProfitCenters, Users) now use STRING_SPLIT with comma-separated
strings from C#
- Complex multi-column lookup (Lots with LotNumber + ItemNumber)
now uses OPENJSON with JSON string from C#
This eliminates the need for TVP type definitions (scripts 033-039)
while maintaining equivalent functionality.
- Remove all List<*FilterEntry> properties and *FilterEnabled computed properties from SearchModel
- Delete TableValuedParameterExtensions.cs
- Delete entire FilterEntries folder and all filter entry model classes
- Delete FilterHandlers folder and all filter handler classes
- Delete IFilterHandler interface and FilterResult model
- Update MisQueryBuilder to use SQL extraction functions instead of model properties
- Update SearchProcessor to get ExtractMisData from database using fn_GetSearchExtractMisData
- Update DependencyInjection to remove filter handler registrations
- Delete obsolete tests for TVP extensions and filter handlers
Filter criteria are now stored as JSON in Search.Criteria column and extracted using SQL functions (fn_GetSearch*) during query execution.
Remove legacy ETL properties from DataSources configuration:
- FetcherTypeName (9 entries)
- PostProcessorTypeName (1 entry - MisData)
- PrepurgeData from MassConfig (9 entries)
- ReIndexData from MassConfig (9 entries)
These properties were used by the old fetcher-based ETL system
and are no longer needed with the new EtlPipeline architecture.
- Remove FetcherTypeName assignments from test fixtures in ScheduleCheckerTests
and SyncOrchestratorTests
- Remove PrepurgeData and ReIndexData assignments from MassConfig in tests
- Mark FetcherTypeName, PostProcessorTypeName, PrepurgeData, and ReIndexData
as obsolete with deprecation warnings pointing to pipelines.json
- Change FetcherTypeName from required to optional to allow tests to compile
without providing the deprecated property
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
- Change interface methods to accept int searchId instead of SearchModel
- Update SqlKataSearchQueryBuilder to generate SQL using extraction functions
- SQL now calls dbo.fn_GetSearchWorkOrders(@SearchId) etc instead of TVPs
- Update SearchProcessor to pass model.Id to query builder
- Update tests for new method signatures
Replace the old sync logic (fetchers, merge configurations, bulk merge
helper, post processors) with the new ETL pipeline factory.
Changes:
- Inject IEtlPipelineFactory instead of old dependencies
- Remove IServiceProvider, IDbConnectionFactory, IBulkMergeHelper,
IMergeConfigurationRegistry dependencies
- Simplify ExecuteSyncCoreAsync to build and execute pipeline
- Keep DataUpdateRepository calls for tracking sync timestamps
- Determine SyncMode from UpdateType (Mass vs Incremental)
- Replace placeholder postScript with actual MIS data post-processing:
1. Sets ObsoleteDate based on BackLevel records
2. Sets ObsoleteDate for remaining NULL cases
3. Rebuilds PK_MisData index
- Add massQuery support to SourceConfig for mode-specific queries
- MisData mass sync now uses query without date filter (like legacy)
- EtlPipelineFactory selects massQuery when in mass mode if available
- Remove unnecessary minDtOffset from MisData mass mode config
Configure ETL pipelines for all 9 sync tables:
- WorkOrder_Curr, Lot, LotUsage_Curr (JDE)
- Item, WorkCenter, ProfitCenter (JDE)
- JdeUser, Branch (JDE)
- MisData (CMS) with postScript for ProcessedFlag update
Each pipeline includes:
- Source query with JDE Julian date/time parameters
- Sync modes for mass (365 day lookback, prePurge, reIndex)
and incremental (7 day lookback) operations
- Destination table with match columns and updateWhen conditions
- Implement IEtlPipelineFactory with ForTable() method returning a builder
- Load pipeline config from JSON file path (from PipelineOptions)
- Parse config using System.Text.Json with PropertyNameCaseInsensitive
- Builder supports WithMode() and WithMinimumDate() fluent methods
- Create DbQuerySource for source with ParameterFormatConverter for JDE dates
- Create DbBulkMergeDestination or DbBulkImportDestination based on sync mode
- Mass mode defaults to bulkImport, incremental defaults to bulkMerge
- Support destination override in sync mode config
- Execute pre/post scripts from config (prePurge, reIndex, custom scripts)
- Validate config: require mass and incremental modes, reject runtime params
- Add comprehensive tests for factory, builder, and config validation
Joseph Doherty