Design document for admin monitoring component that visualizes:
- Pipeline configuration from pipelines.json
- Execution status (last run, next required, overdue)
- Recent execution history (last 10 per schedule type)
- Schedule sections for Mass/Daily/Hourly with defaults vs overrides
- Fix hourly lookback bug: use hourly timestamp/interval (not daily)
- Update DI registrations across DataSync, DataAccess, Api layers
- Add WorkProcessor config to appsettings.json
- Remove deprecated DataSyncService (replaced by WorkProcessor)
All 340 DataSync tests pass. Legacy bug from OLD solution now fixed.
- SearchRepository: Search table operations with Dapper
- SearchExecutionService: Search pipeline with proper cancellation handling
- WorkProcessor: Unified BackgroundService for syncs and searches
- SearchNotificationService: SignalR notifications in Api layer
All 45 new tests pass. Proper shutdown vs timeout distinction
prevents marking searches as error on host shutdown.
- Delete SyncMode.cs enum file
- Remove SyncModes property from PipelineConfig
- Remove SyncModeConfig and DestinationOverride records
- Remove WithMode(SyncMode) from IEtlPipelineBuilder
- Remove BuildWithSyncModes() and related methods from EtlPipelineFactory
- Remove syncModes sections from all pipelines in pipelines.json
- Update tests to use schedules-only configuration
All pipelines now require 'schedules' format (mass/daily/hourly).
WithUpdateType(UpdateTypes) is the only way to set update type.
- Add scheduleDefaults with mass/daily/hourly defaults
- Add massQuery (unfiltered) to each pipeline for Mass schedule
- Add schedules section to each pipeline inheriting defaults
- Keep syncModes for backward compatibility
Add optional customIntervals parameter to GetSyncStatusAsync to allow
per-pipeline interval overrides instead of hardcoded defaults. This
enables tables like MisData to use longer sync intervals (e.g., 70 days)
while other tables use standard intervals.
Key changes:
- IDataUpdateRepository.GetSyncStatusAsync now accepts an optional
Dictionary<string, int> for custom intervals keyed by "TableName_UpdateType"
- GetExpectedInterval and IsOverdue made public static for testing and reuse
- Added GetDefaultInterval method for accessing default values
- Updated DataSyncHealthCheck to use new signature
- Added comprehensive unit tests for custom interval behavior
Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add CreateGiwConnectionAsync to IDbConnectionFactory and DbConnectionFactory
for connecting to the GIW Oracle database. This connection is needed for the
StatusCode data sync pipeline.
- Reuses existing CreateOracleConnectionAsync helper with "GIW" data source
- Follows same pattern as JDE, JDEStage, and CMS connections
- Includes 4 unit tests covering missing/empty/invalid connection strings
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
Add configuration models for the new schedule-based pipeline configuration:
- ScheduleConfig: Per-schedule configuration (Enabled, IntervalMinutes, PrePurge, ReIndex, UpdateWhen)
- ScheduleDefaults: Default configurations for Mass (weekly), Daily, and Hourly schedules
- PipelineSchedules: Per-pipeline schedule overrides
- MergeWith method for merging pipeline overrides with defaults
Part of Task 1 from pipeline-schedule-alignment implementation plan.
- Remove OutputColumnAttribute, OutputTableAttribute, OutputColumnCache
- Remove AttributeTableWriter and ColumnFormatter
- Remove duplicate ExcelFormats from Mapping (use Formatting version)
- Remove OutputColumn model
- Add FilterEntryMaps for criteria sheet filter models
- Update CriteriaSheetGenerator to use FluentTableWriter
- Remove attributes from filter entry models (now use fluent maps)
- Update DI to register filter entry maps and remove old services
- Update tests to use new fluent infrastructure
- Delete obsolete test files for removed infrastructure
Task 16 of fluent-excel-mapping-implementation plan.
- Replace AttributeTableWriter with FluentTableWriter in ExcelExportService
- Inject ExcelMapRegistry for fluent map lookups
- Use registry.GetMap<T>().TabName instead of reflection-based attribute reading
- Update ExcelIO SearchModel to reference Core result types via aliases
- Remove System.Reflection import (no longer needed)
- Add JdeScoping.Core.Models.SearchResults import for result types
Replace JsonZstdFileSource with ProtobufZstdFileSource across all 20
DevEtl pipeline files. This removes the need for manually defined
JsonColumnSchema arrays as protobuf files embed their schema.
Changes per file:
- Remove using JdeScoping.DataSync.Dev.Models
- Remove private JsonColumnSchema[] Schema array
- Change CacheFileName from .json.zstd to .pb.zstd
- Change WithSource from JsonZstdFileSource to ProtobufZstdFileSource
Add centralized SqlObjects class with constants for stored procedure
and function names using usp_/fn_ prefixes. Update LotFinderRepository
and MisQueryBuilder to use the new naming convention.
Remove dependency on deleted SQL Server Table-Valued Parameter types
by refactoring lookup methods to use built-in SQL Server functions:
- Simple single-value lookups (Items, WorkOrders, WorkCenters,
ProfitCenters, Users) now use STRING_SPLIT with comma-separated
strings from C#
- Complex multi-column lookup (Lots with LotNumber + ItemNumber)
now uses OPENJSON with JSON string from C#
This eliminates the need for TVP type definitions (scripts 033-039)
while maintaining equivalent functionality.
Joseph Doherty