Design document for admin monitoring component that visualizes:
- Pipeline configuration from pipelines.json
- Execution status (last run, next required, overdue)
- Recent execution history (last 10 per schedule type)
- Schedule sections for Mass/Daily/Hourly with defaults vs overrides
- WorkProcessorReport.md: Analysis of legacy work processor from OLD solution
- Design document with clean architecture and component specifications
- Implementation plan with 15 TDD tasks
- Fix hourly lookback bug: use hourly timestamp/interval (not daily)
- Update DI registrations across DataSync, DataAccess, Api layers
- Add WorkProcessor config to appsettings.json
- Remove deprecated DataSyncService (replaced by WorkProcessor)
All 340 DataSync tests pass. Legacy bug from OLD solution now fixed.
- SearchRepository: Search table operations with Dapper
- SearchExecutionService: Search pipeline with proper cancellation handling
- WorkProcessor: Unified BackgroundService for syncs and searches
- SearchNotificationService: SignalR notifications in Api layer
All 45 new tests pass. Proper shutdown vs timeout distinction
prevents marking searches as error on host shutdown.
- Delete SyncMode.cs enum file
- Remove SyncModes property from PipelineConfig
- Remove SyncModeConfig and DestinationOverride records
- Remove WithMode(SyncMode) from IEtlPipelineBuilder
- Remove BuildWithSyncModes() and related methods from EtlPipelineFactory
- Remove syncModes sections from all pipelines in pipelines.json
- Update tests to use schedules-only configuration
All pipelines now require 'schedules' format (mass/daily/hourly).
WithUpdateType(UpdateTypes) is the only way to set update type.
- Add scheduleDefaults with mass/daily/hourly defaults
- Add massQuery (unfiltered) to each pipeline for Mass schedule
- Add schedules section to each pipeline inheriting defaults
- Keep syncModes for backward compatibility
Add optional customIntervals parameter to GetSyncStatusAsync to allow
per-pipeline interval overrides instead of hardcoded defaults. This
enables tables like MisData to use longer sync intervals (e.g., 70 days)
while other tables use standard intervals.
Key changes:
- IDataUpdateRepository.GetSyncStatusAsync now accepts an optional
Dictionary<string, int> for custom intervals keyed by "TableName_UpdateType"
- GetExpectedInterval and IsOverdue made public static for testing and reuse
- Added GetDefaultInterval method for accessing default values
- Updated DataSyncHealthCheck to use new signature
- Added comprehensive unit tests for custom interval behavior
Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add CreateGiwConnectionAsync to IDbConnectionFactory and DbConnectionFactory
for connecting to the GIW Oracle database. This connection is needed for the
StatusCode data sync pipeline.
- Reuses existing CreateOracleConnectionAsync helper with "GIW" data source
- Follows same pattern as JDE, JDEStage, and CMS connections
- Includes 4 unit tests covering missing/empty/invalid connection strings
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
Add configuration models for the new schedule-based pipeline configuration:
- ScheduleConfig: Per-schedule configuration (Enabled, IntervalMinutes, PrePurge, ReIndex, UpdateWhen)
- ScheduleDefaults: Default configurations for Mass (weekly), Daily, and Hourly schedules
- PipelineSchedules: Per-pipeline schedule overrides
- MergeWith method for merging pipeline overrides with defaults
Part of Task 1 from pipeline-schedule-alignment implementation plan.
Aligns pipelines.json with legacy DataSyncReport.md:
- Three schedules (Mass/Daily/Hourly) instead of two sync modes
- 8 missing pipelines to add
- GIW connection for StatusCode
- Per-pipeline schedule configuration
- Remove OutputColumnAttribute, OutputTableAttribute, OutputColumnCache
- Remove AttributeTableWriter and ColumnFormatter
- Remove duplicate ExcelFormats from Mapping (use Formatting version)
- Remove OutputColumn model
- Add FilterEntryMaps for criteria sheet filter models
- Update CriteriaSheetGenerator to use FluentTableWriter
- Remove attributes from filter entry models (now use fluent maps)
- Update DI to register filter entry maps and remove old services
- Update tests to use new fluent infrastructure
- Delete obsolete test files for removed infrastructure
Task 16 of fluent-excel-mapping-implementation plan.
- Replace AttributeTableWriter with FluentTableWriter in ExcelExportService
- Inject ExcelMapRegistry for fluent map lookups
- Use registry.GetMap<T>().TabName instead of reflection-based attribute reading
- Update ExcelIO SearchModel to reference Core result types via aliases
- Remove System.Reflection import (no longer needed)
- Add JdeScoping.Core.Models.SearchResults import for result types
- Move file size read after streams are disposed to get accurate compressed size
- Clean up definition files to use working example queries
- Add .gitignore for output directory
Add Verifier class that reads exported .pb.zstd files and:
- Deserializes protobuf data through zstd decompression
- Extracts schema information (column names and types)
- Counts rows for verification
- Optionally computes SHA256 hash and compares against sidecar file
Joseph Doherty