Move SecureStoreManager project and tests to Deprecated folder and remove
from solution. SecureStore functionality is now integrated into ConfigManager.
Add comprehensive tests for services (ConnectionTestService, RuntimeConfigValidation),
ViewModels (PipelineEditor, dialogs, transformers), and Avalonia headless UI tests
for views and forms.
Eliminates placeholder substitution (${KEY}) in favor of storing complete
connection strings as single encrypted values. SecureStore now auto-creates
entries for all connection strings defined in appsettings. ConfigManager
editor reads/writes values directly to SecureStore.
Align ConfigManager with DataSync's per-file pipeline format (pipeline.*.json)
by reusing EtlPipelineConfig types directly, eliminating duplicate models and
simplifying the codebase. Removes ~3200 lines of obsolete code.
- Add pipeline registry with JSON-based configuration and hot-reload support
- Implement manual sync request feature with API, client UI, and database
- Improve ConfigManager: connection string dropdown in pipeline editor,
step delete/reorder functionality, and fix JSON parsing for ConnectionStrings
Adds a new ConnectionStrings section to ConfigManager allowing users to manage
database connection strings with provider selection, connection testing, and
visual feedback for connection state.
Consolidate SecureStoreLockedFormView and SecureStoreUnlockedFormView into
a single SecureStoreInfoFormView that displays store status and metadata.
Simplifies MainWindowViewModel by removing redundant state management.
Also adds design docs for RegexTransformer feature.
Tests cover:
- Loading from model
- Serializing to model
- Test pattern command (success and error cases)
- Mode switching and help text
- Summary display
- Change notification
Enable ConfigManager to validate runtime configuration (SecureStore secrets,
connection strings, LDAP) using the same validators as the Host application.
Adds AddInfrastructureValidators() extension for shared validator registration.
Add ConfigurationValidationRunner with IConfigurationValidator interface for
validating required settings at startup. Includes SecureStore and LDAP validators.
Expand ConfigManager with pipeline editing UI, dialogs, and step editors.
Update documentation with config validation guidance.
Add SecureStore integration to ConfigManager for secure handling of connection
strings and sensitive configuration values. Includes store/secret management
UI, encrypted .store file support, and comprehensive test coverage.
- Add IFileSystem and IDialogService dependencies to constructor
- Implement OnSelectedNodeChanged to create appropriate form ViewModels
- Add LoadConfigForTesting helper for unit testing
- Add MarkAsChanged helper to track unsaved changes
- Update OpenFolderAsync to use IDialogService
- Add comprehensive unit tests for form selection
Add form ViewModels for editing pipeline configurations in the ConfigManager.
ScheduleFormViewModel wraps ScheduleModel for schedule editing.
PipelineFormViewModel wraps PipelineModel with schedule sub-ViewModels.
Implement LdapFormViewModel for editing LDAP configuration section with
properties for ServerUrlsText, GroupDn, SearchBase, ConnectionTimeoutSeconds,
UseFakeAuth, and AdminBypassUsersText. Array properties use newline-separated
text with StringSplitOptions.RemoveEmptyEntries | TrimEntries for splitting.
Implements Task 18 from phases 7-9 plan. SearchFormViewModel wraps
SearchSection model with properties for MaxResultRows, TimeoutSeconds,
and MaxConcurrentSearches. Includes full test coverage with 7 tests
verifying initialization, two-way binding, change notification, and
null argument handling.
Add service for auto-discovering configuration file locations. The
service searches in prioritized order:
1. JDESCOPING_CONFIG_PATH environment variable
2. Same directory as executable
3. ../JdeScoping.Host/ relative to executable
4. User config directory (~/.jdescoping on Unix, %LOCALAPPDATA%\JdeScoping on Windows)
Includes 9 unit tests covering all search locations, priority order,
edge cases (missing directory, missing appsettings.json), and
cancellation token support.
Add config file loading and saving service following TDD approach:
- IConfigFileService interface for loading/saving config files
- ConfigLoadException for descriptive error handling
- ConfigFileService implementation with JSON serialization
- Unit tests with mocked IFileSystem dependency
Add file system abstraction to enable testability for file operations.
- IFileSystem interface with common file operations
- FileSystem implementation wrapping System.IO
- Unit tests for FileExists functionality
Implement deferred code review findings:
- Add IDialogService/IClipboardService interfaces for testable platform operations
- Create AvaloniaDialogService and AvaloniaClipboardService implementations
- Extract dialog strings and file extensions to centralized Constants classes
- Refactor ViewModels to use DI instead of event delegates
- Update tests to use mock services
- Fix CriteriaSheetGenerator.FormatTimestamp to handle all DateTimeKind values
- Update TestWebApplicationFactory to use IAuthenticationService
- Add logger parameter to ExcelParserServiceTests
- Add SecureStoreManager to solution under /utils/ folder
Split MisData table into MisData_Curr (Status='Current') and
MisData_Hist (Status='BackLevel') following existing archival pattern.
- Add MisData_Curr and MisData_Hist table scripts
- Create MisData view (UNION ALL) for backward compatibility
- Update production and DEV pipeline configurations
- Update unit tests for new table count (22 tables)
- Update database documentation
Remove 9 unused types from Core (duplicate extension classes, TableSpec, ColumnSpec, LotLocation), move ComponentLotViewModel and OperatorViewModel from Client to Core, and refactor DataSync.Dev to use pipeline-based configuration. Fix Login.razor to use UserInfoDto directly.
- Fix hourly lookback bug: use hourly timestamp/interval (not daily)
- Update DI registrations across DataSync, DataAccess, Api layers
- Add WorkProcessor config to appsettings.json
- Remove deprecated DataSyncService (replaced by WorkProcessor)
All 340 DataSync tests pass. Legacy bug from OLD solution now fixed.
- SearchRepository: Search table operations with Dapper
- SearchExecutionService: Search pipeline with proper cancellation handling
- WorkProcessor: Unified BackgroundService for syncs and searches
- SearchNotificationService: SignalR notifications in Api layer
All 45 new tests pass. Proper shutdown vs timeout distinction
prevents marking searches as error on host shutdown.
- Delete SyncMode.cs enum file
- Remove SyncModes property from PipelineConfig
- Remove SyncModeConfig and DestinationOverride records
- Remove WithMode(SyncMode) from IEtlPipelineBuilder
- Remove BuildWithSyncModes() and related methods from EtlPipelineFactory
- Remove syncModes sections from all pipelines in pipelines.json
- Update tests to use schedules-only configuration
All pipelines now require 'schedules' format (mass/daily/hourly).
WithUpdateType(UpdateTypes) is the only way to set update type.
Add optional customIntervals parameter to GetSyncStatusAsync to allow
per-pipeline interval overrides instead of hardcoded defaults. This
enables tables like MisData to use longer sync intervals (e.g., 70 days)
while other tables use standard intervals.
Key changes:
- IDataUpdateRepository.GetSyncStatusAsync now accepts an optional
Dictionary<string, int> for custom intervals keyed by "TableName_UpdateType"
- GetExpectedInterval and IsOverdue made public static for testing and reuse
- Added GetDefaultInterval method for accessing default values
- Updated DataSyncHealthCheck to use new signature
- Added comprehensive unit tests for custom interval behavior
Update TableSyncOperation to pass UpdateTypes directly to the pipeline
builder using WithUpdateType() instead of mapping to SyncMode and calling
the deprecated WithMode() method. This enables proper schedule-based
configuration handling where Daily and Hourly have distinct behaviors.
- Remove SyncMode mapping logic from ExecuteSyncCoreAsync
- Call WithUpdateType(task.UpdateType) directly
- Update log message to reflect UpdateType instead of SyncMode
- Add TableSyncOperationTests verifying WithUpdateType is called correctly
- Add WithUpdateType(UpdateTypes) method to IEtlPipelineBuilder interface
- Mark existing WithMode(SyncMode) as [Obsolete("Use WithUpdateType instead")]
- Update PipelineBuilder to store UpdateTypes instead of SyncMode
- Add GetEffectiveScheduleConfig method to merge pipeline schedules with defaults
- Add BuildWithSchedules method for new Schedules-based config
- Update validation to support both old SyncModes and new Schedules formats
- Pass ScheduleDefaults from PipelinesRoot to PipelineBuilder
- For Mass mode: use massQuery, apply prePurge/reIndex from schedule config
- For Daily/Hourly: use regular query with date parameters
- Add 8 new tests for WithUpdateType functionality
Add CreateGiwConnectionAsync to IDbConnectionFactory and DbConnectionFactory
for connecting to the GIW Oracle database. This connection is needed for the
StatusCode data sync pipeline.
- Reuses existing CreateOracleConnectionAsync helper with "GIW" data source
- Follows same pattern as JDE, JDEStage, and CMS connections
- Includes 4 unit tests covering missing/empty/invalid connection strings
Add PipelineSchedules? Schedules parameter to PipelineConfig record between
SyncModes and Transformers for new schedule-based configuration. The existing
SyncModes property is now nullable for backward compatibility during the
transition to the new schedule system.
Add ScheduleDefaults? parameter to PipelinesRoot record and
EffectiveScheduleDefaults computed property that returns
defaults when null.
Updates all existing test usages to include the new parameter.
Joseph Doherty