Files

T

Joseph Doherty 26ff8d9b4f Initial commit: JDE Scoping Tool migration project

Set up repository with legacy .NET Framework 4.8 source (OLD/),
new .NET 10 Blazor solution (NEW/), OpenSpec specifications,
documentation, and project configuration.

2026-01-02 07:43:29 -05:00

3.3 KiB

Raw Permalink Blame History

Implement Data Sync Service

Summary

Implement the background data synchronization service as a .NET BackgroundService that maintains the local SQL Server cache by fetching data from JDE (Oracle) and CMS (Sybase) source systems on configurable schedules.

Scope

In Scope

DataSyncService inheriting from BackgroundService with proper lifecycle management
IDataFetcher<T> interface and fetcher implementations for each table type
DataSyncOptions and DataSourceConfig strongly-typed configuration classes
Schedule-based triggering (Mass/Daily/Hourly) with interval checking
Staging table management with MERGE operations for upserts
DataUpdate logging for audit trail and recovery
Health checks exposing sync status via ASP.NET Core health check framework
Telemetry via System.Diagnostics.Metrics and ActivitySource
AddDataSync extension method for DI registration

Out of Scope

Admin API for manual archive sync triggering (separate change)
Circuit breaker implementation for CMS (can be added later)
Periodic index maintenance (separate change)
Actual JDE/CMS database connectivity (will use mock fetchers initially)

Motivation

The legacy UpdateProcessor runs as a Topshelf Windows service with reflection-based data fetchers and global temp tables. The new implementation uses modern .NET patterns:

BackgroundService for proper ASP.NET Core hosting integration
IDataFetcher<T> interfaces for type-safe, testable data retrieval
Parallel.ForEachAsync for cancellation-aware parallel execution
Local temp tables with unique suffixes for parallel isolation
IOptions<T> pattern for strongly-typed configuration

Acceptance Criteria

DataSyncService starts with the host and respects CancellationToken for graceful shutdown
Service checks schedules and queues sync tasks based on LastDataUpdates timestamps
Sync operations execute in parallel with configurable MaxDegreeOfParallelism
Each sync creates staging tables, bulk copies data, and executes MERGE operations
All sync operations are logged to DataUpdate table with proper start/end/success tracking
Interrupted syncs are marked as failed at startup via CloseOpenUpdateEntries()
Health check reports sync status (Healthy/Degraded/Unhealthy) based on interval compliance
Metrics emitted for operations started/completed/failed and duration histograms
openspec validate implement-data-sync --strict passes

Dependencies

migrate-database-schema - DataUpdate table and related schema must exist
data-access spec - Repository patterns for database operations

Risks

Risk	Mitigation
Complex parallel execution	Use `Parallel.ForEachAsync` with proper scoping; local temp tables with unique suffixes
Schedule calculation edge cases	Comprehensive unit tests for schedule checking logic
Memory pressure from large datasets	`IAsyncEnumerable<T>` streaming with batched bulk copy
Staging table conflicts	Unique `_{OperationId}` suffix on all temp tables

data-sync - Core specification for sync behavior and schedules
domain-models - Entity definitions for synced data
database-schema - Table structures and DataUpdate table

3.3 KiB Raw Permalink Blame History