Joseph Doherty
26ff8d9b4f
Set up repository with legacy .NET Framework 4.8 source (OLD/), new .NET 10 Blazor solution (NEW/), OpenSpec specifications, documentation, and project configuration.
69 lines
3.3 KiB
Markdown
69 lines
3.3 KiB
Markdown
# Implement Data Sync Service
|
|
|
|
## Summary
|
|
|
|
Implement the background data synchronization service as a .NET BackgroundService that maintains the local SQL Server cache by fetching data from JDE (Oracle) and CMS (Sybase) source systems on configurable schedules.
|
|
|
|
## Scope
|
|
|
|
### In Scope
|
|
|
|
- `DataSyncService` inheriting from `BackgroundService` with proper lifecycle management
|
|
- `IDataFetcher<T>` interface and fetcher implementations for each table type
|
|
- `DataSyncOptions` and `DataSourceConfig` strongly-typed configuration classes
|
|
- Schedule-based triggering (Mass/Daily/Hourly) with interval checking
|
|
- Staging table management with MERGE operations for upserts
|
|
- `DataUpdate` logging for audit trail and recovery
|
|
- Health checks exposing sync status via ASP.NET Core health check framework
|
|
- Telemetry via `System.Diagnostics.Metrics` and `ActivitySource`
|
|
- `AddDataSync` extension method for DI registration
|
|
|
|
### Out of Scope
|
|
|
|
- Admin API for manual archive sync triggering (separate change)
|
|
- Circuit breaker implementation for CMS (can be added later)
|
|
- Periodic index maintenance (separate change)
|
|
- Actual JDE/CMS database connectivity (will use mock fetchers initially)
|
|
|
|
## Motivation
|
|
|
|
The legacy `UpdateProcessor` runs as a Topshelf Windows service with reflection-based data fetchers and global temp tables. The new implementation uses modern .NET patterns:
|
|
|
|
- `BackgroundService` for proper ASP.NET Core hosting integration
|
|
- `IDataFetcher<T>` interfaces for type-safe, testable data retrieval
|
|
- `Parallel.ForEachAsync` for cancellation-aware parallel execution
|
|
- Local temp tables with unique suffixes for parallel isolation
|
|
- `IOptions<T>` pattern for strongly-typed configuration
|
|
|
|
## Acceptance Criteria
|
|
|
|
1. `DataSyncService` starts with the host and respects `CancellationToken` for graceful shutdown
|
|
2. Service checks schedules and queues sync tasks based on `LastDataUpdates` timestamps
|
|
3. Sync operations execute in parallel with configurable `MaxDegreeOfParallelism`
|
|
4. Each sync creates staging tables, bulk copies data, and executes MERGE operations
|
|
5. All sync operations are logged to `DataUpdate` table with proper start/end/success tracking
|
|
6. Interrupted syncs are marked as failed at startup via `CloseOpenUpdateEntries()`
|
|
7. Health check reports sync status (Healthy/Degraded/Unhealthy) based on interval compliance
|
|
8. Metrics emitted for operations started/completed/failed and duration histograms
|
|
9. `openspec validate implement-data-sync --strict` passes
|
|
|
|
## Dependencies
|
|
|
|
- `migrate-database-schema` - DataUpdate table and related schema must exist
|
|
- `data-access` spec - Repository patterns for database operations
|
|
|
|
## Risks
|
|
|
|
| Risk | Mitigation |
|
|
|------|------------|
|
|
| Complex parallel execution | Use `Parallel.ForEachAsync` with proper scoping; local temp tables with unique suffixes |
|
|
| Schedule calculation edge cases | Comprehensive unit tests for schedule checking logic |
|
|
| Memory pressure from large datasets | `IAsyncEnumerable<T>` streaming with batched bulk copy |
|
|
| Staging table conflicts | Unique `_{OperationId}` suffix on all temp tables |
|
|
|
|
## Related Specs
|
|
|
|
- `data-sync` - Core specification for sync behavior and schedules
|
|
- `domain-models` - Entity definitions for synced data
|
|
- `database-schema` - Table structures and DataUpdate table
|