Initial commit: JDE Scoping Tool migration project

Set up repository with legacy .NET Framework 4.8 source (OLD/), new .NET 10 Blazor solution (NEW/), OpenSpec specifications, documentation, and project configuration.
2026-01-02 07:43:29 -05:00
commit 26ff8d9b4f
1761 changed files with 596509 additions and 0 deletions
@@ -0,0 +1,68 @@
+# Implement Data Sync Service
+
+## Summary
+
+Implement the background data synchronization service as a .NET BackgroundService that maintains the local SQL Server cache by fetching data from JDE (Oracle) and CMS (Sybase) source systems on configurable schedules.
+
+## Scope
+
+### In Scope
+
+- `DataSyncService` inheriting from `BackgroundService` with proper lifecycle management
+- `IDataFetcher<T>` interface and fetcher implementations for each table type
+- `DataSyncOptions` and `DataSourceConfig` strongly-typed configuration classes
+- Schedule-based triggering (Mass/Daily/Hourly) with interval checking
+- Staging table management with MERGE operations for upserts
+- `DataUpdate` logging for audit trail and recovery
+- Health checks exposing sync status via ASP.NET Core health check framework
+- Telemetry via `System.Diagnostics.Metrics` and `ActivitySource`
+- `AddDataSync` extension method for DI registration
+
+### Out of Scope
+
+- Admin API for manual archive sync triggering (separate change)
+- Circuit breaker implementation for CMS (can be added later)
+- Periodic index maintenance (separate change)
+- Actual JDE/CMS database connectivity (will use mock fetchers initially)
+
+## Motivation
+
+The legacy `UpdateProcessor` runs as a Topshelf Windows service with reflection-based data fetchers and global temp tables. The new implementation uses modern .NET patterns:
+
+- `BackgroundService` for proper ASP.NET Core hosting integration
+- `IDataFetcher<T>` interfaces for type-safe, testable data retrieval
+- `Parallel.ForEachAsync` for cancellation-aware parallel execution
+- Local temp tables with unique suffixes for parallel isolation
+- `IOptions<T>` pattern for strongly-typed configuration
+
+## Acceptance Criteria
+
+1. `DataSyncService` starts with the host and respects `CancellationToken` for graceful shutdown
+2. Service checks schedules and queues sync tasks based on `LastDataUpdates` timestamps
+3. Sync operations execute in parallel with configurable `MaxDegreeOfParallelism`
+4. Each sync creates staging tables, bulk copies data, and executes MERGE operations
+5. All sync operations are logged to `DataUpdate` table with proper start/end/success tracking
+6. Interrupted syncs are marked as failed at startup via `CloseOpenUpdateEntries()`
+7. Health check reports sync status (Healthy/Degraded/Unhealthy) based on interval compliance
+8. Metrics emitted for operations started/completed/failed and duration histograms
+9. `openspec validate implement-data-sync --strict` passes
+
+## Dependencies
+
+- `migrate-database-schema` - DataUpdate table and related schema must exist
+- `data-access` spec - Repository patterns for database operations
+
+## Risks
+
+| Risk | Mitigation |
+|------|------------|
+| Complex parallel execution | Use `Parallel.ForEachAsync` with proper scoping; local temp tables with unique suffixes |
+| Schedule calculation edge cases | Comprehensive unit tests for schedule checking logic |
+| Memory pressure from large datasets | `IAsyncEnumerable<T>` streaming with batched bulk copy |
+| Staging table conflicts | Unique `_{OperationId}` suffix on all temp tables |
+
+## Related Specs
+
+- `data-sync` - Core specification for sync behavior and schedules
+- `domain-models` - Entity definitions for synced data
+- `database-schema` - Table structures and DataUpdate table