jdescopingtool/openspec/changes/archive/2026-01-01-implement-data-sync/specs/data-sync/spec.md

Data Sync Specification - Implementation Additions

Purpose

This specification extends the base data-sync spec with additional implementation-focused requirements for the BackgroundService pattern and parallel fetch isolation.

ADDED Requirements

Requirement: Background service implementation pattern

The system SHALL implement the data synchronization service following .NET BackgroundService best practices for hosted service lifecycle management.

Inputs

• IServiceScopeFactory for creating scoped service instances
• IOptions<DataSyncOptions> for configuration access
• ILogger<DataSyncService> for structured logging
• CancellationToken from ExecuteAsync stoppingToken parameter

Outputs

• Continuously running background task that checks schedules and executes syncs
• Proper cleanup on shutdown with all resources disposed
• Logging scope context for all operations

Business Rules

• The service MUST implement BackgroundService.ExecuteAsync(CancellationToken)
• The main loop MUST use Task.Delay(checkInterval, stoppingToken) between cycles
• Each sync cycle MUST create a new IServiceScope via IServiceScopeFactory.CreateAsyncScope()
• All scoped services MUST be resolved from the current scope, not from root provider
• The scope MUST be disposed using await using pattern after each cycle
• Exception handling MUST catch and log errors without crashing the service
• OperationCanceledException MUST be caught and result in graceful loop exit when stoppingToken.IsCancellationRequested
• The service MUST NOT use static state or shared mutable collections

Scenario: Normal sync cycle execution

• WHEN the BackgroundService enters ExecuteAsync
• THEN the service SHALL call CloseOpenUpdateEntriesAsync to recover from prior crashes
• THEN the service SHALL enter a while loop checking !stoppingToken.IsCancellationRequested
• THEN each iteration SHALL create a new IServiceScope
• THEN the ISyncOrchestrator SHALL be resolved from the scope
• THEN ExecutePendingSyncsAsync SHALL be called with the stoppingToken
• THEN the scope SHALL be disposed after the call completes
• THEN Task.Delay SHALL pause before the next iteration

Scenario: Exception during sync cycle

• WHEN an exception occurs during sync execution (not OperationCanceledException)
• THEN the exception SHALL be caught and logged with LogError
• THEN the service SHALL continue to the next iteration
• THEN the current scope SHALL still be disposed properly
• THEN the service SHALL NOT crash or stop unexpectedly

Scenario: Graceful shutdown request

• WHEN the host signals shutdown by canceling the stoppingToken
• THEN any running Task.Delay SHALL throw OperationCanceledException
• THEN the while loop SHALL exit on the IsCancellationRequested check
• THEN the ExecuteAsync method SHALL complete normally
• THEN any in-progress sync operations SHALL receive the cancellation and complete or cancel

Requirement: Parallel fetch isolation with scoped resources

The system SHALL ensure complete isolation between parallel sync operations using scoped resources and unique identifiers.

Inputs

• List of DataUpdateTask objects to execute in parallel
• MaxDegreeOfParallelism configuration value
• CancellationToken for coordinated cancellation

Outputs

• Concurrent execution of sync operations with no resource conflicts
• Unique staging tables per operation that do not collide
• Independent database connections per operation

Business Rules

• Parallel.ForEachAsync MUST be used with ParallelOptions.CancellationToken set
• Each parallel task MUST create its own IServiceScope inside the parallel delegate
• Database connections MUST NOT be shared across parallel operations
• Staging table names MUST include a unique OperationId suffix (GUID or sequential ID)
• Format: #Staging{TableName}_{OperationId} and #{TableName}_{OperationId}
• Each parallel operation MUST resolve its own instances of all scoped services
• No ConcurrentDictionary, shared counters, or other shared mutable state SHALL exist between operations
• Total record counts SHALL be accumulated via return values, not shared state

Scenario: Parallel sync with isolated scopes

• WHEN multiple DataUpdateTasks are executed via Parallel.ForEachAsync
• THEN each task SHALL execute the async delegate independently
• THEN each delegate SHALL create a new IServiceScope using CreateAsyncScope
• THEN ITableSyncOperation SHALL be resolved from each scope independently
• THEN each operation SHALL use its own database connection from the scope
• THEN staging tables SHALL use unique OperationId suffixes preventing name collisions
• THEN completion of one operation SHALL NOT affect the execution of others

Scenario: Parallel cancellation propagation

• WHEN cancellation is requested during Parallel.ForEachAsync execution
• THEN the CancellationToken SHALL propagate to all running parallel operations
• THEN Parallel.ForEachAsync SHALL stop starting new operations
• THEN running operations SHALL receive the token in their async methods
• THEN each operation SHALL check the token and exit gracefully
• THEN incomplete operations SHALL mark their DataUpdate records as failed

Scenario: Staging table uniqueness verification

• WHEN two sync operations for the same table run in parallel
• THEN each operation SHALL generate a unique OperationId as GUID
• THEN operation A SHALL create staging table with GuidA suffix
• THEN operation B SHALL create staging table with GuidB suffix
• THEN no SQL errors SHALL occur from table name conflicts
• THEN each operation cleanup SHALL only drop its own staging tables

Requirement: Structured logging context

The system SHALL use ILogger.BeginScope to attach contextual information to all log entries during sync operations.

Inputs

• ILogger<T> injected into sync operation classes
• TableName, UpdateType, OperationId values from current operation

Outputs

• All log entries within the scope contain the contextual properties
• Log aggregation systems can filter and group by table, type, or operation

Business Rules

• Each sync operation MUST call _logger.BeginScope(...) at the start
• The scope MUST include at minimum: TableName, UpdateType, OperationId
• The scope MUST be disposed using using statement when operation completes
• Nested scopes for batches SHALL preserve parent scope properties
• LogInformation, LogWarning, LogError calls within the scope SHALL include the context automatically

Scenario: Log scope creation and usage

• WHEN a TableSyncOperation begins execution
• THEN the operation SHALL create a logging scope with TableName, UpdateType, OperationId
• THEN all log calls within ExecuteAsync SHALL include these properties
• THEN when the operation completes the scope SHALL be disposed
• THEN subsequent operations SHALL have their own independent scopes

Migration Notes

Legacy Pattern	New Pattern	Rationale
Static UpdateProcessor methods	Scoped services resolved per operation	Proper DI lifecycle, testability
Shared instance state	Return values and scoped state only	Thread safety in parallel scenarios
Console.WriteLine logging	ILogger<T> with BeginScope	Structured logging, context propagation
Global temp tables ##table	Local temp tables #table_{id}	Session-scoped isolation for parallelism