Document the MisData_Curr/Hist table split in architecture docs, add
post-sync archival scripts to move BackLevel records to history table,
and generate comprehensive unit test report (856 tests, 100% pass rate).
- WorkProcessorReport.md: Analysis of legacy work processor from OLD solution
- Design document with clean architecture and component specifications
- Implementation plan with 15 TDD tasks
Aligns pipelines.json with legacy DataSyncReport.md:
- Three schedules (Mass/Daily/Hourly) instead of two sync modes
- 8 missing pipelines to add
- GIW connection for StatusCode
- Per-pipeline schedule configuration
- Move file size read after streams are disposed to get accurate compressed size
- Clean up definition files to use working example queries
- Add .gitignore for output directory
protobuf-net-data is purpose-built for IDataReader serialization and
returns IDataReader directly from Deserialize(), eliminating the need
for custom streaming reader implementations.
Key changes from Codex feedback:
- THROW cannot be used in UDFs (SQL Server restriction)
- Multi-statement TVFs have poor performance (no stats)
- Changed to inline TVFs for extraction (optimal performance)
- Added validation stored procedure for strict error handling
- Use TRY_CONVERT for safe type conversion
- Use OPENJSON...WITH for complex object extraction
- Replace IsApiError with IsError
- Replace ApiError. with Error.
- Replace ValidationError.Errors with ValidationError.FieldErrors
These match the actual ApiResult<T> implementation. The tests
themselves were already correct - subagents adapted during implementation.
- Create JdeScoping.DataSync.Dev for sandbox testing ETL code
- Create JdeScoping.DataSync.Dev.Tests for associated tests
- Move 22 source files and 8 test files
- Update namespaces from DevEtl to Dev
- Add both projects to solution
8-task plan to create comprehensive ETL pipeline documentation:
- Overview, Sources, Transformers, Destinations, Configuration, Troubleshooting
- Update ComponentMap with ETL source paths
- Final verification of links and commits
12-task TDD implementation plan covering:
- MapOrdinal interface and binary method overrides
- ColumnDropTransformer and JdeDateTransformer ordinal mapping
- JDE date validation with 1900-01-01 sentinel
- Column rename collision detection
- SqlScriptRunner parameters support
- CommonScripts with ParseTableName and QUOTENAME
- Destination command timeouts
- Column mapping with destination schema intersection
- EtlPipelineBuilder.WithCommandTimeout validation
- Full test suite verification
- Codex MCP final review
Move configuration options from Core/DataAccess/DataSync/ExcelIO to
dedicated Options folders within each project for better organization.
Update all references and tests accordingly.
Set up repository with legacy .NET Framework 4.8 source (OLD/),
new .NET 10 Blazor solution (NEW/), OpenSpec specifications,
documentation, and project configuration.
Joseph Doherty