Joseph Doherty
81b07ce027
- Create JdeScoping.DataSync.Dev for sandbox testing ETL code - Create JdeScoping.DataSync.Dev.Tests for associated tests - Move 22 source files and 8 test files - Update namespaces from DevEtl to Dev - Add both projects to solution
163 lines
4.3 KiB
Markdown
163 lines
4.3 KiB
Markdown
# Design: Extract DevEtl to JdeScoping.DataSync.Dev
|
|
|
|
**Date:** 2026-01-06
|
|
**Status:** Approved
|
|
|
|
## Purpose
|
|
|
|
Move dev/testing-specific ETL code from `JdeScoping.DataSync` into a separate `JdeScoping.DataSync.Dev` project. This code loads cached JSON/zstd files into SQL Server for local sandbox development and should not be part of the production DataSync assembly.
|
|
|
|
## Scope
|
|
|
|
### Files to Move (Source)
|
|
|
|
**22 files** from `src/JdeScoping.DataSync/DevEtl/` → `src/JdeScoping.DataSync.Dev/`
|
|
|
|
- `DevEtlRegistry.cs`
|
|
- `BranchDevEtl.cs`
|
|
- `FunctionCodeDevEtl.cs`
|
|
- `ItemDevEtl.cs`
|
|
- `JdeUserDevEtl.cs`
|
|
- `LotDevEtl.cs`
|
|
- `LotUsageCurrDevEtl.cs`
|
|
- `LotUsageHistDevEtl.cs`
|
|
- `MisDataDevEtl.cs`
|
|
- `OrgHierarchyDevEtl.cs`
|
|
- `ProfitCenterDevEtl.cs`
|
|
- `RouteMasterDevEtl.cs`
|
|
- `WorkCenterDevEtl.cs`
|
|
- `WorkOrderComponentCurrDevEtl.cs`
|
|
- `WorkOrderComponentHistDevEtl.cs`
|
|
- `WorkOrderCurrDevEtl.cs`
|
|
- `WorkOrderHistDevEtl.cs`
|
|
- `WorkOrderRoutingDevEtl.cs`
|
|
- `WorkOrderStepCurrDevEtl.cs`
|
|
- `WorkOrderStepHistDevEtl.cs`
|
|
- `WorkOrderTimeCurrDevEtl.cs`
|
|
- `WorkOrderTimeHistDevEtl.cs`
|
|
|
|
### Files to Move (Tests)
|
|
|
|
**8 files** from `tests/JdeScoping.DataSync.Tests/DevEtl/` → `tests/JdeScoping.DataSync.Dev.Tests/`
|
|
|
|
- `BranchDevEtlTests.cs`
|
|
- `FunctionCodeDevEtlTests.cs`
|
|
- `ItemDevEtlTests.cs`
|
|
- `JdeUserDevEtlTests.cs`
|
|
- `OrgHierarchyDevEtlTests.cs`
|
|
- `ProfitCenterDevEtlTests.cs`
|
|
- `RouteMasterDevEtlTests.cs`
|
|
- `WorkCenterDevEtlTests.cs`
|
|
|
|
## New Project Structure
|
|
|
|
```
|
|
NEW/
|
|
├── src/
|
|
│ └── JdeScoping.DataSync.Dev/
|
|
│ ├── JdeScoping.DataSync.Dev.csproj
|
|
│ ├── DevEtlRegistry.cs
|
|
│ ├── BranchDevEtl.cs
|
|
│ └── ... (20 more files)
|
|
└── tests/
|
|
└── JdeScoping.DataSync.Dev.Tests/
|
|
├── JdeScoping.DataSync.Dev.Tests.csproj
|
|
├── BranchDevEtlTests.cs
|
|
└── ... (7 more files)
|
|
```
|
|
|
|
## Dependencies
|
|
|
|
### JdeScoping.DataSync.Dev.csproj
|
|
|
|
```xml
|
|
<Project Sdk="Microsoft.NET.Sdk">
|
|
|
|
<PropertyGroup>
|
|
<TargetFramework>net10.0</TargetFramework>
|
|
<ImplicitUsings>enable</ImplicitUsings>
|
|
<Nullable>enable</Nullable>
|
|
</PropertyGroup>
|
|
|
|
<ItemGroup>
|
|
<ProjectReference Include="..\JdeScoping.DataSync\JdeScoping.DataSync.csproj" />
|
|
</ItemGroup>
|
|
|
|
</Project>
|
|
```
|
|
|
|
Reuses ETL infrastructure from DataSync:
|
|
- `EtlPipeline`, `EtlPipelineBuilder`
|
|
- `JsonZstdFileSource`
|
|
- `DbBulkImportDestination`
|
|
- `JsonColumnSchema`
|
|
|
|
### JdeScoping.DataSync.Dev.Tests.csproj
|
|
|
|
```xml
|
|
<Project Sdk="Microsoft.NET.Sdk">
|
|
|
|
<PropertyGroup>
|
|
<TargetFramework>net10.0</TargetFramework>
|
|
<ImplicitUsings>enable</ImplicitUsings>
|
|
<Nullable>enable</Nullable>
|
|
<IsPackable>false</IsPackable>
|
|
</PropertyGroup>
|
|
|
|
<ItemGroup>
|
|
<ProjectReference Include="..\..\src\JdeScoping.DataSync.Dev\JdeScoping.DataSync.Dev.csproj" />
|
|
</ItemGroup>
|
|
|
|
<ItemGroup>
|
|
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="..." />
|
|
<PackageReference Include="xunit" Version="..." />
|
|
<PackageReference Include="xunit.runner.visualstudio" Version="..." />
|
|
<PackageReference Include="Shouldly" Version="..." />
|
|
<PackageReference Include="NSubstitute" Version="..." />
|
|
</ItemGroup>
|
|
|
|
</Project>
|
|
```
|
|
|
|
## Namespace Changes
|
|
|
|
All source files:
|
|
```csharp
|
|
// Before
|
|
namespace JdeScoping.DataSync.DevEtl;
|
|
|
|
// After
|
|
namespace JdeScoping.DataSync.Dev;
|
|
```
|
|
|
|
All test files:
|
|
```csharp
|
|
// Before
|
|
using JdeScoping.DataSync.DevEtl;
|
|
|
|
// After
|
|
using JdeScoping.DataSync.Dev;
|
|
```
|
|
|
|
## Cleanup
|
|
|
|
After migration, delete:
|
|
- `src/JdeScoping.DataSync/DevEtl/` (entire folder)
|
|
- `tests/JdeScoping.DataSync.Tests/DevEtl/` (entire folder)
|
|
|
|
## Cache Files
|
|
|
|
No change to how cache files are located. The `DevEtlRegistry` continues to accept a `cacheDirectory` parameter at runtime.
|
|
|
|
## Implementation Steps
|
|
|
|
1. Create `src/JdeScoping.DataSync.Dev/JdeScoping.DataSync.Dev.csproj`
|
|
2. Move 22 source files from `DevEtl/` to new project
|
|
3. Update namespace in all source files
|
|
4. Create `tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj`
|
|
5. Move 8 test files to new project
|
|
6. Update using statements in test files
|
|
7. Add both projects to solution
|
|
8. Delete old `DevEtl/` folders
|
|
9. Build and run tests to verify
|