feat: extract DevEtl to JdeScoping.DataSync.Dev project

- Create JdeScoping.DataSync.Dev for sandbox testing ETL code
- Create JdeScoping.DataSync.Dev.Tests for associated tests
- Move 22 source files and 8 test files
- Update namespaces from DevEtl to Dev
- Add both projects to solution
This commit is contained in:
Joseph Doherty
2026-01-06 10:18:09 -05:00
parent 7ad4e3ec1c
commit 81b07ce027
37 changed files with 2458 additions and 83 deletions
@@ -0,0 +1,162 @@
# Design: Extract DevEtl to JdeScoping.DataSync.Dev
**Date:** 2026-01-06
**Status:** Approved
## Purpose
Move dev/testing-specific ETL code from `JdeScoping.DataSync` into a separate `JdeScoping.DataSync.Dev` project. This code loads cached JSON/zstd files into SQL Server for local sandbox development and should not be part of the production DataSync assembly.
## Scope
### Files to Move (Source)
**22 files** from `src/JdeScoping.DataSync/DevEtl/``src/JdeScoping.DataSync.Dev/`
- `DevEtlRegistry.cs`
- `BranchDevEtl.cs`
- `FunctionCodeDevEtl.cs`
- `ItemDevEtl.cs`
- `JdeUserDevEtl.cs`
- `LotDevEtl.cs`
- `LotUsageCurrDevEtl.cs`
- `LotUsageHistDevEtl.cs`
- `MisDataDevEtl.cs`
- `OrgHierarchyDevEtl.cs`
- `ProfitCenterDevEtl.cs`
- `RouteMasterDevEtl.cs`
- `WorkCenterDevEtl.cs`
- `WorkOrderComponentCurrDevEtl.cs`
- `WorkOrderComponentHistDevEtl.cs`
- `WorkOrderCurrDevEtl.cs`
- `WorkOrderHistDevEtl.cs`
- `WorkOrderRoutingDevEtl.cs`
- `WorkOrderStepCurrDevEtl.cs`
- `WorkOrderStepHistDevEtl.cs`
- `WorkOrderTimeCurrDevEtl.cs`
- `WorkOrderTimeHistDevEtl.cs`
### Files to Move (Tests)
**8 files** from `tests/JdeScoping.DataSync.Tests/DevEtl/``tests/JdeScoping.DataSync.Dev.Tests/`
- `BranchDevEtlTests.cs`
- `FunctionCodeDevEtlTests.cs`
- `ItemDevEtlTests.cs`
- `JdeUserDevEtlTests.cs`
- `OrgHierarchyDevEtlTests.cs`
- `ProfitCenterDevEtlTests.cs`
- `RouteMasterDevEtlTests.cs`
- `WorkCenterDevEtlTests.cs`
## New Project Structure
```
NEW/
├── src/
│ └── JdeScoping.DataSync.Dev/
│ ├── JdeScoping.DataSync.Dev.csproj
│ ├── DevEtlRegistry.cs
│ ├── BranchDevEtl.cs
│ └── ... (20 more files)
└── tests/
└── JdeScoping.DataSync.Dev.Tests/
├── JdeScoping.DataSync.Dev.Tests.csproj
├── BranchDevEtlTests.cs
└── ... (7 more files)
```
## Dependencies
### JdeScoping.DataSync.Dev.csproj
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\JdeScoping.DataSync\JdeScoping.DataSync.csproj" />
</ItemGroup>
</Project>
```
Reuses ETL infrastructure from DataSync:
- `EtlPipeline`, `EtlPipelineBuilder`
- `JsonZstdFileSource`
- `DbBulkImportDestination`
- `JsonColumnSchema`
### JdeScoping.DataSync.Dev.Tests.csproj
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\JdeScoping.DataSync.Dev\JdeScoping.DataSync.Dev.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="..." />
<PackageReference Include="xunit" Version="..." />
<PackageReference Include="xunit.runner.visualstudio" Version="..." />
<PackageReference Include="Shouldly" Version="..." />
<PackageReference Include="NSubstitute" Version="..." />
</ItemGroup>
</Project>
```
## Namespace Changes
All source files:
```csharp
// Before
namespace JdeScoping.DataSync.DevEtl;
// After
namespace JdeScoping.DataSync.Dev;
```
All test files:
```csharp
// Before
using JdeScoping.DataSync.DevEtl;
// After
using JdeScoping.DataSync.Dev;
```
## Cleanup
After migration, delete:
- `src/JdeScoping.DataSync/DevEtl/` (entire folder)
- `tests/JdeScoping.DataSync.Tests/DevEtl/` (entire folder)
## Cache Files
No change to how cache files are located. The `DevEtlRegistry` continues to accept a `cacheDirectory` parameter at runtime.
## Implementation Steps
1. Create `src/JdeScoping.DataSync.Dev/JdeScoping.DataSync.Dev.csproj`
2. Move 22 source files from `DevEtl/` to new project
3. Update namespace in all source files
4. Create `tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj`
5. Move 8 test files to new project
6. Update using statements in test files
7. Add both projects to solution
8. Delete old `DevEtl/` folders
9. Build and run tests to verify
+389
View File
@@ -0,0 +1,389 @@
# DataSync.Dev Extraction Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Extract dev/testing ETL code from JdeScoping.DataSync into a separate JdeScoping.DataSync.Dev project with its own test project.
**Architecture:** Create two new projects (DataSync.Dev and DataSync.Dev.Tests) that depend on the existing DataSync project. Move all DevEtl files, update namespaces, update solution file, then delete original folders.
**Tech Stack:** .NET 10, xUnit, Shouldly, NSubstitute
---
## Task 1: Create JdeScoping.DataSync.Dev Project
**Files:**
- Create: `NEW/src/JdeScoping.DataSync.Dev/JdeScoping.DataSync.Dev.csproj`
**Step 1: Create project directory**
Run:
```bash
mkdir -p /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev
```
**Step 2: Create project file**
Create `NEW/src/JdeScoping.DataSync.Dev/JdeScoping.DataSync.Dev.csproj`:
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\JdeScoping.DataSync\JdeScoping.DataSync.csproj" />
</ItemGroup>
</Project>
```
**Step 3: Verify project builds**
Run:
```bash
dotnet build /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev/JdeScoping.DataSync.Dev.csproj
```
Expected: Build succeeded
---
## Task 2: Move Source Files to DataSync.Dev
**Files:**
- Move: `NEW/src/JdeScoping.DataSync/DevEtl/*.cs``NEW/src/JdeScoping.DataSync.Dev/`
**Step 1: Move all 22 source files**
Run:
```bash
mv /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync/DevEtl/*.cs /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev/
```
**Step 2: Remove empty DevEtl directory**
Run:
```bash
rmdir /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync/DevEtl
```
**Step 3: Verify files moved**
Run:
```bash
ls /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev/*.cs | wc -l
```
Expected: 22
---
## Task 3: Update Namespaces in Source Files
**Files:**
- Modify: All 22 `.cs` files in `NEW/src/JdeScoping.DataSync.Dev/`
**Step 1: Update namespace declarations**
Replace in all files:
```csharp
namespace JdeScoping.DataSync.DevEtl;
```
With:
```csharp
namespace JdeScoping.DataSync.Dev;
```
Run:
```bash
cd /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev && \
sed -i '' 's/namespace JdeScoping\.DataSync\.DevEtl;/namespace JdeScoping.DataSync.Dev;/g' *.cs
```
**Step 2: Verify namespace changes**
Run:
```bash
grep -l "JdeScoping.DataSync.DevEtl" /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev/*.cs
```
Expected: No output (no files should contain old namespace)
**Step 3: Verify new namespace exists**
Run:
```bash
grep -c "namespace JdeScoping.DataSync.Dev;" /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev/*.cs | head -5
```
Expected: Each file shows `:1`
**Step 4: Build to verify**
Run:
```bash
dotnet build /Users/dohertj2/Desktop/JdeScopingTool/NEW/src/JdeScoping.DataSync.Dev/JdeScoping.DataSync.Dev.csproj
```
Expected: Build succeeded
---
## Task 4: Create JdeScoping.DataSync.Dev.Tests Project
**Files:**
- Create: `NEW/tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj`
**Step 1: Create test project directory**
Run:
```bash
mkdir -p /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests
```
**Step 2: Create test project file**
Create `NEW/tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj`:
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
<PackageReference Include="xunit" Version="2.9.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.0.2">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="Shouldly" Version="4.3.0" />
<PackageReference Include="NSubstitute" Version="5.3.0" />
<PackageReference Include="coverlet.collector" Version="6.0.4">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.1" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="10.0.1" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="10.0.1" />
<PackageReference Include="Microsoft.Data.SqlClient" Version="6.1.3" />
<PackageReference Include="Dapper" Version="2.1.66" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\JdeScoping.DataAccess\JdeScoping.DataAccess.csproj" />
<ProjectReference Include="..\..\src\JdeScoping.DataSync.Dev\JdeScoping.DataSync.Dev.csproj" />
</ItemGroup>
<ItemGroup>
<Using Include="Xunit" />
</ItemGroup>
<ItemGroup>
<None Include="appsettings.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
</Project>
```
**Step 3: Copy appsettings.json from existing test project**
Run:
```bash
cp /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Tests/appsettings.json /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests/
```
**Step 4: Verify project builds**
Run:
```bash
dotnet build /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj
```
Expected: Build succeeded
---
## Task 5: Move Test Files to DataSync.Dev.Tests
**Files:**
- Move: `NEW/tests/JdeScoping.DataSync.Tests/DevEtl/*.cs``NEW/tests/JdeScoping.DataSync.Dev.Tests/`
**Step 1: Move all 8 test files**
Run:
```bash
mv /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Tests/DevEtl/*.cs /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests/
```
**Step 2: Remove empty DevEtl directory**
Run:
```bash
rmdir /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Tests/DevEtl
```
**Step 3: Verify files moved**
Run:
```bash
ls /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests/*.cs | wc -l
```
Expected: 8
---
## Task 6: Update Namespaces and Usings in Test Files
**Files:**
- Modify: All 8 `.cs` files in `NEW/tests/JdeScoping.DataSync.Dev.Tests/`
**Step 1: Update namespace declarations**
Replace in all files:
```csharp
namespace JdeScoping.DataSync.Tests.DevEtl;
```
With:
```csharp
namespace JdeScoping.DataSync.Dev.Tests;
```
Run:
```bash
cd /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests && \
sed -i '' 's/namespace JdeScoping\.DataSync\.Tests\.DevEtl;/namespace JdeScoping.DataSync.Dev.Tests;/g' *.cs
```
**Step 2: Update using statements**
Replace in all files:
```csharp
using JdeScoping.DataSync.DevEtl;
```
With:
```csharp
using JdeScoping.DataSync.Dev;
```
Run:
```bash
cd /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests && \
sed -i '' 's/using JdeScoping\.DataSync\.DevEtl;/using JdeScoping.DataSync.Dev;/g' *.cs
```
**Step 3: Verify changes**
Run:
```bash
grep -l "DevEtl" /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests/*.cs
```
Expected: No output (no files should contain "DevEtl")
**Step 4: Build to verify**
Run:
```bash
dotnet build /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj
```
Expected: Build succeeded
---
## Task 7: Update Solution File
**Files:**
- Modify: `NEW/JdeScoping.slnx`
**Step 1: Add new projects to solution**
Update `NEW/JdeScoping.slnx` to add two new Project entries:
In the `/src/` folder section, add:
```xml
<Project Path="src/JdeScoping.DataSync.Dev/JdeScoping.DataSync.Dev.csproj" />
```
In the `/tests/` folder section, add:
```xml
<Project Path="tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj" />
```
**Step 2: Verify solution builds**
Run:
```bash
dotnet build /Users/dohertj2/Desktop/JdeScopingTool/NEW/JdeScoping.slnx
```
Expected: Build succeeded
---
## Task 8: Run Tests to Verify
**Step 1: Run new test project**
Run:
```bash
dotnet test /Users/dohertj2/Desktop/JdeScopingTool/NEW/tests/JdeScoping.DataSync.Dev.Tests/JdeScoping.DataSync.Dev.Tests.csproj --verbosity normal
```
Expected: All tests pass (some may skip if cache files don't exist)
**Step 2: Run full solution tests**
Run:
```bash
dotnet test /Users/dohertj2/Desktop/JdeScopingTool/NEW/JdeScoping.slnx --verbosity minimal
```
Expected: All tests pass
---
## Task 9: Commit Changes
**Step 1: Stage all changes**
Run:
```bash
cd /Users/dohertj2/Desktop/JdeScopingTool && \
git add NEW/src/JdeScoping.DataSync.Dev/ && \
git add NEW/tests/JdeScoping.DataSync.Dev.Tests/ && \
git add NEW/JdeScoping.slnx
```
**Step 2: Commit**
Run:
```bash
git commit -m "feat: extract DevEtl to JdeScoping.DataSync.Dev project
- Create JdeScoping.DataSync.Dev for sandbox testing ETL code
- Create JdeScoping.DataSync.Dev.Tests for associated tests
- Move 22 source files and 8 test files
- Update namespaces from DevEtl to Dev
- Add both projects to solution"
```
---
## Summary
| Task | Description | Files Changed |
|------|-------------|---------------|
| 1 | Create DataSync.Dev project | 1 new csproj |
| 2 | Move source files | 22 files moved |
| 3 | Update source namespaces | 22 files modified |
| 4 | Create DataSync.Dev.Tests project | 1 new csproj + appsettings |
| 5 | Move test files | 8 files moved |
| 6 | Update test namespaces/usings | 8 files modified |
| 7 | Update solution file | 1 file modified |
| 8 | Run tests | Verification |
| 9 | Commit | Git |