dohertj2/jdescopingtool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e5fe2f06e94c9880e152ddec2f75393716e44821
jdescopingtool/DATA_SYNC/DataSyncReport.md
T
Joseph Doherty 1618b6664d refactor: remove unused CMS/JDE repositories and data sources 
Remove legacy JDE and CMS direct-access code that is no longer used:
- Delete ICmsDataSource, IJdeDataSource interfaces
- Delete ISearchProcessor, IUpdateProcessor interfaces
- Delete IJdeRepository and ICmsRepository (all partials)
- Delete JdeRepository and CmsRepository implementations
- Delete JdeQueries and CmsQueries
- Delete JdeFileDataSource, JdeOracleDataSource
- Delete CmsFileDataSource, CmsOracleDataSource
- Remove unused methods from LotFinderRepository interfaces
- Delete associated unit tests (CmsRepositoryTests, JdeRepositoryTests)

All data sync now uses ETL pipelines via DataSync project.
2026-01-07 05:04:49 -05:00

588 lines
19 KiB
Markdown
Raw Blame History

# Data Sync Configuration Report
This document describes all data synchronization imports from the legacy JDE Scoping Tool (LotFinder) application.
## Executive Summary
| Source System | Database Type | Entity Count | Connection |
|---------------|---------------|--------------|------------|
| JDE | Oracle (Oracle.ManagedDataAccess) | 21 (16 active + 5 archive) | Config.JDECS |
| CMS | Oracle (DDTek.Oracle) | 1 | Config.CMSCS |
*Note: StatusCode sync uses GIW connection (Config.GIWCS via DDTek.Oracle) instead of standard JDE connection.
**Total Syncs:** 22
- **Active Syncs:** 17 (scheduled to run)
- **Archive Syncs:** 5 (configured but ALL DISABLED)
**Cache Files:** All syncs have corresponding cache files in `CACHED_DB_FILES/` (except StatusCode)
---
## Schedule Legend
| Schedule | Interval (min) | Frequency | PrepurgeData | ReIndexData | Sync Type |
|----------|----------------|-----------|--------------|-------------|-----------|
| Mass | 10080 | Weekly | Yes | Yes | Full Reload |
| Daily | 1440 | Daily | No | No | Incremental Merge |
| Hourly | 60 | Hourly | No | No | Incremental Merge |
- **PrepurgeData=true**: Table is truncated before import (full reload)
- **PrepurgeData=false**: Records are merged/upserted (incremental)
- **Filtered Query**: Uses date/time parameters to fetch only changed records
---
## Master Sync Table
### Active Syncs (17)
| # | Import Name | Source | Dest Table | Mass | Daily | Hourly | Filter | Cache File | Notes |
|---|-------------|--------|------------|------|-------|--------|--------|------------|-------|
| 1 | WorkOrder | JDE | WorkOrder_Curr | Yes | Yes | Yes | Yes | `workorder_curr.pb.zstd` | |
| 2 | LotUsage | JDE | LotUsage_Curr | Yes | Yes | Yes | Yes | `lotusage_curr.pb.zstd` | |
| 3 | Item | JDE | Item | Yes | Yes | Yes | Yes | `item.pb.zstd` | |
| 4 | Lot | JDE | Lot | Yes | Yes | Yes | Yes | `lot.pb.zstd` | |
| 5 | WorkOrderTime | JDE | WorkOrderTime_Curr | Yes | Yes | Yes | Yes | `workordertime_curr.pb.zstd` | |
| 6 | WorkOrderComponent | JDE | WorkOrderComponent_Curr | Yes | Yes | Yes | Yes | `workordercomponent_curr.pb.zstd` | |
| 7 | WorkOrderStep | JDE | WorkOrderStep_Curr | Yes | Yes | Yes | Yes | `workorderstep_curr.pb.zstd` | |
| 8 | WorkOrderRouting | JDE | WorkOrderRouting | Yes | Yes | Yes | Yes | `workorderrouting.pb.zstd` | |
| 9 | Branch | JDE | Branch | Yes | Yes | Yes | Yes | `branch.pb.zstd` | typeCode='BP' |
| 10 | ProfitCenter | JDE | ProfitCenter | Yes | Yes | Yes | Yes | `profitcenter.pb.zstd` | typeCode='I3' |
| 11 | WorkCenter | JDE | WorkCenter | Yes | Yes | Yes | Yes | `workcenter.pb.zstd` | typeCode='WC' |
| 12 | StatusCode | JDE | StatusCode | Yes | Yes | Yes | Yes | *(none)* | GIW connection |
| 13 | JdeUser | JDE | JdeUser | Yes | Yes | Yes | No | `jdeuser.pb.zstd` | Same query both |
| 14 | OrgHierarchy | JDE | OrgHierarchy | Yes | Yes | Yes | Yes | `orghierarchy.pb.zstd` | |
| 15 | RouteMaster | JDE | RouteMaster | Yes | Yes | Yes | Yes | `routemaster.pb.zstd` | |
| 16 | FunctionCode | JDE | FunctionCode | Yes | Yes | Yes | No | `functioncode.pb.zstd` | Always full reload |
| 17 | MisData | CMS | MisData | Yes | Yes | No | Yes | `misdata.pb.zstd` | Hourly disabled |
### Archive Syncs (5) - ALL DISABLED
| # | Import Name | Source | Dest Table | Mass | Daily | Hourly | Filter | Cache File | Notes |
|---|-------------|--------|------------|------|-------|--------|--------|------------|-------|
| 18 | WorkOrder_Archive | JDE | WorkOrder_Hist | No | No | No | No | `workorder_hist.pb.zstd` | DISABLED |
| 19 | LotUsage_Archive | JDE | LotUsage_Hist | No | No | No | No | `lotusage_hist.pb.zstd` | DISABLED |
| 20 | WorkOrderTime_Archive | JDE | WorkOrderTime_Hist | No | No | No | No | `workordertime_hist.pb.zstd` | DISABLED |
| 21 | WorkOrderComponent_Archive | JDE | WorkOrderComponent_Hist | No | No | No | No | `workordercomponent_hist.pb.zstd` | DISABLED |
| 22 | WorkOrderStep_Archive | JDE | WorkOrderStep_Hist | No | No | No | No | `workorderstep_hist.pb.zstd` | DISABLED |
**Cache File Location:** `CACHED_DB_FILES/`
---
## Detailed Entity Configurations
### 1. WorkOrder
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDER |
| Destination Table | WorkOrder_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrders |
| Post Processing | None |
| JDE Table | JDESTAGE.F4801_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_WORKORDERS.sql`
- Filtered: `JDE/SQL_GET_WORKORDERS_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDERS.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 2. LotUsage
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | LOTUSAGE |
| Destination Table | LotUsage_Curr |
| Data Fetch Function | Commons.Process.JDE.GetLotUsages |
| Post Processing | None |
| JDE Table | JDESTAGE.F4111_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_LOT_USAGES.sql`
- Filtered: `JDE/SQL_GET_LOT_USAGES_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_LOT_USAGES.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 3. Item
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | ITEM |
| Destination Table | Item |
| Data Fetch Function | Commons.Process.JDE.GetItems |
| Post Processing | None |
| JDE Table | JDESTAGE.F4101_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_ITEMS.sql`
- Filtered: `JDE/SQL_GET_ITEMS_FILTERED.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 4. Lot
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | LOT |
| Destination Table | Lot |
| Data Fetch Function | Commons.Process.JDE.GetLots |
| Post Processing | None |
| JDE Table | JDESTAGE.F4108_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_LOTS.sql`
- Filtered: `JDE/SQL_GET_LOTS_FILTERED.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 5. WorkOrderTime
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERTIME |
| Destination Table | WorkOrderTime_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderTimes |
| Post Processing | None |
| JDE Table | JDESTAGE.F31122_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_TIMES.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_TIMES_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDER_TIMES.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 6. WorkOrderComponent
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERCOMPONENT |
| Destination Table | WorkOrderComponent_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderComponents |
| Post Processing | None |
| JDE Table | JDESTAGE.F3111_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_COMPONENTS.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_COMPONENTS_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDER_COMPONENTS.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 7. WorkOrderStep
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERSTEP |
| Destination Table | WorkOrderStep_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderSteps |
| Post Processing | None |
| JDE Table | JDESTAGE.F3112_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_STEP.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_STEP_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDER_STEP.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 8. WorkOrderRouting
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERROUTING |
| Destination Table | WorkOrderRouting |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderRoutings |
| Post Processing | None |
| JDE Table | JDESTAGE.F3112Z1_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_ROUTING.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_ROUTING_FILTERED.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
**Special Processing:** Filters out records with invalid dates (year < 1900 or > 2500)
---
### 9. Branch
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | BRANCH |
| Destination Table | Branch |
| Data Fetch Function | Commons.Process.JDE.GetBranches |
| Post Processing | None |
| JDE Table | JDESTAGE.F0006_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_BUSINESS_UNITS.sql`
- Filtered: `JDE/SQL_GET_BUSINESS_UNITS_FILTERED.sql`
**Filter Parameters:** `typeCode='BP'`, `dateUpdated`, `timeUpdated`
---
### 10. ProfitCenter
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | PROFITCENTER |
| Destination Table | ProfitCenter |
| Data Fetch Function | Commons.Process.JDE.GetProfitCenters |
| Post Processing | None |
| JDE Table | JDESTAGE.F0006_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_BUSINESS_UNITS.sql`
- Filtered: `JDE/SQL_GET_BUSINESS_UNITS_FILTERED.sql`
**Filter Parameters:** `typeCode='I3'`, `dateUpdated`, `timeUpdated`
---
### 11. WorkCenter
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKCENTER |
| Destination Table | WorkCenter |
| Data Fetch Function | Commons.Process.JDE.GetWorkCenters |
| Post Processing | None |
| JDE Table | JDESTAGE.F0006_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_BUSINESS_UNITS.sql`
- Filtered: `JDE/SQL_GET_BUSINESS_UNITS_FILTERED.sql`
**Filter Parameters:** `typeCode='WC'`, `dateUpdated`, `timeUpdated`
---
### 12. StatusCode
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | STATUSCODE |
| Destination Table | StatusCode |
| Data Fetch Function | Commons.Process.JDE.GetStatusCodes |
| Post Processing | None |
| JDE Table | JDESTAGE.F0005_VIEW |
| **Connection** | **GIW (not JDE)** |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_STATUS_CODES.sql`
- Filtered: `JDE/SQL_GET_STATUS_CODES_FILTERED.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated`
**Note:** This sync uses the GIW connection (Config.GIWCS) via DDTek.Oracle, not the standard JDE connection.
---
### 13. JdeUser
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | USER |
| Destination Table | JdeUser |
| Data Fetch Function | Commons.Process.JDE.GetUsers |
| Post Processing | None |
| JDE Tables | JDESTAGE.F0101_VIEW, JDESTAGE.F0092_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_USERS.sql`
**Note:** No filtered query variant exists. The same query is used for both full and incremental syncs (filter parameters are passed but not used in query).
---
### 14. OrgHierarchy
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | ORGHIERARCHY |
| Destination Table | OrgHierarchy |
| Data Fetch Function | Commons.Process.JDE.GetOrgHierarchy |
| Post Processing | None |
| JDE Table | JDESTAGE.F30006_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_ORG_HIERARCHY.sql`
- Filtered: `JDE/SQL_GET_ORG_HIERARCHY_FILTERED.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 15. RouteMaster
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | ROUTEMASTER |
| Destination Table | RouteMaster |
| Data Fetch Function | Commons.Process.JDE.GetRouteMasters |
| Post Processing | None |
| JDE Table | JDESTAGE.F3003_VIEW |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |
**Query Files:**
- Full: `JDE/SQL_GET_ROUTE_MASTER.sql`
- Filtered: `JDE/SQL_GET_ROUTE_MASTER_FILTERED.sql`
**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)
---
### 16. FunctionCode
| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | FUNCTIONCODE |
| Destination Table | FunctionCode |
| Data Fetch Function | Commons.Process.JDE.GetFunctionCodes |
| Post Processing | None |
| JDE Table | PRODDTA.F00192 (direct table, not view) |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | Yes | Yes |
| Hourly | Yes | 60 | Yes | Yes |
**Query Files:**
- Full: `JDE/SQL_GET_FUNCTION_CODES.sql`
**Note:** No filtered query variant. Always performs full reload (PrepurgeData=true for all schedules). Uses aggregation with LISTAGG to combine multi-row descriptions.
---
### 17. MisData
| Property | Value |
|----------|-------|
| Source System | CMS |
| Source Data | MISDATA |
| Destination Table | MisData |
| Data Fetch Function | Commons.Process.CMS.GetMisData |
| Post Processing | Commons.Process.LotFinderDB.PostProcessMisData |
| CMS Schema | INFODBA |
**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 100800 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | **No** | 60 | No | No |
**Query Files:**
- Full: `CMS/SQL_GET_MIS_DATA.sql`
- Filtered: `CMS/SQL_GET_MIS_DATA_FILTERED.sql`
**Filter Parameters:** `lastUpdateDT` (standard DateTime)
**Note:**
- Mass interval is 100800 minutes (~10 weeks), much longer than other syncs
- Hourly sync is disabled
- Has post-processing action for additional data transformation
- Query timeout is extended (1200*50 = 60000 seconds)
- ReleaseDate is converted to local time after fetch
---
## Archive Query Pattern
Archive syncs (#18-22) fetch historical data using UNION ALL from both current and archived schemas:
```sql
SELECT ... FROM QADTA.F{table}
UNION ALL
SELECT ... FROM ARCDTAQA.F{table}
```
**Query Files:** Located in `JDE_ARCHIVE/` folder.
**Note:** Archive syncs have `IsEnabled=true` in dsconfig but all schedule types have `Enabled=false`. They are configured for manual/on-demand execution only.
---
## Source Database Reference
### JDE Oracle Connection (Config.JDECS)
- **Driver:** Oracle.ManagedDataAccess.Client
- **Schemas Used:**
- `JDESTAGE` - Views for current production data
- `PRODDTA` - Direct table access (FunctionCode only)
- `QADTA` - Current data for archive queries
- `ARCDTAQA` - Archived historical data
### GIW Oracle Connection (Config.GIWCS)
- **Driver:** DDTek.Oracle
- **Used By:** StatusCode sync only
- **Schema:** JDESTAGE
### CMS Oracle Connection (Config.CMSCS)
- **Driver:** DDTek.Oracle
- **Schema:** INFODBA
- **Tables:** Complex 11-table join for MIS data
### JDE Date/Time Format
JDE uses a special date/time format conversion:
- Dates are converted via `ToJDEDate()` helper
- Times are converted via `ToJDETime()` helper
- Filter conditions use: `(date > :dateUpdated) OR (date = :dateUpdated AND time >= :timeUpdated)`
---
## Source Files Reference
| File Type | Location |
|-----------|----------|
| dsconfig JSON files | OLD/WorkerService/dsconfig/*.json |
| JDE Source Queries | OLD/WorkerService/bin/UPDATER/JdeQueries/*.sql |
| JDE Archive Queries | OLD/WorkerService/bin/UPDATER/JdeArchivalQueries/*.sql |
| CMS Source Queries | OLD/DataModel/CmsQueries/*.sql |
| C# Data Fetch Code | OLD/DataModel/Process/JDE.*.cs, CMS.*.cs |
| Configuration | OLD/DataModel/Config.cs |
| Query Repository | OLD/DataModel/Process/QueryRepository.cs |
Reference in New Issue View Git Blame Copy Permalink