# Data Sync Configuration Report

This document describes all data synchronization imports from the legacy JDE Scoping Tool (LotFinder) application.

## Executive Summary

| Source System | Database Type | Entity Count | Connection |
|---------------|---------------|--------------|------------|
| JDE | Oracle (Oracle.ManagedDataAccess) | 21 (16 active + 5 archive) | Config.JDECS |
| CMS | Oracle (DDTek.Oracle) | 1 | Config.CMSCS |

*Note: StatusCode sync uses GIW connection (Config.GIWCS via DDTek.Oracle) instead of standard JDE connection.

**Total Syncs:** 22
- **Active Syncs:** 17 (scheduled to run)
- **Archive Syncs:** 5 (configured but ALL DISABLED)

**Cache Files:** All syncs have corresponding cache files in `CACHED_DB_FILES/` (except StatusCode)

---

## Schedule Legend

| Schedule | Interval (min) | Frequency | PrepurgeData | ReIndexData | Sync Type |
|----------|----------------|-----------|--------------|-------------|-----------|
| Mass | 10080 | Weekly | Yes | Yes | Full Reload |
| Daily | 1440 | Daily | No | No | Incremental Merge |
| Hourly | 60 | Hourly | No | No | Incremental Merge |

- **PrepurgeData=true**: Table is truncated before import (full reload)
- **PrepurgeData=false**: Records are merged/upserted (incremental)
- **Filtered Query**: Uses date/time parameters to fetch only changed records

---

## Master Sync Table

### Active Syncs (17)

| # | Import Name | Source | Dest Table | Mass | Daily | Hourly | Filter | Cache File | Notes |
|---|-------------|--------|------------|------|-------|--------|--------|------------|-------|
| 1 | WorkOrder | JDE | WorkOrder_Curr | Yes | Yes | Yes | Yes | `workorder_curr.pb.zstd` | |
| 2 | LotUsage | JDE | LotUsage_Curr | Yes | Yes | Yes | Yes | `lotusage_curr.pb.zstd` | |
| 3 | Item | JDE | Item | Yes | Yes | Yes | Yes | `item.pb.zstd` | |
| 4 | Lot | JDE | Lot | Yes | Yes | Yes | Yes | `lot.pb.zstd` | |
| 5 | WorkOrderTime | JDE | WorkOrderTime_Curr | Yes | Yes | Yes | Yes | `workordertime_curr.pb.zstd` | |
| 6 | WorkOrderComponent | JDE | WorkOrderComponent_Curr | Yes | Yes | Yes | Yes | `workordercomponent_curr.pb.zstd` | |
| 7 | WorkOrderStep | JDE | WorkOrderStep_Curr | Yes | Yes | Yes | Yes | `workorderstep_curr.pb.zstd` | |
| 8 | WorkOrderRouting | JDE | WorkOrderRouting | Yes | Yes | Yes | Yes | `workorderrouting.pb.zstd` | |
| 9 | Branch | JDE | Branch | Yes | Yes | Yes | Yes | `branch.pb.zstd` | typeCode='BP' |
| 10 | ProfitCenter | JDE | ProfitCenter | Yes | Yes | Yes | Yes | `profitcenter.pb.zstd` | typeCode='I3' |
| 11 | WorkCenter | JDE | WorkCenter | Yes | Yes | Yes | Yes | `workcenter.pb.zstd` | typeCode='WC' |
| 12 | StatusCode | JDE | StatusCode | Yes | Yes | Yes | Yes | *(none)* | GIW connection |
| 13 | JdeUser | JDE | JdeUser | Yes | Yes | Yes | No | `jdeuser.pb.zstd` | Same query both |
| 14 | OrgHierarchy | JDE | OrgHierarchy | Yes | Yes | Yes | Yes | `orghierarchy.pb.zstd` | |
| 15 | RouteMaster | JDE | RouteMaster | Yes | Yes | Yes | Yes | `routemaster.pb.zstd` | |
| 16 | FunctionCode | JDE | FunctionCode | Yes | Yes | Yes | No | `functioncode.pb.zstd` | Always full reload |
| 17 | MisData | CMS | MisData | Yes | Yes | No | Yes | `misdata.pb.zstd` | Hourly disabled |

### Archive Syncs (5) - ALL DISABLED

| # | Import Name | Source | Dest Table | Mass | Daily | Hourly | Filter | Cache File | Notes |
|---|-------------|--------|------------|------|-------|--------|--------|------------|-------|
| 18 | WorkOrder_Archive | JDE | WorkOrder_Hist | No | No | No | No | `workorder_hist.pb.zstd` | DISABLED |
| 19 | LotUsage_Archive | JDE | LotUsage_Hist | No | No | No | No | `lotusage_hist.pb.zstd` | DISABLED |
| 20 | WorkOrderTime_Archive | JDE | WorkOrderTime_Hist | No | No | No | No | `workordertime_hist.pb.zstd` | DISABLED |
| 21 | WorkOrderComponent_Archive | JDE | WorkOrderComponent_Hist | No | No | No | No | `workordercomponent_hist.pb.zstd` | DISABLED |
| 22 | WorkOrderStep_Archive | JDE | WorkOrderStep_Hist | No | No | No | No | `workorderstep_hist.pb.zstd` | DISABLED |

**Cache File Location:** `CACHED_DB_FILES/`

---

## Detailed Entity Configurations

### 1. WorkOrder

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDER |
| Destination Table | WorkOrder_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrders |
| Post Processing | None |
| JDE Table | JDESTAGE.F4801_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_WORKORDERS.sql`
- Filtered: `JDE/SQL_GET_WORKORDERS_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDERS.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 2. LotUsage

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | LOTUSAGE |
| Destination Table | LotUsage_Curr |
| Data Fetch Function | Commons.Process.JDE.GetLotUsages |
| Post Processing | None |
| JDE Table | JDESTAGE.F4111_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_LOT_USAGES.sql`
- Filtered: `JDE/SQL_GET_LOT_USAGES_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_LOT_USAGES.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 3. Item

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | ITEM |
| Destination Table | Item |
| Data Fetch Function | Commons.Process.JDE.GetItems |
| Post Processing | None |
| JDE Table | JDESTAGE.F4101_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_ITEMS.sql`
- Filtered: `JDE/SQL_GET_ITEMS_FILTERED.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 4. Lot

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | LOT |
| Destination Table | Lot |
| Data Fetch Function | Commons.Process.JDE.GetLots |
| Post Processing | None |
| JDE Table | JDESTAGE.F4108_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_LOTS.sql`
- Filtered: `JDE/SQL_GET_LOTS_FILTERED.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 5. WorkOrderTime

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERTIME |
| Destination Table | WorkOrderTime_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderTimes |
| Post Processing | None |
| JDE Table | JDESTAGE.F31122_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_TIMES.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_TIMES_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDER_TIMES.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 6. WorkOrderComponent

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERCOMPONENT |
| Destination Table | WorkOrderComponent_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderComponents |
| Post Processing | None |
| JDE Table | JDESTAGE.F3111_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_COMPONENTS.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_COMPONENTS_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDER_COMPONENTS.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 7. WorkOrderStep

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERSTEP |
| Destination Table | WorkOrderStep_Curr |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderSteps |
| Post Processing | None |
| JDE Table | JDESTAGE.F3112_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_STEP.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_STEP_FILTERED.sql`
- Archive: `JDE_ARCHIVE/SQL_GET_WORKORDER_STEP.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 8. WorkOrderRouting

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKORDERROUTING |
| Destination Table | WorkOrderRouting |
| Data Fetch Function | Commons.Process.JDE.GetWorkOrderRoutings |
| Post Processing | None |
| JDE Table | JDESTAGE.F3112Z1_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_WORKORDER_ROUTING.sql`
- Filtered: `JDE/SQL_GET_WORKORDER_ROUTING_FILTERED.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

**Special Processing:** Filters out records with invalid dates (year < 1900 or > 2500)

---

### 9. Branch

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | BRANCH |
| Destination Table | Branch |
| Data Fetch Function | Commons.Process.JDE.GetBranches |
| Post Processing | None |
| JDE Table | JDESTAGE.F0006_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_BUSINESS_UNITS.sql`
- Filtered: `JDE/SQL_GET_BUSINESS_UNITS_FILTERED.sql`

**Filter Parameters:** `typeCode='BP'`, `dateUpdated`, `timeUpdated`

---

### 10. ProfitCenter

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | PROFITCENTER |
| Destination Table | ProfitCenter |
| Data Fetch Function | Commons.Process.JDE.GetProfitCenters |
| Post Processing | None |
| JDE Table | JDESTAGE.F0006_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_BUSINESS_UNITS.sql`
- Filtered: `JDE/SQL_GET_BUSINESS_UNITS_FILTERED.sql`

**Filter Parameters:** `typeCode='I3'`, `dateUpdated`, `timeUpdated`

---

### 11. WorkCenter

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | WORKCENTER |
| Destination Table | WorkCenter |
| Data Fetch Function | Commons.Process.JDE.GetWorkCenters |
| Post Processing | None |
| JDE Table | JDESTAGE.F0006_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_BUSINESS_UNITS.sql`
- Filtered: `JDE/SQL_GET_BUSINESS_UNITS_FILTERED.sql`

**Filter Parameters:** `typeCode='WC'`, `dateUpdated`, `timeUpdated`

---

### 12. StatusCode

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | STATUSCODE |
| Destination Table | StatusCode |
| Data Fetch Function | Commons.Process.JDE.GetStatusCodes |
| Post Processing | None |
| JDE Table | JDESTAGE.F0005_VIEW |
| **Connection** | **GIW (not JDE)** |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_STATUS_CODES.sql`
- Filtered: `JDE/SQL_GET_STATUS_CODES_FILTERED.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated`

**Note:** This sync uses the GIW connection (Config.GIWCS) via DDTek.Oracle, not the standard JDE connection.

---

### 13. JdeUser

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | USER |
| Destination Table | JdeUser |
| Data Fetch Function | Commons.Process.JDE.GetUsers |
| Post Processing | None |
| JDE Tables | JDESTAGE.F0101_VIEW, JDESTAGE.F0092_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_USERS.sql`

**Note:** No filtered query variant exists. The same query is used for both full and incremental syncs (filter parameters are passed but not used in query).

---

### 14. OrgHierarchy

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | ORGHIERARCHY |
| Destination Table | OrgHierarchy |
| Data Fetch Function | Commons.Process.JDE.GetOrgHierarchy |
| Post Processing | None |
| JDE Table | JDESTAGE.F30006_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_ORG_HIERARCHY.sql`
- Filtered: `JDE/SQL_GET_ORG_HIERARCHY_FILTERED.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 15. RouteMaster

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | ROUTEMASTER |
| Destination Table | RouteMaster |
| Data Fetch Function | Commons.Process.JDE.GetRouteMasters |
| Post Processing | None |
| JDE Table | JDESTAGE.F3003_VIEW |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | Yes | 60 | No | No |

**Query Files:**
- Full: `JDE/SQL_GET_ROUTE_MASTER.sql`
- Filtered: `JDE/SQL_GET_ROUTE_MASTER_FILTERED.sql`

**Filter Parameters:** `dateUpdated`, `timeUpdated` (JDE date/time format)

---

### 16. FunctionCode

| Property | Value |
|----------|-------|
| Source System | JDE |
| Source Data | FUNCTIONCODE |
| Destination Table | FunctionCode |
| Data Fetch Function | Commons.Process.JDE.GetFunctionCodes |
| Post Processing | None |
| JDE Table | PRODDTA.F00192 (direct table, not view) |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 10080 | Yes | Yes |
| Daily | Yes | 1440 | Yes | Yes |
| Hourly | Yes | 60 | Yes | Yes |

**Query Files:**
- Full: `JDE/SQL_GET_FUNCTION_CODES.sql`

**Note:** No filtered query variant. Always performs full reload (PrepurgeData=true for all schedules). Uses aggregation with LISTAGG to combine multi-row descriptions.

---

### 17. MisData

| Property | Value |
|----------|-------|
| Source System | CMS |
| Source Data | MISDATA |
| Destination Table | MisData |
| Data Fetch Function | Commons.Process.CMS.GetMisData |
| Post Processing | Commons.Process.LotFinderDB.PostProcessMisData |
| CMS Schema | INFODBA |

**Schedule Configuration:**
| Schedule | Enabled | Interval | Prepurge | ReIndex |
|----------|---------|----------|----------|---------|
| Mass | Yes | 100800 | Yes | Yes |
| Daily | Yes | 1440 | No | No |
| Hourly | **No** | 60 | No | No |

**Query Files:**
- Full: `CMS/SQL_GET_MIS_DATA.sql`
- Filtered: `CMS/SQL_GET_MIS_DATA_FILTERED.sql`

**Filter Parameters:** `lastUpdateDT` (standard DateTime)

**Note:**
- Mass interval is 100800 minutes (~10 weeks), much longer than other syncs
- Hourly sync is disabled
- Has post-processing action for additional data transformation
- Query timeout is extended (1200*50 = 60000 seconds)
- ReleaseDate is converted to local time after fetch

---

## Archive Query Pattern

Archive syncs (#18-22) fetch historical data using UNION ALL from both current and archived schemas:

```sql
SELECT ... FROM QADTA.F{table}
UNION ALL
SELECT ... FROM ARCDTAQA.F{table}
```

**Query Files:** Located in `JDE_ARCHIVE/` folder.

**Note:** Archive syncs have `IsEnabled=true` in dsconfig but all schedule types have `Enabled=false`. They are configured for manual/on-demand execution only.

---

## Source Database Reference

### JDE Oracle Connection (Config.JDECS)
- **Driver:** Oracle.ManagedDataAccess.Client
- **Schemas Used:**
  - `JDESTAGE` - Views for current production data
  - `PRODDTA` - Direct table access (FunctionCode only)
  - `QADTA` - Current data for archive queries
  - `ARCDTAQA` - Archived historical data

### GIW Oracle Connection (Config.GIWCS)
- **Driver:** DDTek.Oracle
- **Used By:** StatusCode sync only
- **Schema:** JDESTAGE

### CMS Oracle Connection (Config.CMSCS)
- **Driver:** DDTek.Oracle
- **Schema:** INFODBA
- **Tables:** Complex 11-table join for MIS data

### JDE Date/Time Format
JDE uses a special date/time format conversion:
- Dates are converted via `ToJDEDate()` helper
- Times are converted via `ToJDETime()` helper
- Filter conditions use: `(date > :dateUpdated) OR (date = :dateUpdated AND time >= :timeUpdated)`

---

## Source Files Reference

| File Type | Location |
|-----------|----------|
| dsconfig JSON files | OLD/WorkerService/dsconfig/*.json |
| JDE Source Queries | OLD/WorkerService/bin/UPDATER/JdeQueries/*.sql |
| JDE Archive Queries | OLD/WorkerService/bin/UPDATER/JdeArchivalQueries/*.sql |
| CMS Source Queries | OLD/DataModel/CmsQueries/*.sql |
| C# Data Fetch Code | OLD/DataModel/Process/JDE.*.cs, CMS.*.cs |
| Configuration | OLD/DataModel/Config.cs |
| Query Repository | OLD/DataModel/Process/QueryRepository.cs |