jdescopingtool/PLANS/MisData_Archival_Implementation.md

MisData Archival Implementation Plan

Overview

Split the MisData table into MisData_Curr and MisData_Hist following the existing archival pattern used for WorkOrder, LotUsage, WorkOrderComponent, WorkOrderStep, and WorkOrderTime tables.

Design Decisions

Decision	Choice	Rationale
Partitioning Strategy	Status-based	Status='Current' → MisData_Curr, Status='BackLevel' → MisData_Hist
DEV Data Files	Split existing file	Create misdata_curr.pb.zstd (Current rows), misdata_hist.pb.zstd (BackLevel rows)
PostScripts	Run on both tables	ObsoleteDate update logic applies to both Current and BackLevel records
View Name	Keep MisData	Maintains backward compatibility with fn_MatchMIS function

Implementation Steps

Phase 1: Database Project Updates

1.1 Create MisData_Hist Table Script

• File: Scripts/012a_CreateMisDataHistTable.sql
• Action: Create new script for MisData_Hist (identical schema to current MisData)
• PK: Rename to PK_MisData_Hist

1.2 Create MisData_Curr Table Script

• File: Scripts/012b_CreateMisDataCurrTable.sql
• Action: Create new script for MisData_Curr (identical schema)
• PK: PK_MisData_Curr

1.3 Create MisData View Script

• File: Scripts/033_CreateMisDataView.sql
• Action: Create UNION ALL view combining both tables
• Pattern: Follow 030_CreateLotUsageView.sql pattern

1.4 Update Original Script (for new installs)

• File: Scripts/012_CreateMisDataTable.sql
• Action: Rename to create MisData_Hist instead

Phase 2: Live Database Migration

Execute directly against local database (no migrations):

-- Step 1: Rename existing table and constraint
EXEC sp_rename 'dbo.MisData', 'MisData_Hist';
EXEC sp_rename 'dbo.PK_MisData', 'PK_MisData_Hist';

-- Step 2: Create MisData_Curr with identical schema
CREATE TABLE [dbo].[MisData_Curr] (
    -- identical columns
    CONSTRAINT [PK_MisData_Curr] PRIMARY KEY CLUSTERED(...)
);

-- Step 3: Move Current status rows to _Curr table
INSERT INTO dbo.MisData_Curr SELECT * FROM dbo.MisData_Hist WHERE Status = 'Current';
DELETE FROM dbo.MisData_Hist WHERE Status = 'Current';

-- Step 4: Create unified view
CREATE OR ALTER VIEW [dbo].[MisData] AS (
    SELECT hist.* FROM dbo.MisData_Hist hist
    UNION ALL
    SELECT curr.* FROM dbo.MisData_Curr curr
);

Phase 3: Production Pipeline Updates

File: src/JdeScoping.DataSync/Pipelines/pipelines.json

3.1 MisData_Curr Pipeline (Daily Incremental Sync)

• Daily schedule: Merge into MisData_Curr (no purge/reload)
• Mass schedule: Full purge/reload
• Hourly: Disabled
• PostScripts (in order):
  1. Update ObsoleteDate for Current records where a BackLevel exists
  2. Update ObsoleteDate for remaining NULL values
  3. MERGE BackLevel rows from MisData_Curr into MisData_Hist (overwrite if exists)
  4. DELETE BackLevel rows from MisData_Curr
  5. Rebuild index

3.2 MisData_Hist Pipeline (Mass Refresh Only)

• Daily schedule: Disabled (populated via MisData_Curr postScripts)
• Mass schedule: Full purge/reload of BackLevel data
• Hourly: Disabled
• PostScripts: ObsoleteDate updates + index rebuild

3.3 Data Flow

Daily Sync:
  CMS (Current status) → MERGE → MisData_Curr
                                    ↓
                         [postScripts move BackLevel rows]
                                    ↓
                               MisData_Hist

Mass Sync:
  CMS (Current) → PURGE/RELOAD → MisData_Curr
  CMS (BackLevel) → PURGE/RELOAD → MisData_Hist

Phase 4: DEV Pipeline Updates

File: src/JdeScoping.DataSync.Dev/Pipelines/dev-pipelines.json

4.1 Update Size Categories

"sizeCategories": {
    "large": ["Lot", "MisData_Hist", "WorkOrder_Curr", ...],
    "medium": [..., "MisData_Curr"]  // Current status rows are fewer
}

4.2 Split Pipeline Entry

"MisData_Curr": {
    "source": { "fileName": "misdata_curr.pb.zstd" },
    "destination": { "table": "MisData_Curr" }
},
"MisData_Hist": {
    "source": { "fileName": "misdata_hist.pb.zstd" },
    "destination": { "table": "MisData_Hist" }
}

4.3 Data File Split

• Rename existing misdata.pb.zstd → misdata_hist.pb.zstd
• Create misdata_curr.pb.zstd with Current status rows extracted
• Note: This requires processing the existing file to split by Status

Phase 5: Unit Test Updates

5.1 DevEtlPipelineFactoryTests

• Update tests expecting MisData pipeline → expect MisData_Curr and MisData_Hist
• Add tests for size category assignments

5.2 DevEtlRegistryTests

• Update registry enumeration tests for new pipeline names

Phase 6: Documentation Updates

Update data sync documentation to reflect:

• MisData split into _Curr (Current status) and _Hist (BackLevel status)
• View combining both tables
• Pipeline configuration changes

Verification Checklist

• Database project builds without errors
• Local database has MisData_Curr, MisData_Hist tables and MisData view
• fn_MatchMIS function works with the view
• Production pipeline JSON is valid
• DEV pipeline JSON is valid
• All unit tests pass
• Solution builds successfully

Rollback Plan

If issues arise:

1. Drop view: DROP VIEW dbo.MisData
2. Rename table back: EXEC sp_rename 'dbo.MisData_Hist', 'MisData'
3. Rename constraint: EXEC sp_rename 'dbo.PK_MisData_Hist', 'PK_MisData'
4. Drop _Curr table: DROP TABLE dbo.MisData_Curr
5. Revert pipeline JSON changes