jdescopingtool/PLANS/MisData_Archival_Implementation.md

# MisData Archival Implementation Plan

## Overview

Split the MisData table into `MisData_Curr` and `MisData_Hist` following the existing archival pattern used for WorkOrder, LotUsage, WorkOrderComponent, WorkOrderStep, and WorkOrderTime tables.

## Design Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Partitioning Strategy | Status-based | `Status='Current'` → MisData_Curr, `Status='BackLevel'` → MisData_Hist |
| DEV Data Files | Split existing file | Create misdata_curr.pb.zstd (Current rows), misdata_hist.pb.zstd (BackLevel rows) |
| PostScripts | Run on both tables | ObsoleteDate update logic applies to both Current and BackLevel records |
| View Name | Keep `MisData` | Maintains backward compatibility with fn_MatchMIS function |

## Implementation Steps

### Phase 1: Database Project Updates

#### 1.1 Create MisData_Hist Table Script
- **File**: `Scripts/012a_CreateMisDataHistTable.sql`
- **Action**: Create new script for MisData_Hist (identical schema to current MisData)
- **PK**: Rename to `PK_MisData_Hist`

#### 1.2 Create MisData_Curr Table Script
- **File**: `Scripts/012b_CreateMisDataCurrTable.sql`
- **Action**: Create new script for MisData_Curr (identical schema)
- **PK**: `PK_MisData_Curr`

#### 1.3 Create MisData View Script
- **File**: `Scripts/033_CreateMisDataView.sql`
- **Action**: Create UNION ALL view combining both tables
- **Pattern**: Follow 030_CreateLotUsageView.sql pattern

#### 1.4 Update Original Script (for new installs)
- **File**: `Scripts/012_CreateMisDataTable.sql`
- **Action**: Rename to create MisData_Hist instead

### Phase 2: Live Database Migration

Execute directly against local database (no migrations):

```sql
-- Step 1: Rename existing table and constraint
EXEC sp_rename 'dbo.MisData', 'MisData_Hist';
EXEC sp_rename 'dbo.PK_MisData', 'PK_MisData_Hist';

-- Step 2: Create MisData_Curr with identical schema
CREATE TABLE [dbo].[MisData_Curr] (
    -- identical columns
    CONSTRAINT [PK_MisData_Curr] PRIMARY KEY CLUSTERED(...)
);

-- Step 3: Move Current status rows to _Curr table
INSERT INTO dbo.MisData_Curr SELECT * FROM dbo.MisData_Hist WHERE Status = 'Current';
DELETE FROM dbo.MisData_Hist WHERE Status = 'Current';

-- Step 4: Create unified view
CREATE OR ALTER VIEW [dbo].[MisData] AS (
    SELECT hist.* FROM dbo.MisData_Hist hist
    UNION ALL
    SELECT curr.* FROM dbo.MisData_Curr curr
);
```

### Phase 3: Production Pipeline Updates

**File**: `src/JdeScoping.DataSync/Pipelines/pipelines.json`

#### 3.1 MisData_Curr Pipeline (Daily Incremental Sync)

- **Daily schedule**: Merge into MisData_Curr (no purge/reload)
- **Mass schedule**: Full purge/reload
- **Hourly**: Disabled
- **PostScripts** (in order):
  1. Update ObsoleteDate for Current records where a BackLevel exists
  2. Update ObsoleteDate for remaining NULL values
  3. MERGE BackLevel rows from MisData_Curr into MisData_Hist (overwrite if exists)
  4. DELETE BackLevel rows from MisData_Curr
  5. Rebuild index

#### 3.2 MisData_Hist Pipeline (Mass Refresh Only)

- **Daily schedule**: Disabled (populated via MisData_Curr postScripts)
- **Mass schedule**: Full purge/reload of BackLevel data
- **Hourly**: Disabled
- **PostScripts**: ObsoleteDate updates + index rebuild

#### 3.3 Data Flow

```
Daily Sync:
  CMS (Current status) → MERGE → MisData_Curr
                                    ↓
                         [postScripts move BackLevel rows]
                                    ↓
                               MisData_Hist

Mass Sync:
  CMS (Current) → PURGE/RELOAD → MisData_Curr
  CMS (BackLevel) → PURGE/RELOAD → MisData_Hist
```

### Phase 4: DEV Pipeline Updates

**File**: `src/JdeScoping.DataSync.Dev/Pipelines/dev-pipelines.json`

#### 4.1 Update Size Categories
```json
"sizeCategories": {
    "large": ["Lot", "MisData_Hist", "WorkOrder_Curr", ...],
    "medium": [..., "MisData_Curr"]  // Current status rows are fewer
}
```

#### 4.2 Split Pipeline Entry
```json
"MisData_Curr": {
    "source": { "fileName": "misdata_curr.pb.zstd" },
    "destination": { "table": "MisData_Curr" }
},
"MisData_Hist": {
    "source": { "fileName": "misdata_hist.pb.zstd" },
    "destination": { "table": "MisData_Hist" }
}
```

#### 4.3 Data File Split
- Rename existing `misdata.pb.zstd` → `misdata_hist.pb.zstd`
- Create `misdata_curr.pb.zstd` with Current status rows extracted
- **Note**: This requires processing the existing file to split by Status

### Phase 5: Unit Test Updates

#### 5.1 DevEtlPipelineFactoryTests
- Update tests expecting `MisData` pipeline → expect `MisData_Curr` and `MisData_Hist`
- Add tests for size category assignments

#### 5.2 DevEtlRegistryTests
- Update registry enumeration tests for new pipeline names

### Phase 6: Documentation Updates

Update data sync documentation to reflect:
- MisData split into _Curr (Current status) and _Hist (BackLevel status)
- View combining both tables
- Pipeline configuration changes

## Verification Checklist

- [ ] Database project builds without errors
- [ ] Local database has MisData_Curr, MisData_Hist tables and MisData view
- [ ] fn_MatchMIS function works with the view
- [ ] Production pipeline JSON is valid
- [ ] DEV pipeline JSON is valid
- [ ] All unit tests pass
- [ ] Solution builds successfully

## Rollback Plan

If issues arise:
1. Drop view: `DROP VIEW dbo.MisData`
2. Rename table back: `EXEC sp_rename 'dbo.MisData_Hist', 'MisData'`
3. Rename constraint: `EXEC sp_rename 'dbo.PK_MisData_Hist', 'PK_MisData'`
4. Drop _Curr table: `DROP TABLE dbo.MisData_Curr`
5. Revert pipeline JSON changes