docs: add MisData archival implementation and unit test report

Document the MisData_Curr/Hist table split in architecture docs, add post-sync archival scripts to move BackLevel records to history table, and generate comprehensive unit test report (856 tests, 100% pass rate).
2026-01-19 04:03:25 -05:00
parent 1e5b1bb3da
commit 08f5aa1447
4 changed files with 339 additions and 1 deletions
@@ -0,0 +1,166 @@
+# MisData Archival Implementation Plan
+
+## Overview
+
+Split the MisData table into `MisData_Curr` and `MisData_Hist` following the existing archival pattern used for WorkOrder, LotUsage, WorkOrderComponent, WorkOrderStep, and WorkOrderTime tables.
+
+## Design Decisions
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| Partitioning Strategy | Status-based | `Status='Current'` → MisData_Curr, `Status='BackLevel'` → MisData_Hist |
+| DEV Data Files | Split existing file | Create misdata_curr.pb.zstd (Current rows), misdata_hist.pb.zstd (BackLevel rows) |
+| PostScripts | Run on both tables | ObsoleteDate update logic applies to both Current and BackLevel records |
+| View Name | Keep `MisData` | Maintains backward compatibility with fn_MatchMIS function |
+
+## Implementation Steps
+
+### Phase 1: Database Project Updates
+
+#### 1.1 Create MisData_Hist Table Script
+- **File**: `Scripts/012a_CreateMisDataHistTable.sql`
+- **Action**: Create new script for MisData_Hist (identical schema to current MisData)
+- **PK**: Rename to `PK_MisData_Hist`
+
+#### 1.2 Create MisData_Curr Table Script
+- **File**: `Scripts/012b_CreateMisDataCurrTable.sql`
+- **Action**: Create new script for MisData_Curr (identical schema)
+- **PK**: `PK_MisData_Curr`
+
+#### 1.3 Create MisData View Script
+- **File**: `Scripts/033_CreateMisDataView.sql`
+- **Action**: Create UNION ALL view combining both tables
+- **Pattern**: Follow 030_CreateLotUsageView.sql pattern
+
+#### 1.4 Update Original Script (for new installs)
+- **File**: `Scripts/012_CreateMisDataTable.sql`
+- **Action**: Rename to create MisData_Hist instead
+
+### Phase 2: Live Database Migration
+
+Execute directly against local database (no migrations):
+
+```sql
+-- Step 1: Rename existing table and constraint
+EXEC sp_rename 'dbo.MisData', 'MisData_Hist';
+EXEC sp_rename 'dbo.PK_MisData', 'PK_MisData_Hist';
+
+-- Step 2: Create MisData_Curr with identical schema
+CREATE TABLE [dbo].[MisData_Curr] (
+    -- identical columns
+    CONSTRAINT [PK_MisData_Curr] PRIMARY KEY CLUSTERED(...)
+);
+
+-- Step 3: Move Current status rows to _Curr table
+INSERT INTO dbo.MisData_Curr SELECT * FROM dbo.MisData_Hist WHERE Status = 'Current';
+DELETE FROM dbo.MisData_Hist WHERE Status = 'Current';
+
+-- Step 4: Create unified view
+CREATE OR ALTER VIEW [dbo].[MisData] AS (
+    SELECT hist.* FROM dbo.MisData_Hist hist
+    UNION ALL
+    SELECT curr.* FROM dbo.MisData_Curr curr
+);
+```
+
+### Phase 3: Production Pipeline Updates
+
+**File**: `src/JdeScoping.DataSync/Pipelines/pipelines.json`
+
+#### 3.1 MisData_Curr Pipeline (Daily Incremental Sync)
+
+- **Daily schedule**: Merge into MisData_Curr (no purge/reload)
+- **Mass schedule**: Full purge/reload
+- **Hourly**: Disabled
+- **PostScripts** (in order):
+  1. Update ObsoleteDate for Current records where a BackLevel exists
+  2. Update ObsoleteDate for remaining NULL values
+  3. MERGE BackLevel rows from MisData_Curr into MisData_Hist (overwrite if exists)
+  4. DELETE BackLevel rows from MisData_Curr
+  5. Rebuild index
+
+#### 3.2 MisData_Hist Pipeline (Mass Refresh Only)
+
+- **Daily schedule**: Disabled (populated via MisData_Curr postScripts)
+- **Mass schedule**: Full purge/reload of BackLevel data
+- **Hourly**: Disabled
+- **PostScripts**: ObsoleteDate updates + index rebuild
+
+#### 3.3 Data Flow
+
+```
+Daily Sync:
+  CMS (Current status) → MERGE → MisData_Curr
+                                    ↓
+                         [postScripts move BackLevel rows]
+                                    ↓
+                               MisData_Hist
+
+Mass Sync:
+  CMS (Current) → PURGE/RELOAD → MisData_Curr
+  CMS (BackLevel) → PURGE/RELOAD → MisData_Hist
+```
+
+### Phase 4: DEV Pipeline Updates
+
+**File**: `src/JdeScoping.DataSync.Dev/Pipelines/dev-pipelines.json`
+
+#### 4.1 Update Size Categories
+```json
+"sizeCategories": {
+    "large": ["Lot", "MisData_Hist", "WorkOrder_Curr", ...],
+    "medium": [..., "MisData_Curr"]  // Current status rows are fewer
+}
+```
+
+#### 4.2 Split Pipeline Entry
+```json
+"MisData_Curr": {
+    "source": { "fileName": "misdata_curr.pb.zstd" },
+    "destination": { "table": "MisData_Curr" }
+},
+"MisData_Hist": {
+    "source": { "fileName": "misdata_hist.pb.zstd" },
+    "destination": { "table": "MisData_Hist" }
+}
+```
+
+#### 4.3 Data File Split
+- Rename existing `misdata.pb.zstd` → `misdata_hist.pb.zstd`
+- Create `misdata_curr.pb.zstd` with Current status rows extracted
+- **Note**: This requires processing the existing file to split by Status
+
+### Phase 5: Unit Test Updates
+
+#### 5.1 DevEtlPipelineFactoryTests
+- Update tests expecting `MisData` pipeline → expect `MisData_Curr` and `MisData_Hist`
+- Add tests for size category assignments
+
+#### 5.2 DevEtlRegistryTests
+- Update registry enumeration tests for new pipeline names
+
+### Phase 6: Documentation Updates
+
+Update data sync documentation to reflect:
+- MisData split into _Curr (Current status) and _Hist (BackLevel status)
+- View combining both tables
+- Pipeline configuration changes
+
+## Verification Checklist
+
+- [ ] Database project builds without errors
+- [ ] Local database has MisData_Curr, MisData_Hist tables and MisData view
+- [ ] fn_MatchMIS function works with the view
+- [ ] Production pipeline JSON is valid
+- [ ] DEV pipeline JSON is valid
+- [ ] All unit tests pass
+- [ ] Solution builds successfully
+
+## Rollback Plan
+
+If issues arise:
+1. Drop view: `DROP VIEW dbo.MisData`
+2. Rename table back: `EXEC sp_rename 'dbo.MisData_Hist', 'MisData'`
+3. Rename constraint: `EXEC sp_rename 'dbo.PK_MisData_Hist', 'PK_MisData'`
+4. Drop _Curr table: `DROP TABLE dbo.MisData_Curr`
+5. Revert pipeline JSON changes