docs: add MisData archival implementation and unit test report

Document the MisData_Curr/Hist table split in architecture docs, add post-sync archival scripts to move BackLevel records to history table, and generate comprehensive unit test report (856 tests, 100% pass rate).
2026-01-19 04:03:25 -05:00
parent 1e5b1bb3da
commit 08f5aa1447
4 changed files with 339 additions and 1 deletions
@@ -127,6 +127,8 @@ The scoping tool cache database includes these primary tables:
 | `WorkOrder_Hist` | Historical work orders from JDE |
 | `LotUsage_Curr` | Current lot usage from CMS |
 | `LotUsage_Hist` | Historical lot usage from CMS |
+| `MisData_Curr` | Current MIS data (Status='Current') from JDE |
+| `MisData_Hist` | Historical MIS data (Status='BackLevel') from JDE |
 | `Lot` | Lot reference data |
 | `Item` | Item master reference data |
 | `WorkCenter` | Work center reference data |
@@ -134,6 +136,12 @@ The scoping tool cache database includes these primary tables:
 | `ProfitCenter` | Profit center reference data |
 | `SchemaVersions` | DbUp tracking table (auto-created) |

+### Views
+
+| View | Purpose |
+|------|---------|
+| `MisData` | UNION ALL of `MisData_Curr` and `MisData_Hist` for unified MIS data access |
+
 ## Search Criteria Extraction Functions

 The database includes SQL functions that extract filter criteria from the `Search.Criteria` JSON column. These functions enable the query builder to pass only a `SearchId` parameter, with all filter extraction happening in SQL Server.
@@ -194,6 +194,8 @@
      "postScripts": [
        "SET ANSI_WARNINGS OFF; WITH cte AS (SELECT md.MisNumber, md.RevID, md.Status, MIN(md.ReleaseDate) Released FROM dbo.MisData_Curr AS md GROUP BY md.MisNumber, md.RevID, md.Status) UPDATE dbo.MisData_Curr SET ObsoleteDate = bl.Released FROM cte bl WHERE MisData_Curr.MisNumber = bl.MisNumber AND MisData_Curr.RevID = bl.RevID AND MisData_Curr.Status = 'Current' AND bl.Status = 'BackLevel';",
        "WITH cte AS (SELECT md.MisNumber, md.RevID, md.Status, MIN(md.ReleaseDate) Released FROM dbo.MisData_Curr AS md GROUP BY md.MisNumber, md.RevID, md.Status) UPDATE dbo.MisData_Curr SET ObsoleteDate = (SELECT TOP 1 nl.Released FROM cte nl WHERE MisData_Curr.MisNumber = nl.MisNumber AND MisData_Curr.RevID < nl.RevID AND MisData_Curr.Status = nl.Status ORDER BY nl.RevID) WHERE ObsoleteDate IS NULL;",
+        "MERGE INTO dbo.MisData_Hist AS target USING (SELECT * FROM dbo.MisData_Curr WHERE Status = 'BackLevel') AS source ON target.ItemNumber = source.ItemNumber AND target.BranchCode = source.BranchCode AND target.SequenceNumber = source.SequenceNumber AND target.MisNumber = source.MisNumber AND target.CharNumber = source.CharNumber WHEN MATCHED THEN UPDATE SET target.RevID = source.RevID, target.TestDescription = source.TestDescription, target.SamplingType = source.SamplingType, target.SamplingValue = source.SamplingValue, target.ToolsGauges = source.ToolsGauges, target.WorkInstructions = source.WorkInstructions, target.Status = source.Status, target.ReleaseDate = source.ReleaseDate, target.ObsoleteDate = source.ObsoleteDate WHEN NOT MATCHED THEN INSERT (ItemNumber, BranchCode, SequenceNumber, MisNumber, RevID, CharNumber, TestDescription, SamplingType, SamplingValue, ToolsGauges, WorkInstructions, Status, ReleaseDate, ObsoleteDate) VALUES (source.ItemNumber, source.BranchCode, source.SequenceNumber, source.MisNumber, source.RevID, source.CharNumber, source.TestDescription, source.SamplingType, source.SamplingValue, source.ToolsGauges, source.WorkInstructions, source.Status, source.ReleaseDate, source.ObsoleteDate);",
+        "DELETE FROM dbo.MisData_Curr WHERE Status = 'BackLevel';",
        "ALTER INDEX [PK_MisData_Curr] ON [dbo].[MisData_Curr] REBUILD;"
      ]
    },
@@ -208,7 +210,7 @@
      },
      "schedules": {
        "mass": { "intervalMinutes": 100800 },
-        "daily": {},
+        "daily": { "enabled": false },
        "hourly": { "enabled": false }
      },
      "destination": {
@@ -0,0 +1,166 @@
+# MisData Archival Implementation Plan
+
+## Overview
+
+Split the MisData table into `MisData_Curr` and `MisData_Hist` following the existing archival pattern used for WorkOrder, LotUsage, WorkOrderComponent, WorkOrderStep, and WorkOrderTime tables.
+
+## Design Decisions
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| Partitioning Strategy | Status-based | `Status='Current'` → MisData_Curr, `Status='BackLevel'` → MisData_Hist |
+| DEV Data Files | Split existing file | Create misdata_curr.pb.zstd (Current rows), misdata_hist.pb.zstd (BackLevel rows) |
+| PostScripts | Run on both tables | ObsoleteDate update logic applies to both Current and BackLevel records |
+| View Name | Keep `MisData` | Maintains backward compatibility with fn_MatchMIS function |
+
+## Implementation Steps
+
+### Phase 1: Database Project Updates
+
+#### 1.1 Create MisData_Hist Table Script
+- **File**: `Scripts/012a_CreateMisDataHistTable.sql`
+- **Action**: Create new script for MisData_Hist (identical schema to current MisData)
+- **PK**: Rename to `PK_MisData_Hist`
+
+#### 1.2 Create MisData_Curr Table Script
+- **File**: `Scripts/012b_CreateMisDataCurrTable.sql`
+- **Action**: Create new script for MisData_Curr (identical schema)
+- **PK**: `PK_MisData_Curr`
+
+#### 1.3 Create MisData View Script
+- **File**: `Scripts/033_CreateMisDataView.sql`
+- **Action**: Create UNION ALL view combining both tables
+- **Pattern**: Follow 030_CreateLotUsageView.sql pattern
+
+#### 1.4 Update Original Script (for new installs)
+- **File**: `Scripts/012_CreateMisDataTable.sql`
+- **Action**: Rename to create MisData_Hist instead
+
+### Phase 2: Live Database Migration
+
+Execute directly against local database (no migrations):
+
+```sql
+-- Step 1: Rename existing table and constraint
+EXEC sp_rename 'dbo.MisData', 'MisData_Hist';
+EXEC sp_rename 'dbo.PK_MisData', 'PK_MisData_Hist';
+
+-- Step 2: Create MisData_Curr with identical schema
+CREATE TABLE [dbo].[MisData_Curr] (
+    -- identical columns
+    CONSTRAINT [PK_MisData_Curr] PRIMARY KEY CLUSTERED(...)
+);
+
+-- Step 3: Move Current status rows to _Curr table
+INSERT INTO dbo.MisData_Curr SELECT * FROM dbo.MisData_Hist WHERE Status = 'Current';
+DELETE FROM dbo.MisData_Hist WHERE Status = 'Current';
+
+-- Step 4: Create unified view
+CREATE OR ALTER VIEW [dbo].[MisData] AS (
+    SELECT hist.* FROM dbo.MisData_Hist hist
+    UNION ALL
+    SELECT curr.* FROM dbo.MisData_Curr curr
+);
+```
+
+### Phase 3: Production Pipeline Updates
+
+**File**: `src/JdeScoping.DataSync/Pipelines/pipelines.json`
+
+#### 3.1 MisData_Curr Pipeline (Daily Incremental Sync)
+
+- **Daily schedule**: Merge into MisData_Curr (no purge/reload)
+- **Mass schedule**: Full purge/reload
+- **Hourly**: Disabled
+- **PostScripts** (in order):
+  1. Update ObsoleteDate for Current records where a BackLevel exists
+  2. Update ObsoleteDate for remaining NULL values
+  3. MERGE BackLevel rows from MisData_Curr into MisData_Hist (overwrite if exists)
+  4. DELETE BackLevel rows from MisData_Curr
+  5. Rebuild index
+
+#### 3.2 MisData_Hist Pipeline (Mass Refresh Only)
+
+- **Daily schedule**: Disabled (populated via MisData_Curr postScripts)
+- **Mass schedule**: Full purge/reload of BackLevel data
+- **Hourly**: Disabled
+- **PostScripts**: ObsoleteDate updates + index rebuild
+
+#### 3.3 Data Flow
+
+```
+Daily Sync:
+  CMS (Current status) → MERGE → MisData_Curr
+                                    ↓
+                         [postScripts move BackLevel rows]
+                                    ↓
+                               MisData_Hist
+
+Mass Sync:
+  CMS (Current) → PURGE/RELOAD → MisData_Curr
+  CMS (BackLevel) → PURGE/RELOAD → MisData_Hist
+```
+
+### Phase 4: DEV Pipeline Updates
+
+**File**: `src/JdeScoping.DataSync.Dev/Pipelines/dev-pipelines.json`
+
+#### 4.1 Update Size Categories
+```json
+"sizeCategories": {
+    "large": ["Lot", "MisData_Hist", "WorkOrder_Curr", ...],
+    "medium": [..., "MisData_Curr"]  // Current status rows are fewer
+}
+```
+
+#### 4.2 Split Pipeline Entry
+```json
+"MisData_Curr": {
+    "source": { "fileName": "misdata_curr.pb.zstd" },
+    "destination": { "table": "MisData_Curr" }
+},
+"MisData_Hist": {
+    "source": { "fileName": "misdata_hist.pb.zstd" },
+    "destination": { "table": "MisData_Hist" }
+}
+```
+
+#### 4.3 Data File Split
+- Rename existing `misdata.pb.zstd` → `misdata_hist.pb.zstd`
+- Create `misdata_curr.pb.zstd` with Current status rows extracted
+- **Note**: This requires processing the existing file to split by Status
+
+### Phase 5: Unit Test Updates
+
+#### 5.1 DevEtlPipelineFactoryTests
+- Update tests expecting `MisData` pipeline → expect `MisData_Curr` and `MisData_Hist`
+- Add tests for size category assignments
+
+#### 5.2 DevEtlRegistryTests
+- Update registry enumeration tests for new pipeline names
+
+### Phase 6: Documentation Updates
+
+Update data sync documentation to reflect:
+- MisData split into _Curr (Current status) and _Hist (BackLevel status)
+- View combining both tables
+- Pipeline configuration changes
+
+## Verification Checklist
+
+- [ ] Database project builds without errors
+- [ ] Local database has MisData_Curr, MisData_Hist tables and MisData view
+- [ ] fn_MatchMIS function works with the view
+- [ ] Production pipeline JSON is valid
+- [ ] DEV pipeline JSON is valid
+- [ ] All unit tests pass
+- [ ] Solution builds successfully
+
+## Rollback Plan
+
+If issues arise:
+1. Drop view: `DROP VIEW dbo.MisData`
+2. Rename table back: `EXEC sp_rename 'dbo.MisData_Hist', 'MisData'`
+3. Rename constraint: `EXEC sp_rename 'dbo.PK_MisData_Hist', 'PK_MisData'`
+4. Drop _Curr table: `DROP TABLE dbo.MisData_Curr`
+5. Revert pipeline JSON changes
@@ -0,0 +1,162 @@
+# Unit Test Report
+
+**Generated:** 2026-01-19
+**Framework:** xUnit 2.9.3 on .NET 10
+**Status:** ✅ All Tests Passing
+
+## Summary
+
+| Metric | Value |
+|--------|-------|
+| Total Test Projects | 11 |
+| Projects with Tests | 10 |
+| Total Tests | 856 |
+| Passed | 856 |
+| Failed | 0 |
+| Skipped | 0 |
+| Pass Rate | 100% |
+
+## Test Results by Project
+
+| Project | Status | Tests | Passed | Failed | Skipped | Time |
+|---------|--------|-------|--------|--------|---------|------|
+| JdeScoping.Api.Tests | ✅ Pass | 37 | 37 | 0 | 0 | 2.12s |
+| JdeScoping.Api.IntegrationTests | ✅ Pass | 24 | 24 | 0 | 0 | 1.24s |
+| JdeScoping.Client.Tests | ✅ Pass | 73 | 73 | 0 | 0 | 2.07s |
+| JdeScoping.Core.Tests | ✅ Pass | 77 | 77 | 0 | 0 | 1.62s |
+| JdeScoping.DataAccess.Tests | ✅ Pass | 67 | 67 | 0 | 0 | 1.92s |
+| JdeScoping.Database.Tests | ✅ Pass | 82 | 82 | 0 | 0 | 1.66s |
+| JdeScoping.DataSync.Tests | ✅ Pass | 340 | 340 | 0 | 0 | 3.73s |
+| JdeScoping.DataSync.Dev.Tests | ✅ Pass | 15 | 15 | 0 | 0 | 1.78s |
+| JdeScoping.ExcelIO.Tests | ✅ Pass | 122 | 122 | 0 | 0 | 4:10s |
+| JdeScoping.Host.Tests | ⚠️ No Tests | 0 | 0 | 0 | 0 | 1.45s |
+| JdeScoping.Infrastructure.Tests | ✅ Pass | 19 | 19 | 0 | 0 | 1.99s |
+
+## Project Details
+
+### JdeScoping.Api.Tests (37 tests)
+Tests for API controllers, SignalR hubs, and services.
+
+**Test Classes:**
+- `AuthControllerTests` - Authentication endpoints
+- `SearchControllerTests` - Search CRUD operations
+- `LookupControllerTests` - Autocomplete/lookup APIs
+- `StatusHubTests` - SignalR real-time updates
+- `CurrentUserServiceTests` - User context service
+
+### JdeScoping.Api.IntegrationTests (24 tests)
+End-to-end tests using WebApplicationFactory.
+
+**Test Classes:**
+- `AuthApiClientIntegrationTests` - Auth flow testing
+- `LookupApiClientIntegrationTests` - Lookup endpoint testing
+- `SearchApiClientIntegrationTests` - Search endpoint testing
+
+### JdeScoping.Client.Tests (73 tests)
+Blazor client service tests.
+
+**Test Classes:**
+- `ApiClientBaseTests` - Base HTTP client behavior
+- `AuthApiClientTests` - Auth client operations
+- `LookupApiClientTests` - Lookup client operations
+- `SearchApiClientTests` - Search client operations
+- `SearchStateTests` - Client-side state management
+
+### JdeScoping.Core.Tests (77 tests)
+Core domain model tests.
+
+**Test Classes:**
+- `SearchCriteriaTests` - Search criteria model validation
+- `QueryTypesTests` - Query type identification logic
+
+### JdeScoping.DataAccess.Tests (67 tests)
+Data access layer tests.
+
+**Test Classes:**
+- `DbConnectionFactoryTests` - Connection management
+- `SearchRepositoryTests` - Search persistence
+- `LookupRepositoryTests` - Reference data queries
+- `DataUpdateRepositoryTests` - Cache timestamps
+
+### JdeScoping.Database.Tests (82 tests)
+SQL Server function and procedure tests (requires Docker container).
+
+**Test Classes:**
+- `SimpleTableFunctionTests` - TVFs for criteria extraction
+- `ScalarFunctionTests` - Scalar criteria functions
+- `ComplexTableFunctionTests` - Component/operation TVFs
+- `ValidateSearchCriteriaProcedureTests` - Validation stored procedure
+
+### JdeScoping.DataSync.Tests (340 tests)
+ETL pipeline and transformer tests.
+
+**Coverage Areas:**
+- Source data transformers (JDE, CMS)
+- Cache table transformers
+- Work processor logic
+- Pipeline configuration
+- ETL registry and factory
+
+### JdeScoping.DataSync.Dev.Tests (15 tests)
+Development pipeline tests.
+
+**Test Classes:**
+- `DevEtlRegistryTests` - Dev ETL configuration
+- `DevPipelineTests` - Pipeline execution
+
+### JdeScoping.ExcelIO.Tests (122 tests)
+Excel export functionality tests.
+
+**Test Classes:**
+- `ExcelExportServiceTests` - Full export workflow
+- `ExcelPackageBuilderTests` - Package construction
+- `WorksheetBuilderTests` - Individual sheet building
+- Various formatter tests for data types
+
+**Note:** Longer execution time (4:10) due to actual Excel file generation in tests.
+
+### JdeScoping.Host.Tests (0 tests)
+Placeholder project with no tests implemented yet.
+
+**File:** `Placeholder.cs` - Contains only a placeholder class with comments indicating tests will be added.
+
+### JdeScoping.Infrastructure.Tests (19 tests)
+Infrastructure utility tests.
+
+**Test Classes:**
+- `RsaKeyServiceTests` - RSA key generation/persistence
+- `EncryptionHelperTests` - Password encryption
+- `ConfigExtensionsTests` - Configuration helpers
+
+## Execution Strategy
+
+Tests were executed in two phases:
+
+### Phase 1: Parallel Execution (9 projects)
+Projects with no shared state ran concurrently:
+- Api.Tests, Client.Tests, Core.Tests, DataAccess.Tests
+- DataSync.Tests, DataSync.Dev.Tests, ExcelIO.Tests
+- Host.Tests, Infrastructure.Tests
+
+### Phase 2: Sequential Execution (2 projects)
+Projects with external dependencies ran sequentially:
+1. **Database.Tests** - Requires Docker SQL Server (`scopingtool-sqlserver` on port 1434)
+2. **Api.IntegrationTests** - Uses WebApplicationFactory with database connection
+
+## Build Warnings
+
+Two warnings were noted during build (non-blocking):
+
+1. `StatusHubTests.cs(107,59)`: CS0252 - Possible unintended reference comparison
+2. Static file middleware warning - WebRootPath not found (expected for API-only tests)
+
+## Failures and Resolutions
+
+**No failures encountered.** All 856 tests passed successfully.
+
+## Notes
+
+- All tests use xUnit 2.9.3 with .NET 10
+- Database.Tests require the Docker SQL Server container to be running
+- ExcelIO.Tests have longer execution due to actual Excel file I/O operations
+- Host.Tests project exists but has no implemented tests yet (placeholder)