docs: add MisData archival implementation and unit test report
Document the MisData_Curr/Hist table split in architecture docs, add post-sync archival scripts to move BackLevel records to history table, and generate comprehensive unit test report (856 tests, 100% pass rate).
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,166 @@
|
||||
# MisData Archival Implementation Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Split the MisData table into `MisData_Curr` and `MisData_Hist` following the existing archival pattern used for WorkOrder, LotUsage, WorkOrderComponent, WorkOrderStep, and WorkOrderTime tables.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| Partitioning Strategy | Status-based | `Status='Current'` → MisData_Curr, `Status='BackLevel'` → MisData_Hist |
|
||||
| DEV Data Files | Split existing file | Create misdata_curr.pb.zstd (Current rows), misdata_hist.pb.zstd (BackLevel rows) |
|
||||
| PostScripts | Run on both tables | ObsoleteDate update logic applies to both Current and BackLevel records |
|
||||
| View Name | Keep `MisData` | Maintains backward compatibility with fn_MatchMIS function |
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Database Project Updates
|
||||
|
||||
#### 1.1 Create MisData_Hist Table Script
|
||||
- **File**: `Scripts/012a_CreateMisDataHistTable.sql`
|
||||
- **Action**: Create new script for MisData_Hist (identical schema to current MisData)
|
||||
- **PK**: Rename to `PK_MisData_Hist`
|
||||
|
||||
#### 1.2 Create MisData_Curr Table Script
|
||||
- **File**: `Scripts/012b_CreateMisDataCurrTable.sql`
|
||||
- **Action**: Create new script for MisData_Curr (identical schema)
|
||||
- **PK**: `PK_MisData_Curr`
|
||||
|
||||
#### 1.3 Create MisData View Script
|
||||
- **File**: `Scripts/033_CreateMisDataView.sql`
|
||||
- **Action**: Create UNION ALL view combining both tables
|
||||
- **Pattern**: Follow 030_CreateLotUsageView.sql pattern
|
||||
|
||||
#### 1.4 Update Original Script (for new installs)
|
||||
- **File**: `Scripts/012_CreateMisDataTable.sql`
|
||||
- **Action**: Rename to create MisData_Hist instead
|
||||
|
||||
### Phase 2: Live Database Migration
|
||||
|
||||
Execute directly against local database (no migrations):
|
||||
|
||||
```sql
|
||||
-- Step 1: Rename existing table and constraint
|
||||
EXEC sp_rename 'dbo.MisData', 'MisData_Hist';
|
||||
EXEC sp_rename 'dbo.PK_MisData', 'PK_MisData_Hist';
|
||||
|
||||
-- Step 2: Create MisData_Curr with identical schema
|
||||
CREATE TABLE [dbo].[MisData_Curr] (
|
||||
-- identical columns
|
||||
CONSTRAINT [PK_MisData_Curr] PRIMARY KEY CLUSTERED(...)
|
||||
);
|
||||
|
||||
-- Step 3: Move Current status rows to _Curr table
|
||||
INSERT INTO dbo.MisData_Curr SELECT * FROM dbo.MisData_Hist WHERE Status = 'Current';
|
||||
DELETE FROM dbo.MisData_Hist WHERE Status = 'Current';
|
||||
|
||||
-- Step 4: Create unified view
|
||||
CREATE OR ALTER VIEW [dbo].[MisData] AS (
|
||||
SELECT hist.* FROM dbo.MisData_Hist hist
|
||||
UNION ALL
|
||||
SELECT curr.* FROM dbo.MisData_Curr curr
|
||||
);
|
||||
```
|
||||
|
||||
### Phase 3: Production Pipeline Updates
|
||||
|
||||
**File**: `src/JdeScoping.DataSync/Pipelines/pipelines.json`
|
||||
|
||||
#### 3.1 MisData_Curr Pipeline (Daily Incremental Sync)
|
||||
|
||||
- **Daily schedule**: Merge into MisData_Curr (no purge/reload)
|
||||
- **Mass schedule**: Full purge/reload
|
||||
- **Hourly**: Disabled
|
||||
- **PostScripts** (in order):
|
||||
1. Update ObsoleteDate for Current records where a BackLevel exists
|
||||
2. Update ObsoleteDate for remaining NULL values
|
||||
3. MERGE BackLevel rows from MisData_Curr into MisData_Hist (overwrite if exists)
|
||||
4. DELETE BackLevel rows from MisData_Curr
|
||||
5. Rebuild index
|
||||
|
||||
#### 3.2 MisData_Hist Pipeline (Mass Refresh Only)
|
||||
|
||||
- **Daily schedule**: Disabled (populated via MisData_Curr postScripts)
|
||||
- **Mass schedule**: Full purge/reload of BackLevel data
|
||||
- **Hourly**: Disabled
|
||||
- **PostScripts**: ObsoleteDate updates + index rebuild
|
||||
|
||||
#### 3.3 Data Flow
|
||||
|
||||
```
|
||||
Daily Sync:
|
||||
CMS (Current status) → MERGE → MisData_Curr
|
||||
↓
|
||||
[postScripts move BackLevel rows]
|
||||
↓
|
||||
MisData_Hist
|
||||
|
||||
Mass Sync:
|
||||
CMS (Current) → PURGE/RELOAD → MisData_Curr
|
||||
CMS (BackLevel) → PURGE/RELOAD → MisData_Hist
|
||||
```
|
||||
|
||||
### Phase 4: DEV Pipeline Updates
|
||||
|
||||
**File**: `src/JdeScoping.DataSync.Dev/Pipelines/dev-pipelines.json`
|
||||
|
||||
#### 4.1 Update Size Categories
|
||||
```json
|
||||
"sizeCategories": {
|
||||
"large": ["Lot", "MisData_Hist", "WorkOrder_Curr", ...],
|
||||
"medium": [..., "MisData_Curr"] // Current status rows are fewer
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.2 Split Pipeline Entry
|
||||
```json
|
||||
"MisData_Curr": {
|
||||
"source": { "fileName": "misdata_curr.pb.zstd" },
|
||||
"destination": { "table": "MisData_Curr" }
|
||||
},
|
||||
"MisData_Hist": {
|
||||
"source": { "fileName": "misdata_hist.pb.zstd" },
|
||||
"destination": { "table": "MisData_Hist" }
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.3 Data File Split
|
||||
- Rename existing `misdata.pb.zstd` → `misdata_hist.pb.zstd`
|
||||
- Create `misdata_curr.pb.zstd` with Current status rows extracted
|
||||
- **Note**: This requires processing the existing file to split by Status
|
||||
|
||||
### Phase 5: Unit Test Updates
|
||||
|
||||
#### 5.1 DevEtlPipelineFactoryTests
|
||||
- Update tests expecting `MisData` pipeline → expect `MisData_Curr` and `MisData_Hist`
|
||||
- Add tests for size category assignments
|
||||
|
||||
#### 5.2 DevEtlRegistryTests
|
||||
- Update registry enumeration tests for new pipeline names
|
||||
|
||||
### Phase 6: Documentation Updates
|
||||
|
||||
Update data sync documentation to reflect:
|
||||
- MisData split into _Curr (Current status) and _Hist (BackLevel status)
|
||||
- View combining both tables
|
||||
- Pipeline configuration changes
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
- [ ] Database project builds without errors
|
||||
- [ ] Local database has MisData_Curr, MisData_Hist tables and MisData view
|
||||
- [ ] fn_MatchMIS function works with the view
|
||||
- [ ] Production pipeline JSON is valid
|
||||
- [ ] DEV pipeline JSON is valid
|
||||
- [ ] All unit tests pass
|
||||
- [ ] Solution builds successfully
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Drop view: `DROP VIEW dbo.MisData`
|
||||
2. Rename table back: `EXEC sp_rename 'dbo.MisData_Hist', 'MisData'`
|
||||
3. Rename constraint: `EXEC sp_rename 'dbo.PK_MisData_Hist', 'PK_MisData'`
|
||||
4. Drop _Curr table: `DROP TABLE dbo.MisData_Curr`
|
||||
5. Revert pipeline JSON changes
|
||||
@@ -0,0 +1,162 @@
|
||||
# Unit Test Report
|
||||
|
||||
**Generated:** 2026-01-19
|
||||
**Framework:** xUnit 2.9.3 on .NET 10
|
||||
**Status:** ✅ All Tests Passing
|
||||
|
||||
## Summary
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total Test Projects | 11 |
|
||||
| Projects with Tests | 10 |
|
||||
| Total Tests | 856 |
|
||||
| Passed | 856 |
|
||||
| Failed | 0 |
|
||||
| Skipped | 0 |
|
||||
| Pass Rate | 100% |
|
||||
|
||||
## Test Results by Project
|
||||
|
||||
| Project | Status | Tests | Passed | Failed | Skipped | Time |
|
||||
|---------|--------|-------|--------|--------|---------|------|
|
||||
| JdeScoping.Api.Tests | ✅ Pass | 37 | 37 | 0 | 0 | 2.12s |
|
||||
| JdeScoping.Api.IntegrationTests | ✅ Pass | 24 | 24 | 0 | 0 | 1.24s |
|
||||
| JdeScoping.Client.Tests | ✅ Pass | 73 | 73 | 0 | 0 | 2.07s |
|
||||
| JdeScoping.Core.Tests | ✅ Pass | 77 | 77 | 0 | 0 | 1.62s |
|
||||
| JdeScoping.DataAccess.Tests | ✅ Pass | 67 | 67 | 0 | 0 | 1.92s |
|
||||
| JdeScoping.Database.Tests | ✅ Pass | 82 | 82 | 0 | 0 | 1.66s |
|
||||
| JdeScoping.DataSync.Tests | ✅ Pass | 340 | 340 | 0 | 0 | 3.73s |
|
||||
| JdeScoping.DataSync.Dev.Tests | ✅ Pass | 15 | 15 | 0 | 0 | 1.78s |
|
||||
| JdeScoping.ExcelIO.Tests | ✅ Pass | 122 | 122 | 0 | 0 | 4:10s |
|
||||
| JdeScoping.Host.Tests | ⚠️ No Tests | 0 | 0 | 0 | 0 | 1.45s |
|
||||
| JdeScoping.Infrastructure.Tests | ✅ Pass | 19 | 19 | 0 | 0 | 1.99s |
|
||||
|
||||
## Project Details
|
||||
|
||||
### JdeScoping.Api.Tests (37 tests)
|
||||
Tests for API controllers, SignalR hubs, and services.
|
||||
|
||||
**Test Classes:**
|
||||
- `AuthControllerTests` - Authentication endpoints
|
||||
- `SearchControllerTests` - Search CRUD operations
|
||||
- `LookupControllerTests` - Autocomplete/lookup APIs
|
||||
- `StatusHubTests` - SignalR real-time updates
|
||||
- `CurrentUserServiceTests` - User context service
|
||||
|
||||
### JdeScoping.Api.IntegrationTests (24 tests)
|
||||
End-to-end tests using WebApplicationFactory.
|
||||
|
||||
**Test Classes:**
|
||||
- `AuthApiClientIntegrationTests` - Auth flow testing
|
||||
- `LookupApiClientIntegrationTests` - Lookup endpoint testing
|
||||
- `SearchApiClientIntegrationTests` - Search endpoint testing
|
||||
|
||||
### JdeScoping.Client.Tests (73 tests)
|
||||
Blazor client service tests.
|
||||
|
||||
**Test Classes:**
|
||||
- `ApiClientBaseTests` - Base HTTP client behavior
|
||||
- `AuthApiClientTests` - Auth client operations
|
||||
- `LookupApiClientTests` - Lookup client operations
|
||||
- `SearchApiClientTests` - Search client operations
|
||||
- `SearchStateTests` - Client-side state management
|
||||
|
||||
### JdeScoping.Core.Tests (77 tests)
|
||||
Core domain model tests.
|
||||
|
||||
**Test Classes:**
|
||||
- `SearchCriteriaTests` - Search criteria model validation
|
||||
- `QueryTypesTests` - Query type identification logic
|
||||
|
||||
### JdeScoping.DataAccess.Tests (67 tests)
|
||||
Data access layer tests.
|
||||
|
||||
**Test Classes:**
|
||||
- `DbConnectionFactoryTests` - Connection management
|
||||
- `SearchRepositoryTests` - Search persistence
|
||||
- `LookupRepositoryTests` - Reference data queries
|
||||
- `DataUpdateRepositoryTests` - Cache timestamps
|
||||
|
||||
### JdeScoping.Database.Tests (82 tests)
|
||||
SQL Server function and procedure tests (requires Docker container).
|
||||
|
||||
**Test Classes:**
|
||||
- `SimpleTableFunctionTests` - TVFs for criteria extraction
|
||||
- `ScalarFunctionTests` - Scalar criteria functions
|
||||
- `ComplexTableFunctionTests` - Component/operation TVFs
|
||||
- `ValidateSearchCriteriaProcedureTests` - Validation stored procedure
|
||||
|
||||
### JdeScoping.DataSync.Tests (340 tests)
|
||||
ETL pipeline and transformer tests.
|
||||
|
||||
**Coverage Areas:**
|
||||
- Source data transformers (JDE, CMS)
|
||||
- Cache table transformers
|
||||
- Work processor logic
|
||||
- Pipeline configuration
|
||||
- ETL registry and factory
|
||||
|
||||
### JdeScoping.DataSync.Dev.Tests (15 tests)
|
||||
Development pipeline tests.
|
||||
|
||||
**Test Classes:**
|
||||
- `DevEtlRegistryTests` - Dev ETL configuration
|
||||
- `DevPipelineTests` - Pipeline execution
|
||||
|
||||
### JdeScoping.ExcelIO.Tests (122 tests)
|
||||
Excel export functionality tests.
|
||||
|
||||
**Test Classes:**
|
||||
- `ExcelExportServiceTests` - Full export workflow
|
||||
- `ExcelPackageBuilderTests` - Package construction
|
||||
- `WorksheetBuilderTests` - Individual sheet building
|
||||
- Various formatter tests for data types
|
||||
|
||||
**Note:** Longer execution time (4:10) due to actual Excel file generation in tests.
|
||||
|
||||
### JdeScoping.Host.Tests (0 tests)
|
||||
Placeholder project with no tests implemented yet.
|
||||
|
||||
**File:** `Placeholder.cs` - Contains only a placeholder class with comments indicating tests will be added.
|
||||
|
||||
### JdeScoping.Infrastructure.Tests (19 tests)
|
||||
Infrastructure utility tests.
|
||||
|
||||
**Test Classes:**
|
||||
- `RsaKeyServiceTests` - RSA key generation/persistence
|
||||
- `EncryptionHelperTests` - Password encryption
|
||||
- `ConfigExtensionsTests` - Configuration helpers
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
Tests were executed in two phases:
|
||||
|
||||
### Phase 1: Parallel Execution (9 projects)
|
||||
Projects with no shared state ran concurrently:
|
||||
- Api.Tests, Client.Tests, Core.Tests, DataAccess.Tests
|
||||
- DataSync.Tests, DataSync.Dev.Tests, ExcelIO.Tests
|
||||
- Host.Tests, Infrastructure.Tests
|
||||
|
||||
### Phase 2: Sequential Execution (2 projects)
|
||||
Projects with external dependencies ran sequentially:
|
||||
1. **Database.Tests** - Requires Docker SQL Server (`scopingtool-sqlserver` on port 1434)
|
||||
2. **Api.IntegrationTests** - Uses WebApplicationFactory with database connection
|
||||
|
||||
## Build Warnings
|
||||
|
||||
Two warnings were noted during build (non-blocking):
|
||||
|
||||
1. `StatusHubTests.cs(107,59)`: CS0252 - Possible unintended reference comparison
|
||||
2. Static file middleware warning - WebRootPath not found (expected for API-only tests)
|
||||
|
||||
## Failures and Resolutions
|
||||
|
||||
**No failures encountered.** All 856 tests passed successfully.
|
||||
|
||||
## Notes
|
||||
|
||||
- All tests use xUnit 2.9.3 with .NET 10
|
||||
- Database.Tests require the Docker SQL Server container to be running
|
||||
- ExcelIO.Tests have longer execution due to actual Excel file I/O operations
|
||||
- Host.Tests project exists but has no implemented tests yet (placeholder)
|
||||
Reference in New Issue
Block a user