From 5040a8488f526c7675ab248974f8f6e280daf8df Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 7 Jan 2026 07:40:49 -0500 Subject: [PATCH] docs: add ETL pipeline viewer component design Design document for admin monitoring component that visualizes: - Pipeline configuration from pipelines.json - Execution status (last run, next required, overdue) - Recent execution history (last 10 per schedule type) - Schedule sections for Mass/Daily/Hourly with defaults vs overrides --- .../2026-01-07-pipeline-viewer-design.md | 319 ++++++++++++++++++ 1 file changed, 319 insertions(+) create mode 100644 NEW/docs/plans/2026-01-07-pipeline-viewer-design.md diff --git a/NEW/docs/plans/2026-01-07-pipeline-viewer-design.md b/NEW/docs/plans/2026-01-07-pipeline-viewer-design.md new file mode 100644 index 0000000..3f99460 --- /dev/null +++ b/NEW/docs/plans/2026-01-07-pipeline-viewer-design.md @@ -0,0 +1,319 @@ +# ETL Pipeline Viewer Component Design + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status. + +**Architecture:** Single-page admin component with pipeline selector, execution status tables, and vertical configuration sections for each schedule type (Mass, Daily, Hourly). + +**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataUpdate repository + +--- + +## Overview + +A read-only admin monitoring page that displays: +1. Pipeline configuration details from `pipelines.json` +2. Execution status (last run, next required, overdue status) +3. Recent execution history (last 10 runs per schedule type) + +## Component Structure + +### Page Layout + +``` +┌─────────────────────────────────────────────────────────────┐ +│ ETL Pipeline Configuration Viewer │ +├─────────────────────────────────────────────────────────────┤ +│ Pipeline: [Dropdown - Alphabetical list] │ +├─────────────────────────────────────────────────────────────┤ +│ SCHEDULE STATUS SUMMARY (Table 1) │ +│ ┌─────────┬────────────────────┬─────────────────┬────────┐ │ +│ │ Type │ Last Successful │ Next Required │ Status │ │ +│ ├─────────┼────────────────────┼─────────────────┼────────┤ │ +│ │ Mass │ 2026-01-05 02:00 │ 2026-01-12 02:00│ ✓ OK │ │ +│ │ Daily │ 2026-01-07 01:00 │ 2026-01-08 01:00│ ✓ OK │ │ +│ │ Hourly │ 2026-01-07 10:00 │ 2026-01-07 11:00│ ⚠ Over │ │ +│ └─────────┴────────────────────┴─────────────────┴────────┘ │ +├─────────────────────────────────────────────────────────────┤ +│ RECENT EXECUTION HISTORY (Table 2 - Last 10 per type) │ +│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │ +│ │ Type │ Start │ End │ Duration│ Rows│ Result│ │ +│ ├─────────┼────────────┼──────────┼─────────┼─────┼───────┤ │ +│ │ Hourly │ 10:00 AM │ 10:02 AM │ 2m 15s │1,247│ ✓ │ │ +│ │ Hourly │ 09:00 AM │ 09:01 AM │ 1m 42s │ 892 │ ✓ │ │ +│ │ ... │ │ │ │ │ │ │ +│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │ +├─────────────────────────────────────────────────────────────┤ +│ COMMON PIPELINE INFO │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Source: [JDE] Connection │ │ +│ │ Parameters: dateUpdated (jdeJulian), timeUpdated (jdeTime)│ +│ └─────────────────────────────────────────────────────────┘ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Destination: WorkOrder_Curr │ │ +│ │ Operation: Bulk Merge │ │ +│ │ Match: WorkOrderNumber, BranchCode │ │ +│ │ Exclude: WorkOrderNumber, BranchCode, LastUpdateDt │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Pre-Scripts: 0 scripts │ │ +│ │ Post-Scripts: 3 scripts [View] │ │ +│ └─────────────────────────────────────────────────────────┘ │ +├─────────────────────────────────────────────────────────────┤ +│ MASS REFRESH [Enabled] │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Schedule Settings │ │ +│ │ Interval: 10080 min (7 days) [Default] │ │ +│ │ Pre-Purge: Yes [Default] │ │ +│ │ Re-Index: Yes [Default] │ │ +│ ├─────────────────────────────────────────────────────────┤ │ +│ │ Query: SELECT wo.WADOCO AS Work... [View Full Query] │ │ +│ └─────────────────────────────────────────────────────────┘ │ +├─────────────────────────────────────────────────────────────┤ +│ DAILY REFRESH [Enabled] │ +│ └─ (Same structure as Mass) │ +├─────────────────────────────────────────────────────────────┤ +│ HOURLY REFRESH [Enabled] │ +│ └─ (Same structure as Mass) │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Files to Create + +### Client Project (`src/JdeScoping.Client/`) + +| File | Description | +|------|-------------| +| `Pages/Admin/PipelineViewer.razor` | Main page component | +| `Components/Admin/PipelineScheduleSection.razor` | Reusable schedule section (Mass/Daily/Hourly) | +| `Components/Admin/SqlQueryModal.razor` | Modal for displaying SQL queries/scripts | +| `Services/IPipelineConfigService.cs` | Interface for pipeline config access | +| `Services/PipelineConfigService.cs` | Implementation - loads pipelines.json | +| `Services/IPipelineStatusService.cs` | Interface for execution status | +| `Services/PipelineStatusService.cs` | Implementation - calls API for status | +| `Models/PipelineScheduleStatus.cs` | View model for schedule status row | +| `Models/PipelineExecution.cs` | View model for execution history row | + +### API Project (`src/JdeScoping.Api/`) + +| File | Description | +|------|-------------| +| `Controllers/PipelineController.cs` | API endpoints for pipeline config and status | + +### DataSync Project (`src/JdeScoping.DataSync/`) + +| File | Description | +|------|-------------| +| `Contracts/IDataUpdateRepository.cs` | Add `GetRecentUpdatesAsync` method | +| `Services/DataUpdateRepository.cs` | Implement new method | + +## Data Models + +### View Models (Client) + +```csharp +public record PipelineScheduleStatus( + UpdateTypes ScheduleType, + DateTime? LastSuccessfulRun, + DateTime? NextRequiredRun, + bool IsOverdue, + int IntervalMinutes); + +public record PipelineExecution( + UpdateTypes ScheduleType, + DateTime StartTime, + DateTime? EndTime, + TimeSpan Duration, + long RecordCount, + bool WasSuccessful); +``` + +### Config Models (reuse or create in Contracts) + +```csharp +public record PipelinesRoot( + PipelineSettings? Settings, + ScheduleDefaults? ScheduleDefaults, + Dictionary Pipelines); + +public record ScheduleDefaults( + ScheduleConfig Mass, + ScheduleConfig Daily, + ScheduleConfig Hourly); + +public record PipelineConfig( + SourceConfig Source, + DestinationConfig Destination, + PipelineSchedules? Schedules, + List? PreScripts, + List? PostScripts); + +public record SourceConfig( + string Connection, + string Query, + string? MassQuery, + Dictionary? Parameters); + +public record ParameterConfig( + string Name, + string? Format, + string? Source); + +public record DestinationConfig( + string Table, + string[]? MatchColumns, + string[]? ExcludeFromUpdate); + +public record PipelineSchedules( + ScheduleConfig? Mass, + ScheduleConfig? Daily, + ScheduleConfig? Hourly); + +public record ScheduleConfig( + bool? Enabled, + int? IntervalMinutes, + bool? PrePurge, + bool? ReIndex); +``` + +## Service Interfaces + +### IPipelineConfigService + +```csharp +public interface IPipelineConfigService +{ + /// + /// Gets the full pipelines configuration. + /// + Task GetPipelinesConfigAsync(); + + /// + /// Gets configuration for a specific pipeline. + /// + Task GetPipelineAsync(string pipelineName); + + /// + /// Gets list of all pipeline names (sorted alphabetically). + /// + IEnumerable GetPipelineNames(); + + /// + /// Gets schedule defaults for computing effective values. + /// + ScheduleDefaults GetScheduleDefaults(); +} +``` + +### IPipelineStatusService + +```csharp +public interface IPipelineStatusService +{ + /// + /// Gets schedule status summary for a pipeline (one row per schedule type). + /// + Task> GetScheduleStatusAsync( + string pipelineName, + CancellationToken cancellationToken = default); + + /// + /// Gets recent execution history for a pipeline. + /// + Task> GetRecentExecutionsAsync( + string pipelineName, + int count = 10, + CancellationToken cancellationToken = default); +} +``` + +### IDataUpdateRepository (addition) + +```csharp +/// +/// Gets the last N execution records for a specific table. +/// +Task> GetRecentUpdatesAsync( + string tableName, + int count = 10, + CancellationToken cancellationToken = default); +``` + +## SQL Query Modal Component + +### Features +- Large modal (80% viewport width) +- Monospace font display (`
`)
+- Basic SQL formatting (line breaks at clauses)
+- Copy to Clipboard button
+- Dynamic title based on context
+
+### Usage
+```razor
+
+```
+
+## Schedule Section Component
+
+### Features
+- Header with schedule type name and Enabled/Disabled badge
+- Settings table showing:
+  - Interval (with "Default" or "Override" indicator)
+  - Pre-Purge flag
+  - Re-Index flag
+- Query preview (truncated to ~100 chars)
+- "View Full Query" button
+
+### Props
+```csharp
+[Parameter] public string ScheduleType { get; set; }  // "Mass", "Daily", "Hourly"
+[Parameter] public ScheduleConfig? Config { get; set; }
+[Parameter] public ScheduleConfig DefaultConfig { get; set; }
+[Parameter] public string? Query { get; set; }
+[Parameter] public EventCallback OnViewQuery { get; set; }
+```
+
+## API Endpoints
+
+| Method | Route | Description |
+|--------|-------|-------------|
+| GET | `/api/pipelines` | Get all pipeline names |
+| GET | `/api/pipelines/{name}` | Get pipeline configuration |
+| GET | `/api/pipelines/{name}/status` | Get schedule status summary |
+| GET | `/api/pipelines/{name}/executions?count=10` | Get recent executions |
+
+## Implementation Notes
+
+### Override Detection
+To show "Default" vs "Override" for schedule settings:
+```csharp
+bool IsOverride(int? pipelineValue, int defaultValue) =>
+    pipelineValue.HasValue && pipelineValue.Value != defaultValue;
+```
+
+### Next Required Calculation
+```csharp
+DateTime? nextRequired = lastSuccessful?.AddMinutes(intervalMinutes);
+bool isOverdue = nextRequired.HasValue && nextRequired.Value < DateTime.UtcNow;
+```
+
+### Query Truncation
+```csharp
+string Truncate(string sql, int maxLength = 100) =>
+    sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
+```
+
+### Connection Type Badge Colors
+- JDE: Blue (`BadgeStyle.Info`)
+- CMS: Green (`BadgeStyle.Success`)
+- GIW: Orange (`BadgeStyle.Warning`)
+
+## Dependencies
+
+- Radzen.Blazor (already installed)
+- Existing `IDataUpdateRepository`
+- Existing `pipelines.json` configuration
+- Existing `UpdateTypes` enum (Hourly, Daily, Mass)