# ETL Pipeline Viewer Component Design

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status.

**Architecture:** Single-page admin component with pipeline selector, execution status tables, and vertical configuration sections for each schedule type (Mass, Daily, Hourly).

**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataUpdate repository

---

## Overview

A read-only admin monitoring page that displays:
1. Pipeline configuration details from `pipelines.json`
2. Execution status (last run, next required, overdue status)
3. Recent execution history (last 10 runs per schedule type)

## Component Structure

### Page Layout

```
┌─────────────────────────────────────────────────────────────┐
│ ETL Pipeline Configuration Viewer                           │
├─────────────────────────────────────────────────────────────┤
│ Pipeline: [Dropdown - Alphabetical list]                    │
├─────────────────────────────────────────────────────────────┤
│ SCHEDULE STATUS SUMMARY (Table 1)                           │
│ ┌─────────┬────────────────────┬─────────────────┬────────┐ │
│ │ Type    │ Last Successful    │ Next Required   │ Status │ │
│ ├─────────┼────────────────────┼─────────────────┼────────┤ │
│ │ Mass    │ 2026-01-05 02:00   │ 2026-01-12 02:00│ ✓ OK   │ │
│ │ Daily   │ 2026-01-07 01:00   │ 2026-01-08 01:00│ ✓ OK   │ │
│ │ Hourly  │ 2026-01-07 10:00   │ 2026-01-07 11:00│ ⚠ Over │ │
│ └─────────┴────────────────────┴─────────────────┴────────┘ │
├─────────────────────────────────────────────────────────────┤
│ RECENT EXECUTION HISTORY (Table 2 - Last 10 per type)       │
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
│ │ Type    │ Start      │ End      │ Duration│ Rows│ Result│ │
│ ├─────────┼────────────┼──────────┼─────────┼─────┼───────┤ │
│ │ Hourly  │ 10:00 AM   │ 10:02 AM │ 2m 15s  │1,247│ ✓     │ │
│ │ Hourly  │ 09:00 AM   │ 09:01 AM │ 1m 42s  │ 892 │ ✓     │ │
│ │ ...     │            │          │         │     │       │ │
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ COMMON PIPELINE INFO                                         │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Source: [JDE] Connection                                │ │
│ │ Parameters: dateUpdated (jdeJulian), timeUpdated (jdeTime)│
│ └─────────────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Destination: WorkOrder_Curr                             │ │
│ │ Operation: Bulk Merge                                   │ │
│ │ Match: WorkOrderNumber, BranchCode                      │ │
│ │ Exclude: WorkOrderNumber, BranchCode, LastUpdateDt      │ │
│ └─────────────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Pre-Scripts: 0 scripts                                  │ │
│ │ Post-Scripts: 3 scripts [View]                          │ │
│ └─────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ MASS REFRESH                                    [Enabled]   │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Schedule Settings                                       │ │
│ │ Interval: 10080 min (7 days)         [Default]          │ │
│ │ Pre-Purge: Yes                       [Default]          │ │
│ │ Re-Index: Yes                        [Default]          │ │
│ ├─────────────────────────────────────────────────────────┤ │
│ │ Query: SELECT wo.WADOCO AS Work... [View Full Query]    │ │
│ └─────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ DAILY REFRESH                                   [Enabled]   │
│ └─ (Same structure as Mass)                                 │
├─────────────────────────────────────────────────────────────┤
│ HOURLY REFRESH                                  [Enabled]   │
│ └─ (Same structure as Mass)                                 │
└─────────────────────────────────────────────────────────────┘
```

## Files to Create

### Client Project (`src/JdeScoping.Client/`)

| File | Description |
|------|-------------|
| `Pages/Admin/PipelineViewer.razor` | Main page component |
| `Components/Admin/PipelineScheduleSection.razor` | Reusable schedule section (Mass/Daily/Hourly) |
| `Components/Admin/SqlQueryModal.razor` | Modal for displaying SQL queries/scripts |
| `Services/IPipelineConfigService.cs` | Interface for pipeline config access |
| `Services/PipelineConfigService.cs` | Implementation - loads pipelines.json |
| `Services/IPipelineStatusService.cs` | Interface for execution status |
| `Services/PipelineStatusService.cs` | Implementation - calls API for status |
| `Models/PipelineScheduleStatus.cs` | View model for schedule status row |
| `Models/PipelineExecution.cs` | View model for execution history row |

### API Project (`src/JdeScoping.Api/`)

| File | Description |
|------|-------------|
| `Controllers/PipelineController.cs` | API endpoints for pipeline config and status |

### DataSync Project (`src/JdeScoping.DataSync/`)

| File | Description |
|------|-------------|
| `Contracts/IDataUpdateRepository.cs` | Add `GetRecentUpdatesAsync` method |
| `Services/DataUpdateRepository.cs` | Implement new method |

## Data Models

### View Models (Client)

```csharp
public record PipelineScheduleStatus(
    UpdateTypes ScheduleType,
    DateTime? LastSuccessfulRun,
    DateTime? NextRequiredRun,
    bool IsOverdue,
    int IntervalMinutes);

public record PipelineExecution(
    UpdateTypes ScheduleType,
    DateTime StartTime,
    DateTime? EndTime,
    TimeSpan Duration,
    long RecordCount,
    bool WasSuccessful);
```

### Config Models (reuse or create in Contracts)

```csharp
public record PipelinesRoot(
    PipelineSettings? Settings,
    ScheduleDefaults? ScheduleDefaults,
    Dictionary<string, PipelineConfig> Pipelines);

public record ScheduleDefaults(
    ScheduleConfig Mass,
    ScheduleConfig Daily,
    ScheduleConfig Hourly);

public record PipelineConfig(
    SourceConfig Source,
    DestinationConfig Destination,
    PipelineSchedules? Schedules,
    List<string>? PreScripts,
    List<string>? PostScripts);

public record SourceConfig(
    string Connection,
    string Query,
    string? MassQuery,
    Dictionary<string, ParameterConfig>? Parameters);

public record ParameterConfig(
    string Name,
    string? Format,
    string? Source);

public record DestinationConfig(
    string Table,
    string[]? MatchColumns,
    string[]? ExcludeFromUpdate);

public record PipelineSchedules(
    ScheduleConfig? Mass,
    ScheduleConfig? Daily,
    ScheduleConfig? Hourly);

public record ScheduleConfig(
    bool? Enabled,
    int? IntervalMinutes,
    bool? PrePurge,
    bool? ReIndex);
```

## Service Interfaces

### IPipelineConfigService

```csharp
public interface IPipelineConfigService
{
    /// <summary>
    /// Gets the full pipelines configuration.
    /// </summary>
    Task<PipelinesRoot> GetPipelinesConfigAsync();

    /// <summary>
    /// Gets configuration for a specific pipeline.
    /// </summary>
    Task<PipelineConfig?> GetPipelineAsync(string pipelineName);

    /// <summary>
    /// Gets list of all pipeline names (sorted alphabetically).
    /// </summary>
    IEnumerable<string> GetPipelineNames();

    /// <summary>
    /// Gets schedule defaults for computing effective values.
    /// </summary>
    ScheduleDefaults GetScheduleDefaults();
}
```

### IPipelineStatusService

```csharp
public interface IPipelineStatusService
{
    /// <summary>
    /// Gets schedule status summary for a pipeline (one row per schedule type).
    /// </summary>
    Task<List<PipelineScheduleStatus>> GetScheduleStatusAsync(
        string pipelineName,
        CancellationToken cancellationToken = default);

    /// <summary>
    /// Gets recent execution history for a pipeline.
    /// </summary>
    Task<List<PipelineExecution>> GetRecentExecutionsAsync(
        string pipelineName,
        int count = 10,
        CancellationToken cancellationToken = default);
}
```

### IDataUpdateRepository (addition)

```csharp
/// <summary>
/// Gets the last N execution records for a specific table.
/// </summary>
Task<List<DataUpdate>> GetRecentUpdatesAsync(
    string tableName,
    int count = 10,
    CancellationToken cancellationToken = default);
```

## SQL Query Modal Component

### Features
- Large modal (80% viewport width)
- Monospace font display (`<pre><code>`)
- Basic SQL formatting (line breaks at clauses)
- Copy to Clipboard button
- Dynamic title based on context

### Usage
```razor
<SqlQueryModal @bind-Visible="_showModal"
               Title="Mass Query - WorkOrder_Curr"
               Sql="@_selectedSql" />
```

## Schedule Section Component

### Features
- Header with schedule type name and Enabled/Disabled badge
- Settings table showing:
  - Interval (with "Default" or "Override" indicator)
  - Pre-Purge flag
  - Re-Index flag
- Query preview (truncated to ~100 chars)
- "View Full Query" button

### Props
```csharp
[Parameter] public string ScheduleType { get; set; }  // "Mass", "Daily", "Hourly"
[Parameter] public ScheduleConfig? Config { get; set; }
[Parameter] public ScheduleConfig DefaultConfig { get; set; }
[Parameter] public string? Query { get; set; }
[Parameter] public EventCallback<string> OnViewQuery { get; set; }
```

## API Endpoints

| Method | Route | Description |
|--------|-------|-------------|
| GET | `/api/pipelines` | Get all pipeline names |
| GET | `/api/pipelines/{name}` | Get pipeline configuration |
| GET | `/api/pipelines/{name}/status` | Get schedule status summary |
| GET | `/api/pipelines/{name}/executions?count=10` | Get recent executions |

## Implementation Notes

### Override Detection
To show "Default" vs "Override" for schedule settings:
```csharp
bool IsOverride(int? pipelineValue, int defaultValue) =>
    pipelineValue.HasValue && pipelineValue.Value != defaultValue;
```

### Next Required Calculation
```csharp
DateTime? nextRequired = lastSuccessful?.AddMinutes(intervalMinutes);
bool isOverdue = nextRequired.HasValue && nextRequired.Value < DateTime.UtcNow;
```

### Query Truncation
```csharp
string Truncate(string sql, int maxLength = 100) =>
    sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
```

### Connection Type Badge Colors
- JDE: Blue (`BadgeStyle.Info`)
- CMS: Green (`BadgeStyle.Success`)
- GIW: Orange (`BadgeStyle.Warning`)

## Dependencies

- Radzen.Blazor (already installed)
- Existing `IDataUpdateRepository`
- Existing `pipelines.json` configuration
- Existing `UpdateTypes` enum (Hourly, Daily, Mass)