# ETL Pipeline Viewer Component Design

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status.

**Architecture:** Server-side API loads pipeline config and exposes DTOs; Blazor client consumes via typed HttpClient following existing ApiClientBase pattern.

**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataSync config types, existing DataUpdateRepository

---

## Codex Review Feedback (Addressed)

| Issue | Resolution |
|-------|------------|
| Client loading pipelines.json directly breaks CLEAN | API loads config, returns DTO to client |
| Duplicate config records diverge from DataSync | Reuse existing DataSync Configuration types |
| GetRecentUpdatesAsync missing UpdateType filter | Add UpdateType parameter to signature |
| Status/overdue logic diverges from DataSync | Reuse existing `DataUpdateRepository.IsOverdue()` |
| Only LastSuccessfulRun, hiding failures | Add LastRun + LastRunWasSuccessful |
| API patterns don't match ApiRoutes + ApiClientBase | Add `ApiRoutes.Pipelines`, implement `PipelineApiClient` |
| Duration non-nullable for in-progress | Make Duration nullable |
| ScheduleType as string | Use `UpdateTypes` enum |
| Missing authorization | Add `[Authorize]` to page and controller |

---

## Component Structure

### Page Layout

```
┌─────────────────────────────────────────────────────────────┐
│ ETL Pipeline Configuration Viewer          [Authorize]      │
├─────────────────────────────────────────────────────────────┤
│ Pipeline: [Dropdown - Alphabetical list]                    │
├─────────────────────────────────────────────────────────────┤
│ SCHEDULE STATUS SUMMARY                                     │
│ ┌─────────┬────────────┬────────────┬─────────────┬───────┐ │
│ │ Type    │ Last Run   │ Success?   │ Next Req.   │Status │ │
│ ├─────────┼────────────┼────────────┼─────────────┼───────┤ │
│ │ Mass    │ 01/05 02:00│ ✓          │ 01/12 02:00 │ OK    │ │
│ │ Daily   │ 01/07 01:00│ ✓          │ 01/08 01:00 │ OK    │ │
│ │ Hourly  │ 01/07 10:00│ ✗          │ 01/07 11:00 │ Over  │ │
│ └─────────┴────────────┴────────────┴─────────────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ RECENT EXECUTION HISTORY (Last 10 per type)                 │
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
│ │ Type    │ Start      │ End      │ Duration│ Rows│ Result│ │
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ COMMON PIPELINE INFO                                         │
│ [Source Card] [Destination Card] [Scripts Card]             │
├─────────────────────────────────────────────────────────────┤
│ MASS REFRESH [Enabled]                                      │
│ [Schedule Settings] [Query Preview + Modal]                 │
├─────────────────────────────────────────────────────────────┤
│ DAILY REFRESH [Enabled]                                     │
├─────────────────────────────────────────────────────────────┤
│ HOURLY REFRESH [Enabled]                                    │
└─────────────────────────────────────────────────────────────┘
```

---

## Architecture

### CLEAN Architecture Layers

```
┌─────────────────────────────────────────────────────────────┐
│ PRESENTATION (JdeScoping.Client)                            │
│  - PipelineViewer.razor (page)                              │
│  - PipelineScheduleSection.razor (component)                │
│  - SqlQueryModal.razor (component)                          │
│  - PipelineApiClient : ApiClientBase                        │
└─────────────────────────────────────────────────────────────┘
                              │ HTTP via ApiRoutes.Pipelines
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ APPLICATION (JdeScoping.Api)                                │
│  - PipelineController                                       │
│  - Returns DTOs from Core.ApiContracts                      │
└─────────────────────────────────────────────────────────────┘
                              │ Depends on
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ INFRASTRUCTURE (JdeScoping.DataSync)                        │
│  - IEtlPipelineFactory (existing - loads pipelines.json)    │
│  - IDataUpdateRepository (add GetRecentUpdatesAsync)        │
│  - Reuse existing Configuration/* types                     │
└─────────────────────────────────────────────────────────────┘
                              │ Depends on
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ DOMAIN (JdeScoping.Core)                                    │
│  - ApiRoutes.Pipelines (route constants)                    │
│  - ApiContracts.Pipelines/* (DTOs)                          │
│  - Models.Enums.UpdateTypes (existing)                      │
└─────────────────────────────────────────────────────────────┘
```

---

## Files to Create/Modify

### Core Project (`src/JdeScoping.Core/`)

| File | Action | Description |
|------|--------|-------------|
| `ApiContracts/ApiRoutes.cs` | Modify | Add `Pipelines` routes |
| `ApiContracts/Pipelines/PipelineListResponse.cs` | Create | List of pipeline names |
| `ApiContracts/Pipelines/PipelineConfigDto.cs` | Create | Pipeline config DTO (sanitized) |
| `ApiContracts/Pipelines/PipelineStatusDto.cs` | Create | Schedule status DTO |
| `ApiContracts/Pipelines/PipelineExecutionDto.cs` | Create | Execution history DTO |

### DataSync Project (`src/JdeScoping.DataSync/`)

| File | Action | Description |
|------|--------|-------------|
| `Contracts/IDataUpdateRepository.cs` | Modify | Add `GetRecentUpdatesAsync` |
| `Services/DataUpdateRepository.cs` | Modify | Implement new method |

### API Project (`src/JdeScoping.Api/`)

| File | Action | Description |
|------|--------|-------------|
| `Controllers/PipelineController.cs` | Create | API endpoints |

### Client Project (`src/JdeScoping.Client/`)

| File | Action | Description |
|------|--------|-------------|
| `Services/IPipelineApiClient.cs` | Create | Client interface |
| `Services/PipelineApiClient.cs` | Create | Client implementation |
| `Pages/Admin/PipelineViewer.razor` | Create | Main page |
| `Components/Admin/PipelineScheduleSection.razor` | Create | Schedule section |
| `Components/Admin/SqlQueryModal.razor` | Create | SQL modal |

---

## API Routes (add to `ApiRoutes.cs`)

```csharp
/// <summary>
/// Routes for pipeline configuration API endpoints.
/// </summary>
public static class Pipelines
{
    /// <summary>Base route for pipeline endpoints.</summary>
    public const string Base = "api/pipelines";

    /// <summary>Route template for getting a pipeline by name.</summary>
    public const string ByName = "{name}";

    /// <summary>Route template for getting pipeline status.</summary>
    public const string Status = "{name}/status";

    /// <summary>Route template for getting pipeline executions.</summary>
    public const string Executions = "{name}/executions";

    /// <summary>Builds the route to get a specific pipeline config.</summary>
    public static string GetByName(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}";

    /// <summary>Builds the route to get pipeline status.</summary>
    public static string GetStatus(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}/status";

    /// <summary>Builds the route to get pipeline executions.</summary>
    public static string GetExecutions(string name, int count = 10) =>
        $"api/pipelines/{Uri.EscapeDataString(name)}/executions?count={count}";
}
```

---

## DTOs (in `Core/ApiContracts/Pipelines/`)

### PipelineListResponse.cs

```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;

/// <summary>
/// Response containing list of available pipeline names.
/// </summary>
public record PipelineListResponse(List<string> PipelineNames);
```

### PipelineConfigDto.cs

```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;

/// <summary>
/// Pipeline configuration DTO for display (sanitized - no raw SQL exposed by default).
/// </summary>
public record PipelineConfigDto(
    string Name,
    PipelineSourceDto Source,
    PipelineDestinationDto Destination,
    PipelineSchedulesDto Schedules,
    int PreScriptCount,
    int PostScriptCount,
    List<string>? PreScripts,   // Only populated if user has admin role
    List<string>? PostScripts); // Only populated if user has admin role

public record PipelineSourceDto(
    string Connection,
    string? QueryPreview,       // First 100 chars
    string? MassQueryPreview,   // First 100 chars
    string? Query,              // Full query - only if admin
    string? MassQuery,          // Full query - only if admin
    List<PipelineParameterDto> Parameters);

public record PipelineParameterDto(
    string Name,
    string? Format,
    string Source);

public record PipelineDestinationDto(
    string Table,
    string OperationType,       // "BulkMerge" or "BulkImport"
    List<string>? MatchColumns,
    List<string>? ExcludeFromUpdate);

public record PipelineSchedulesDto(
    PipelineScheduleDto Mass,
    PipelineScheduleDto Daily,
    PipelineScheduleDto Hourly);

public record PipelineScheduleDto(
    bool Enabled,
    int IntervalMinutes,
    bool PrePurge,
    bool ReIndex,
    bool IntervalIsOverride,
    bool PrePurgeIsOverride,
    bool ReIndexIsOverride);
```

### PipelineStatusDto.cs

```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;

using JdeScoping.Core.Models.Enums;

/// <summary>
/// Pipeline schedule status for each update type.
/// </summary>
public record PipelineStatusResponse(List<PipelineScheduleStatusDto> Statuses);

public record PipelineScheduleStatusDto(
    UpdateTypes ScheduleType,
    DateTime? LastRun,
    bool LastRunWasSuccessful,
    DateTime? LastSuccessfulRun,
    DateTime? NextRequiredRun,
    bool IsOverdue,
    int IntervalMinutes);
```

### PipelineExecutionDto.cs

```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;

using JdeScoping.Core.Models.Enums;

/// <summary>
/// Pipeline execution history.
/// </summary>
public record PipelineExecutionsResponse(List<PipelineExecutionDto> Executions);

public record PipelineExecutionDto(
    UpdateTypes ScheduleType,
    DateTime StartTime,
    DateTime? EndTime,
    TimeSpan? Duration,         // Nullable for in-progress runs
    long RecordCount,
    bool WasSuccessful);
```

---

## Repository Update (IDataUpdateRepository)

```csharp
/// <summary>
/// Gets the last N execution records for a specific table and optional update type.
/// </summary>
/// <param name="tableName">The table name to filter by.</param>
/// <param name="updateType">Optional update type filter. If null, returns all types.</param>
/// <param name="count">Maximum records to return per update type.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>List of DataUpdate records ordered by StartDt descending.</returns>
Task<List<DataUpdate>> GetRecentUpdatesAsync(
    string tableName,
    UpdateTypes? updateType = null,
    int count = 10,
    CancellationToken cancellationToken = default);

/// <summary>
/// Gets the last run (successful or not) for each update type for a table.
/// </summary>
/// <param name="tableName">The table name.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>Dictionary keyed by UpdateType.</returns>
Task<Dictionary<UpdateTypes, DataUpdate>> GetLastRunsAsync(
    string tableName,
    CancellationToken cancellationToken = default);
```

---

## API Controller

```csharp
[ApiController]
[Route(ApiRoutes.Pipelines.Base)]
[Authorize]
public class PipelineController : ControllerBase
{
    private readonly IEtlPipelineFactory _pipelineFactory;
    private readonly IDataUpdateRepository _dataUpdateRepository;

    [HttpGet]
    public ActionResult<PipelineListResponse> GetPipelineNames()
    {
        var names = _pipelineFactory.GetAvailableTables()
            .OrderBy(n => n)
            .ToList();
        return Ok(new PipelineListResponse(names));
    }

    [HttpGet(ApiRoutes.Pipelines.ByName)]
    public ActionResult<PipelineConfigDto> GetPipeline(string name)
    {
        // Map from internal config to DTO
        // Truncate queries for preview, include full if admin
    }

    [HttpGet(ApiRoutes.Pipelines.Status)]
    public async Task<ActionResult<PipelineStatusResponse>> GetStatus(string name)
    {
        // Get last runs, calculate IsOverdue using DataUpdateRepository.IsOverdue()
    }

    [HttpGet(ApiRoutes.Pipelines.Executions)]
    public async Task<ActionResult<PipelineExecutionsResponse>> GetExecutions(
        string name, [FromQuery] int count = 10)
    {
        // Get recent updates from repository
    }
}
```

---

## Client Implementation

### PipelineApiClient.cs

```csharp
public interface IPipelineApiClient
{
    Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default);
    Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default);
    Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default);
    Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default);
}

public class PipelineApiClient : ApiClientBase, IPipelineApiClient
{
    public PipelineApiClient(HttpClient httpClient) : base(httpClient) { }

    public Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default)
        => GetAsync<PipelineListResponse>(ApiRoutes.Pipelines.Base, ct);

    public Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default)
        => GetAsync<PipelineConfigDto>(ApiRoutes.Pipelines.GetByName(name), ct);

    public Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default)
        => GetAsync<PipelineStatusResponse>(ApiRoutes.Pipelines.GetStatus(name), ct);

    public Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default)
        => GetAsync<PipelineExecutionsResponse>(ApiRoutes.Pipelines.GetExecutions(name, count), ct);
}
```

---

## Blazor Components

### PipelineViewer.razor Structure

```razor
@page "/admin/pipeline-viewer"
@attribute [Authorize]
@inject IPipelineApiClient PipelineApi

<PageTitle>Pipeline Configuration Viewer</PageTitle>

<RadzenText TextStyle="TextStyle.H4">ETL Pipeline Configuration Viewer</RadzenText>

<!-- Pipeline Selector -->
<RadzenDropDown @bind-Value="_selectedPipeline"
                Data="_pipelineNames"
                Change="OnPipelineChanged" />

@if (_config is not null)
{
    <!-- Status Summary Table -->
    <!-- Execution History Table -->
    <!-- Common Info Cards -->
    <!-- Schedule Sections (Mass, Daily, Hourly) -->
}

<SqlQueryModal @bind-Visible="_showSqlModal" Title="@_sqlModalTitle" Sql="@_sqlModalContent" />
```

### SqlQueryModal.razor

- Large modal (80% width)
- `<pre><code>` with monospace font
- Copy to clipboard button
- Basic SQL formatting (line breaks at clauses)

### PipelineScheduleSection.razor

- Props: ScheduleType (UpdateTypes), Config (PipelineScheduleDto), Query, OnViewQuery callback
- Shows enabled badge
- Settings table with Default/Override indicators
- Query preview with "View Full" button

---

## Implementation Notes

### Override Detection (in API)

```csharp
bool IsOverride<T>(T? pipelineValue, T defaultValue) where T : struct =>
    pipelineValue.HasValue && !pipelineValue.Value.Equals(defaultValue);
```

### Overdue Calculation (reuse existing)

```csharp
// Use DataUpdateRepository.IsOverdue() which includes 50% grace period
var isOverdue = DataUpdateRepository.IsOverdue(
    lastSuccessfulRun, tableName, updateType, customIntervals);
```

### Connection Type Badge Colors

| Connection | Badge Style |
|------------|-------------|
| jde | `BadgeStyle.Info` (blue) |
| cms | `BadgeStyle.Success` (green) |
| giw | `BadgeStyle.Warning` (orange) |

### SQL Query Truncation

```csharp
static string Truncate(string? sql, int maxLength = 100) =>
    sql is null ? "" :
    sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
```

---

## Security Considerations

1. **Authorization**: Page and API require `[Authorize]`
2. **SQL Visibility**: Full SQL/scripts only exposed if user has appropriate permissions (consider admin-only)
3. **No write operations**: This is read-only, no mutations exposed

---

## Dependencies

- Radzen.Blazor (existing)
- Existing `IEtlPipelineFactory` and `IDataUpdateRepository`
- Existing `pipelines.json` configuration
- Existing `UpdateTypes` enum
- Existing `ApiClientBase` pattern