jdescopingtool/NEW/docs/plans/2026-01-07-pipeline-viewer-design.md

ETL Pipeline Viewer Component Design

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Build a Blazor component that visualizes ETL pipeline configuration and execution status.

Architecture: Server-side API loads pipeline config and exposes DTOs; Blazor client consumes via typed HttpClient following existing ApiClientBase pattern.

Tech Stack: Blazor WebAssembly, Radzen components, existing DataSync config types, existing DataUpdateRepository

---

Codex Review Feedback (Addressed)

Issue	Resolution
Client loading pipelines.json directly breaks CLEAN	API loads config, returns DTO to client
Duplicate config records diverge from DataSync	Reuse existing DataSync Configuration types
GetRecentUpdatesAsync missing UpdateType filter	Add UpdateType parameter to signature
Status/overdue logic diverges from DataSync	Reuse existing DataUpdateRepository.IsOverdue()
Only LastSuccessfulRun, hiding failures	Add LastRun + LastRunWasSuccessful
API patterns don't match ApiRoutes + ApiClientBase	Add ApiRoutes.Pipelines, implement PipelineApiClient
Duration non-nullable for in-progress	Make Duration nullable
ScheduleType as string	Use UpdateTypes enum
Missing authorization	Add [Authorize] to page and controller

---

Component Structure

Page Layout

┌─────────────────────────────────────────────────────────────┐
│ ETL Pipeline Configuration Viewer          [Authorize]      │
├─────────────────────────────────────────────────────────────┤
│ Pipeline: [Dropdown - Alphabetical list]                    │
├─────────────────────────────────────────────────────────────┤
│ SCHEDULE STATUS SUMMARY                                     │
│ ┌─────────┬────────────┬────────────┬─────────────┬───────┐ │
│ │ Type    │ Last Run   │ Success?   │ Next Req.   │Status │ │
│ ├─────────┼────────────┼────────────┼─────────────┼───────┤ │
│ │ Mass    │ 01/05 02:00│ ✓          │ 01/12 02:00 │ OK    │ │
│ │ Daily   │ 01/07 01:00│ ✓          │ 01/08 01:00 │ OK    │ │
│ │ Hourly  │ 01/07 10:00│ ✗          │ 01/07 11:00 │ Over  │ │
│ └─────────┴────────────┴────────────┴─────────────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ RECENT EXECUTION HISTORY (Last 10 per type)                 │
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
│ │ Type    │ Start      │ End      │ Duration│ Rows│ Result│ │
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ COMMON PIPELINE INFO                                         │
│ [Source Card] [Destination Card] [Scripts Card]             │
├─────────────────────────────────────────────────────────────┤
│ MASS REFRESH [Enabled]                                      │
│ [Schedule Settings] [Query Preview + Modal]                 │
├─────────────────────────────────────────────────────────────┤
│ DAILY REFRESH [Enabled]                                     │
├─────────────────────────────────────────────────────────────┤
│ HOURLY REFRESH [Enabled]                                    │
└─────────────────────────────────────────────────────────────┘

---

Architecture

CLEAN Architecture Layers

┌─────────────────────────────────────────────────────────────┐
│ PRESENTATION (JdeScoping.Client)                            │
│  - PipelineViewer.razor (page)                              │
│  - PipelineScheduleSection.razor (component)                │
│  - SqlQueryModal.razor (component)                          │
│  - PipelineApiClient : ApiClientBase                        │
└─────────────────────────────────────────────────────────────┘
                              │ HTTP via ApiRoutes.Pipelines
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ APPLICATION (JdeScoping.Api)                                │
│  - PipelineController                                       │
│  - Returns DTOs from Core.ApiContracts                      │
└─────────────────────────────────────────────────────────────┘
                              │ Depends on
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ INFRASTRUCTURE (JdeScoping.DataSync)                        │
│  - IEtlPipelineFactory (existing - loads pipelines.json)    │
│  - IDataUpdateRepository (add GetRecentUpdatesAsync)        │
│  - Reuse existing Configuration/* types                     │
└─────────────────────────────────────────────────────────────┘
                              │ Depends on
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ DOMAIN (JdeScoping.Core)                                    │
│  - ApiRoutes.Pipelines (route constants)                    │
│  - ApiContracts.Pipelines/* (DTOs)                          │
│  - Models.Enums.UpdateTypes (existing)                      │
└─────────────────────────────────────────────────────────────┘

---

Files to Create/Modify

Core Project (src/JdeScoping.Core/)

File	Action	Description
ApiContracts/ApiRoutes.cs	Modify	Add Pipelines routes
ApiContracts/Pipelines/PipelineListResponse.cs	Create	List of pipeline names
ApiContracts/Pipelines/PipelineConfigDto.cs	Create	Pipeline config DTO (sanitized)
ApiContracts/Pipelines/PipelineStatusDto.cs	Create	Schedule status DTO
ApiContracts/Pipelines/PipelineExecutionDto.cs	Create	Execution history DTO

DataSync Project (src/JdeScoping.DataSync/)

File	Action	Description
Contracts/IDataUpdateRepository.cs	Modify	Add GetRecentUpdatesAsync
Services/DataUpdateRepository.cs	Modify	Implement new method

API Project (src/JdeScoping.Api/)

File	Action	Description
Controllers/PipelineController.cs	Create	API endpoints

Client Project (src/JdeScoping.Client/)

File	Action	Description
Services/IPipelineApiClient.cs	Create	Client interface
Services/PipelineApiClient.cs	Create	Client implementation
Pages/Admin/PipelineViewer.razor	Create	Main page
Components/Admin/PipelineScheduleSection.razor	Create	Schedule section
Components/Admin/SqlQueryModal.razor	Create	SQL modal

---

API Routes (add to ApiRoutes.cs)

/// <summary>
/// Routes for pipeline configuration API endpoints.
/// </summary>
public static class Pipelines
{
    /// <summary>Base route for pipeline endpoints.</summary>
    public const string Base = "api/pipelines";

    /// <summary>Route template for getting a pipeline by name.</summary>
    public const string ByName = "{name}";

    /// <summary>Route template for getting pipeline status.</summary>
    public const string Status = "{name}/status";

    /// <summary>Route template for getting pipeline executions.</summary>
    public const string Executions = "{name}/executions";

    /// <summary>Builds the route to get a specific pipeline config.</summary>
    public static string GetByName(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}";

    /// <summary>Builds the route to get pipeline status.</summary>
    public static string GetStatus(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}/status";

    /// <summary>Builds the route to get pipeline executions.</summary>
    public static string GetExecutions(string name, int count = 10) =>
        $"api/pipelines/{Uri.EscapeDataString(name)}/executions?count={count}";
}

---

DTOs (in Core/ApiContracts/Pipelines/)

PipelineListResponse.cs

namespace JdeScoping.Core.ApiContracts.Pipelines;

/// <summary>
/// Response containing list of available pipeline names.
/// </summary>
public record PipelineListResponse(List<string> PipelineNames);

PipelineConfigDto.cs

namespace JdeScoping.Core.ApiContracts.Pipelines;

/// <summary>
/// Pipeline configuration DTO for display (sanitized - no raw SQL exposed by default).
/// </summary>
public record PipelineConfigDto(
    string Name,
    PipelineSourceDto Source,
    PipelineDestinationDto Destination,
    PipelineSchedulesDto Schedules,
    int PreScriptCount,
    int PostScriptCount,
    List<string>? PreScripts,   // Only populated if user has admin role
    List<string>? PostScripts); // Only populated if user has admin role

public record PipelineSourceDto(
    string Connection,
    string? QueryPreview,       // First 100 chars
    string? MassQueryPreview,   // First 100 chars
    string? Query,              // Full query - only if admin
    string? MassQuery,          // Full query - only if admin
    List<PipelineParameterDto> Parameters);

public record PipelineParameterDto(
    string Name,
    string? Format,
    string Source);

public record PipelineDestinationDto(
    string Table,
    string OperationType,       // "BulkMerge" or "BulkImport"
    List<string>? MatchColumns,
    List<string>? ExcludeFromUpdate);

public record PipelineSchedulesDto(
    PipelineScheduleDto Mass,
    PipelineScheduleDto Daily,
    PipelineScheduleDto Hourly);

public record PipelineScheduleDto(
    bool Enabled,
    int IntervalMinutes,
    bool PrePurge,
    bool ReIndex,
    bool IntervalIsOverride,
    bool PrePurgeIsOverride,
    bool ReIndexIsOverride);

PipelineStatusDto.cs

namespace JdeScoping.Core.ApiContracts.Pipelines;

using JdeScoping.Core.Models.Enums;

/// <summary>
/// Pipeline schedule status for each update type.
/// </summary>
public record PipelineStatusResponse(List<PipelineScheduleStatusDto> Statuses);

public record PipelineScheduleStatusDto(
    UpdateTypes ScheduleType,
    DateTime? LastRun,
    bool LastRunWasSuccessful,
    DateTime? LastSuccessfulRun,
    DateTime? NextRequiredRun,
    bool IsOverdue,
    int IntervalMinutes);

PipelineExecutionDto.cs

namespace JdeScoping.Core.ApiContracts.Pipelines;

using JdeScoping.Core.Models.Enums;

/// <summary>
/// Pipeline execution history.
/// </summary>
public record PipelineExecutionsResponse(List<PipelineExecutionDto> Executions);

public record PipelineExecutionDto(
    UpdateTypes ScheduleType,
    DateTime StartTime,
    DateTime? EndTime,
    TimeSpan? Duration,         // Nullable for in-progress runs
    long RecordCount,
    bool WasSuccessful);

---

Repository Update (IDataUpdateRepository)

/// <summary>
/// Gets the last N execution records for a specific table and optional update type.
/// </summary>
/// <param name="tableName">The table name to filter by.</param>
/// <param name="updateType">Optional update type filter. If null, returns all types.</param>
/// <param name="count">Maximum records to return per update type.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>List of DataUpdate records ordered by StartDt descending.</returns>
Task<List<DataUpdate>> GetRecentUpdatesAsync(
    string tableName,
    UpdateTypes? updateType = null,
    int count = 10,
    CancellationToken cancellationToken = default);

/// <summary>
/// Gets the last run (successful or not) for each update type for a table.
/// </summary>
/// <param name="tableName">The table name.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>Dictionary keyed by UpdateType.</returns>
Task<Dictionary<UpdateTypes, DataUpdate>> GetLastRunsAsync(
    string tableName,
    CancellationToken cancellationToken = default);

---

API Controller

[ApiController]
[Route(ApiRoutes.Pipelines.Base)]
[Authorize]
public class PipelineController : ControllerBase
{
    private readonly IEtlPipelineFactory _pipelineFactory;
    private readonly IDataUpdateRepository _dataUpdateRepository;

    [HttpGet]
    public ActionResult<PipelineListResponse> GetPipelineNames()
    {
        var names = _pipelineFactory.GetAvailableTables()
            .OrderBy(n => n)
            .ToList();
        return Ok(new PipelineListResponse(names));
    }

    [HttpGet(ApiRoutes.Pipelines.ByName)]
    public ActionResult<PipelineConfigDto> GetPipeline(string name)
    {
        // Map from internal config to DTO
        // Truncate queries for preview, include full if admin
    }

    [HttpGet(ApiRoutes.Pipelines.Status)]
    public async Task<ActionResult<PipelineStatusResponse>> GetStatus(string name)
    {
        // Get last runs, calculate IsOverdue using DataUpdateRepository.IsOverdue()
    }

    [HttpGet(ApiRoutes.Pipelines.Executions)]
    public async Task<ActionResult<PipelineExecutionsResponse>> GetExecutions(
        string name, [FromQuery] int count = 10)
    {
        // Get recent updates from repository
    }
}

---

Client Implementation

PipelineApiClient.cs

public interface IPipelineApiClient
{
    Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default);
    Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default);
    Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default);
    Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default);
}

public class PipelineApiClient : ApiClientBase, IPipelineApiClient
{
    public PipelineApiClient(HttpClient httpClient) : base(httpClient) { }

    public Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default)
        => GetAsync<PipelineListResponse>(ApiRoutes.Pipelines.Base, ct);

    public Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default)
        => GetAsync<PipelineConfigDto>(ApiRoutes.Pipelines.GetByName(name), ct);

    public Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default)
        => GetAsync<PipelineStatusResponse>(ApiRoutes.Pipelines.GetStatus(name), ct);

    public Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default)
        => GetAsync<PipelineExecutionsResponse>(ApiRoutes.Pipelines.GetExecutions(name, count), ct);
}

---

Blazor Components

PipelineViewer.razor Structure

@page "/admin/pipeline-viewer"
@attribute [Authorize]
@inject IPipelineApiClient PipelineApi

<PageTitle>Pipeline Configuration Viewer</PageTitle>

<RadzenText TextStyle="TextStyle.H4">ETL Pipeline Configuration Viewer</RadzenText>

<!-- Pipeline Selector -->
<RadzenDropDown @bind-Value="_selectedPipeline"
                Data="_pipelineNames"
                Change="OnPipelineChanged" />

@if (_config is not null)
{
    <!-- Status Summary Table -->
    <!-- Execution History Table -->
    <!-- Common Info Cards -->
    <!-- Schedule Sections (Mass, Daily, Hourly) -->
}

<SqlQueryModal @bind-Visible="_showSqlModal" Title="@_sqlModalTitle" Sql="@_sqlModalContent" />

SqlQueryModal.razor

• Large modal (80% width)
• <pre><code> with monospace font
• Copy to clipboard button
• Basic SQL formatting (line breaks at clauses)

PipelineScheduleSection.razor

• Props: ScheduleType (UpdateTypes), Config (PipelineScheduleDto), Query, OnViewQuery callback
• Shows enabled badge
• Settings table with Default/Override indicators
• Query preview with "View Full" button

---

Implementation Notes

Override Detection (in API)

bool IsOverride<T>(T? pipelineValue, T defaultValue) where T : struct =>
    pipelineValue.HasValue && !pipelineValue.Value.Equals(defaultValue);

Overdue Calculation (reuse existing)

// Use DataUpdateRepository.IsOverdue() which includes 50% grace period
var isOverdue = DataUpdateRepository.IsOverdue(
    lastSuccessfulRun, tableName, updateType, customIntervals);

Connection Type Badge Colors

Connection	Badge Style
jde	BadgeStyle.Info (blue)
cms	BadgeStyle.Success (green)
giw	BadgeStyle.Warning (orange)

SQL Query Truncation

static string Truncate(string? sql, int maxLength = 100) =>
    sql is null ? "" :
    sql.Length <= maxLength ? sql : sql[..maxLength] + "...";

---

Security Considerations

1. Authorization: Page and API require [Authorize]
2. SQL Visibility: Full SQL/scripts only exposed if user has appropriate permissions (consider admin-only)
3. No write operations: This is read-only, no mutations exposed

---

Dependencies

• Radzen.Blazor (existing)
• Existing IEtlPipelineFactory and IDataUpdateRepository
• Existing pipelines.json configuration
• Existing UpdateTypes enum
• Existing ApiClientBase pattern