jdescopingtool/NEW/docs/plans/2026-01-07-pipeline-viewer-design.md

ETL Pipeline Viewer Component Design

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Build a Blazor component that visualizes ETL pipeline configuration and execution status.

Architecture: Single-page admin component with pipeline selector, execution status tables, and vertical configuration sections for each schedule type (Mass, Daily, Hourly).

Tech Stack: Blazor WebAssembly, Radzen components, existing DataUpdate repository

---

Overview

A read-only admin monitoring page that displays:

1. Pipeline configuration details from pipelines.json
2. Execution status (last run, next required, overdue status)
3. Recent execution history (last 10 runs per schedule type)

Component Structure

Page Layout

┌─────────────────────────────────────────────────────────────┐
│ ETL Pipeline Configuration Viewer                           │
├─────────────────────────────────────────────────────────────┤
│ Pipeline: [Dropdown - Alphabetical list]                    │
├─────────────────────────────────────────────────────────────┤
│ SCHEDULE STATUS SUMMARY (Table 1)                           │
│ ┌─────────┬────────────────────┬─────────────────┬────────┐ │
│ │ Type    │ Last Successful    │ Next Required   │ Status │ │
│ ├─────────┼────────────────────┼─────────────────┼────────┤ │
│ │ Mass    │ 2026-01-05 02:00   │ 2026-01-12 02:00│ ✓ OK   │ │
│ │ Daily   │ 2026-01-07 01:00   │ 2026-01-08 01:00│ ✓ OK   │ │
│ │ Hourly  │ 2026-01-07 10:00   │ 2026-01-07 11:00│ ⚠ Over │ │
│ └─────────┴────────────────────┴─────────────────┴────────┘ │
├─────────────────────────────────────────────────────────────┤
│ RECENT EXECUTION HISTORY (Table 2 - Last 10 per type)       │
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
│ │ Type    │ Start      │ End      │ Duration│ Rows│ Result│ │
│ ├─────────┼────────────┼──────────┼─────────┼─────┼───────┤ │
│ │ Hourly  │ 10:00 AM   │ 10:02 AM │ 2m 15s  │1,247│ ✓     │ │
│ │ Hourly  │ 09:00 AM   │ 09:01 AM │ 1m 42s  │ 892 │ ✓     │ │
│ │ ...     │            │          │         │     │       │ │
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ COMMON PIPELINE INFO                                         │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Source: [JDE] Connection                                │ │
│ │ Parameters: dateUpdated (jdeJulian), timeUpdated (jdeTime)│
│ └─────────────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Destination: WorkOrder_Curr                             │ │
│ │ Operation: Bulk Merge                                   │ │
│ │ Match: WorkOrderNumber, BranchCode                      │ │
│ │ Exclude: WorkOrderNumber, BranchCode, LastUpdateDt      │ │
│ └─────────────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Pre-Scripts: 0 scripts                                  │ │
│ │ Post-Scripts: 3 scripts [View]                          │ │
│ └─────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ MASS REFRESH                                    [Enabled]   │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Schedule Settings                                       │ │
│ │ Interval: 10080 min (7 days)         [Default]          │ │
│ │ Pre-Purge: Yes                       [Default]          │ │
│ │ Re-Index: Yes                        [Default]          │ │
│ ├─────────────────────────────────────────────────────────┤ │
│ │ Query: SELECT wo.WADOCO AS Work... [View Full Query]    │ │
│ └─────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ DAILY REFRESH                                   [Enabled]   │
│ └─ (Same structure as Mass)                                 │
├─────────────────────────────────────────────────────────────┤
│ HOURLY REFRESH                                  [Enabled]   │
│ └─ (Same structure as Mass)                                 │
└─────────────────────────────────────────────────────────────┘

Files to Create

Client Project (src/JdeScoping.Client/)

File	Description
Pages/Admin/PipelineViewer.razor	Main page component
Components/Admin/PipelineScheduleSection.razor	Reusable schedule section (Mass/Daily/Hourly)
Components/Admin/SqlQueryModal.razor	Modal for displaying SQL queries/scripts
Services/IPipelineConfigService.cs	Interface for pipeline config access
Services/PipelineConfigService.cs	Implementation - loads pipelines.json
Services/IPipelineStatusService.cs	Interface for execution status
Services/PipelineStatusService.cs	Implementation - calls API for status
Models/PipelineScheduleStatus.cs	View model for schedule status row
Models/PipelineExecution.cs	View model for execution history row

API Project (src/JdeScoping.Api/)

File	Description
Controllers/PipelineController.cs	API endpoints for pipeline config and status

DataSync Project (src/JdeScoping.DataSync/)

File	Description
Contracts/IDataUpdateRepository.cs	Add GetRecentUpdatesAsync method
Services/DataUpdateRepository.cs	Implement new method

Data Models

View Models (Client)

public record PipelineScheduleStatus(
    UpdateTypes ScheduleType,
    DateTime? LastSuccessfulRun,
    DateTime? NextRequiredRun,
    bool IsOverdue,
    int IntervalMinutes);

public record PipelineExecution(
    UpdateTypes ScheduleType,
    DateTime StartTime,
    DateTime? EndTime,
    TimeSpan Duration,
    long RecordCount,
    bool WasSuccessful);

Config Models (reuse or create in Contracts)

public record PipelinesRoot(
    PipelineSettings? Settings,
    ScheduleDefaults? ScheduleDefaults,
    Dictionary<string, PipelineConfig> Pipelines);

public record ScheduleDefaults(
    ScheduleConfig Mass,
    ScheduleConfig Daily,
    ScheduleConfig Hourly);

public record PipelineConfig(
    SourceConfig Source,
    DestinationConfig Destination,
    PipelineSchedules? Schedules,
    List<string>? PreScripts,
    List<string>? PostScripts);

public record SourceConfig(
    string Connection,
    string Query,
    string? MassQuery,
    Dictionary<string, ParameterConfig>? Parameters);

public record ParameterConfig(
    string Name,
    string? Format,
    string? Source);

public record DestinationConfig(
    string Table,
    string[]? MatchColumns,
    string[]? ExcludeFromUpdate);

public record PipelineSchedules(
    ScheduleConfig? Mass,
    ScheduleConfig? Daily,
    ScheduleConfig? Hourly);

public record ScheduleConfig(
    bool? Enabled,
    int? IntervalMinutes,
    bool? PrePurge,
    bool? ReIndex);

Service Interfaces

IPipelineConfigService

public interface IPipelineConfigService
{
    /// <summary>
    /// Gets the full pipelines configuration.
    /// </summary>
    Task<PipelinesRoot> GetPipelinesConfigAsync();

    /// <summary>
    /// Gets configuration for a specific pipeline.
    /// </summary>
    Task<PipelineConfig?> GetPipelineAsync(string pipelineName);

    /// <summary>
    /// Gets list of all pipeline names (sorted alphabetically).
    /// </summary>
    IEnumerable<string> GetPipelineNames();

    /// <summary>
    /// Gets schedule defaults for computing effective values.
    /// </summary>
    ScheduleDefaults GetScheduleDefaults();
}

IPipelineStatusService

public interface IPipelineStatusService
{
    /// <summary>
    /// Gets schedule status summary for a pipeline (one row per schedule type).
    /// </summary>
    Task<List<PipelineScheduleStatus>> GetScheduleStatusAsync(
        string pipelineName,
        CancellationToken cancellationToken = default);

    /// <summary>
    /// Gets recent execution history for a pipeline.
    /// </summary>
    Task<List<PipelineExecution>> GetRecentExecutionsAsync(
        string pipelineName,
        int count = 10,
        CancellationToken cancellationToken = default);
}

IDataUpdateRepository (addition)

/// <summary>
/// Gets the last N execution records for a specific table.
/// </summary>
Task<List<DataUpdate>> GetRecentUpdatesAsync(
    string tableName,
    int count = 10,
    CancellationToken cancellationToken = default);

SQL Query Modal Component

Features

• Large modal (80% viewport width)
• Monospace font display (<pre><code>)
• Basic SQL formatting (line breaks at clauses)
• Copy to Clipboard button
• Dynamic title based on context

Usage

<SqlQueryModal @bind-Visible="_showModal"
               Title="Mass Query - WorkOrder_Curr"
               Sql="@_selectedSql" />

Schedule Section Component

Features

• Header with schedule type name and Enabled/Disabled badge
• Settings table showing:
  • Interval (with "Default" or "Override" indicator)
  • Pre-Purge flag
  • Re-Index flag
• Query preview (truncated to ~100 chars)
• "View Full Query" button

Props

[Parameter] public string ScheduleType { get; set; }  // "Mass", "Daily", "Hourly"
[Parameter] public ScheduleConfig? Config { get; set; }
[Parameter] public ScheduleConfig DefaultConfig { get; set; }
[Parameter] public string? Query { get; set; }
[Parameter] public EventCallback<string> OnViewQuery { get; set; }

API Endpoints

Method	Route	Description
GET	/api/pipelines	Get all pipeline names
GET	/api/pipelines/{name}	Get pipeline configuration
GET	/api/pipelines/{name}/status	Get schedule status summary
GET	/api/pipelines/{name}/executions?count=10	Get recent executions

Implementation Notes

Override Detection

To show "Default" vs "Override" for schedule settings:

bool IsOverride(int? pipelineValue, int defaultValue) =>
    pipelineValue.HasValue && pipelineValue.Value != defaultValue;

Next Required Calculation

DateTime? nextRequired = lastSuccessful?.AddMinutes(intervalMinutes);
bool isOverdue = nextRequired.HasValue && nextRequired.Value < DateTime.UtcNow;

Query Truncation

string Truncate(string sql, int maxLength = 100) =>
    sql.Length <= maxLength ? sql : sql[..maxLength] + "...";

Connection Type Badge Colors

• JDE: Blue (BadgeStyle.Info)
• CMS: Green (BadgeStyle.Success)
• GIW: Orange (BadgeStyle.Warning)

Dependencies

• Radzen.Blazor (already installed)
• Existing IDataUpdateRepository
• Existing pipelines.json configuration
• Existing UpdateTypes enum (Hourly, Daily, Mass)