docs: add ETL pipeline viewer component design
Design document for admin monitoring component that visualizes: - Pipeline configuration from pipelines.json - Execution status (last run, next required, overdue) - Recent execution history (last 10 per schedule type) - Schedule sections for Mass/Daily/Hourly with defaults vs overrides
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,319 @@
|
|||||||
|
# ETL Pipeline Viewer Component Design
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||||
|
|
||||||
|
**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status.
|
||||||
|
|
||||||
|
**Architecture:** Single-page admin component with pipeline selector, execution status tables, and vertical configuration sections for each schedule type (Mass, Daily, Hourly).
|
||||||
|
|
||||||
|
**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataUpdate repository
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
A read-only admin monitoring page that displays:
|
||||||
|
1. Pipeline configuration details from `pipelines.json`
|
||||||
|
2. Execution status (last run, next required, overdue status)
|
||||||
|
3. Recent execution history (last 10 runs per schedule type)
|
||||||
|
|
||||||
|
## Component Structure
|
||||||
|
|
||||||
|
### Page Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ ETL Pipeline Configuration Viewer │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ Pipeline: [Dropdown - Alphabetical list] │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ SCHEDULE STATUS SUMMARY (Table 1) │
|
||||||
|
│ ┌─────────┬────────────────────┬─────────────────┬────────┐ │
|
||||||
|
│ │ Type │ Last Successful │ Next Required │ Status │ │
|
||||||
|
│ ├─────────┼────────────────────┼─────────────────┼────────┤ │
|
||||||
|
│ │ Mass │ 2026-01-05 02:00 │ 2026-01-12 02:00│ ✓ OK │ │
|
||||||
|
│ │ Daily │ 2026-01-07 01:00 │ 2026-01-08 01:00│ ✓ OK │ │
|
||||||
|
│ │ Hourly │ 2026-01-07 10:00 │ 2026-01-07 11:00│ ⚠ Over │ │
|
||||||
|
│ └─────────┴────────────────────┴─────────────────┴────────┘ │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ RECENT EXECUTION HISTORY (Table 2 - Last 10 per type) │
|
||||||
|
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
|
||||||
|
│ │ Type │ Start │ End │ Duration│ Rows│ Result│ │
|
||||||
|
│ ├─────────┼────────────┼──────────┼─────────┼─────┼───────┤ │
|
||||||
|
│ │ Hourly │ 10:00 AM │ 10:02 AM │ 2m 15s │1,247│ ✓ │ │
|
||||||
|
│ │ Hourly │ 09:00 AM │ 09:01 AM │ 1m 42s │ 892 │ ✓ │ │
|
||||||
|
│ │ ... │ │ │ │ │ │ │
|
||||||
|
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ COMMON PIPELINE INFO │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Source: [JDE] Connection │ │
|
||||||
|
│ │ Parameters: dateUpdated (jdeJulian), timeUpdated (jdeTime)│
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Destination: WorkOrder_Curr │ │
|
||||||
|
│ │ Operation: Bulk Merge │ │
|
||||||
|
│ │ Match: WorkOrderNumber, BranchCode │ │
|
||||||
|
│ │ Exclude: WorkOrderNumber, BranchCode, LastUpdateDt │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Pre-Scripts: 0 scripts │ │
|
||||||
|
│ │ Post-Scripts: 3 scripts [View] │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ MASS REFRESH [Enabled] │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Schedule Settings │ │
|
||||||
|
│ │ Interval: 10080 min (7 days) [Default] │ │
|
||||||
|
│ │ Pre-Purge: Yes [Default] │ │
|
||||||
|
│ │ Re-Index: Yes [Default] │ │
|
||||||
|
│ ├─────────────────────────────────────────────────────────┤ │
|
||||||
|
│ │ Query: SELECT wo.WADOCO AS Work... [View Full Query] │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ DAILY REFRESH [Enabled] │
|
||||||
|
│ └─ (Same structure as Mass) │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ HOURLY REFRESH [Enabled] │
|
||||||
|
│ └─ (Same structure as Mass) │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files to Create
|
||||||
|
|
||||||
|
### Client Project (`src/JdeScoping.Client/`)
|
||||||
|
|
||||||
|
| File | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `Pages/Admin/PipelineViewer.razor` | Main page component |
|
||||||
|
| `Components/Admin/PipelineScheduleSection.razor` | Reusable schedule section (Mass/Daily/Hourly) |
|
||||||
|
| `Components/Admin/SqlQueryModal.razor` | Modal for displaying SQL queries/scripts |
|
||||||
|
| `Services/IPipelineConfigService.cs` | Interface for pipeline config access |
|
||||||
|
| `Services/PipelineConfigService.cs` | Implementation - loads pipelines.json |
|
||||||
|
| `Services/IPipelineStatusService.cs` | Interface for execution status |
|
||||||
|
| `Services/PipelineStatusService.cs` | Implementation - calls API for status |
|
||||||
|
| `Models/PipelineScheduleStatus.cs` | View model for schedule status row |
|
||||||
|
| `Models/PipelineExecution.cs` | View model for execution history row |
|
||||||
|
|
||||||
|
### API Project (`src/JdeScoping.Api/`)
|
||||||
|
|
||||||
|
| File | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `Controllers/PipelineController.cs` | API endpoints for pipeline config and status |
|
||||||
|
|
||||||
|
### DataSync Project (`src/JdeScoping.DataSync/`)
|
||||||
|
|
||||||
|
| File | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `Contracts/IDataUpdateRepository.cs` | Add `GetRecentUpdatesAsync` method |
|
||||||
|
| `Services/DataUpdateRepository.cs` | Implement new method |
|
||||||
|
|
||||||
|
## Data Models
|
||||||
|
|
||||||
|
### View Models (Client)
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
public record PipelineScheduleStatus(
|
||||||
|
UpdateTypes ScheduleType,
|
||||||
|
DateTime? LastSuccessfulRun,
|
||||||
|
DateTime? NextRequiredRun,
|
||||||
|
bool IsOverdue,
|
||||||
|
int IntervalMinutes);
|
||||||
|
|
||||||
|
public record PipelineExecution(
|
||||||
|
UpdateTypes ScheduleType,
|
||||||
|
DateTime StartTime,
|
||||||
|
DateTime? EndTime,
|
||||||
|
TimeSpan Duration,
|
||||||
|
long RecordCount,
|
||||||
|
bool WasSuccessful);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Config Models (reuse or create in Contracts)
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
public record PipelinesRoot(
|
||||||
|
PipelineSettings? Settings,
|
||||||
|
ScheduleDefaults? ScheduleDefaults,
|
||||||
|
Dictionary<string, PipelineConfig> Pipelines);
|
||||||
|
|
||||||
|
public record ScheduleDefaults(
|
||||||
|
ScheduleConfig Mass,
|
||||||
|
ScheduleConfig Daily,
|
||||||
|
ScheduleConfig Hourly);
|
||||||
|
|
||||||
|
public record PipelineConfig(
|
||||||
|
SourceConfig Source,
|
||||||
|
DestinationConfig Destination,
|
||||||
|
PipelineSchedules? Schedules,
|
||||||
|
List<string>? PreScripts,
|
||||||
|
List<string>? PostScripts);
|
||||||
|
|
||||||
|
public record SourceConfig(
|
||||||
|
string Connection,
|
||||||
|
string Query,
|
||||||
|
string? MassQuery,
|
||||||
|
Dictionary<string, ParameterConfig>? Parameters);
|
||||||
|
|
||||||
|
public record ParameterConfig(
|
||||||
|
string Name,
|
||||||
|
string? Format,
|
||||||
|
string? Source);
|
||||||
|
|
||||||
|
public record DestinationConfig(
|
||||||
|
string Table,
|
||||||
|
string[]? MatchColumns,
|
||||||
|
string[]? ExcludeFromUpdate);
|
||||||
|
|
||||||
|
public record PipelineSchedules(
|
||||||
|
ScheduleConfig? Mass,
|
||||||
|
ScheduleConfig? Daily,
|
||||||
|
ScheduleConfig? Hourly);
|
||||||
|
|
||||||
|
public record ScheduleConfig(
|
||||||
|
bool? Enabled,
|
||||||
|
int? IntervalMinutes,
|
||||||
|
bool? PrePurge,
|
||||||
|
bool? ReIndex);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Service Interfaces
|
||||||
|
|
||||||
|
### IPipelineConfigService
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
public interface IPipelineConfigService
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Gets the full pipelines configuration.
|
||||||
|
/// </summary>
|
||||||
|
Task<PipelinesRoot> GetPipelinesConfigAsync();
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets configuration for a specific pipeline.
|
||||||
|
/// </summary>
|
||||||
|
Task<PipelineConfig?> GetPipelineAsync(string pipelineName);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets list of all pipeline names (sorted alphabetically).
|
||||||
|
/// </summary>
|
||||||
|
IEnumerable<string> GetPipelineNames();
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets schedule defaults for computing effective values.
|
||||||
|
/// </summary>
|
||||||
|
ScheduleDefaults GetScheduleDefaults();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### IPipelineStatusService
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
public interface IPipelineStatusService
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Gets schedule status summary for a pipeline (one row per schedule type).
|
||||||
|
/// </summary>
|
||||||
|
Task<List<PipelineScheduleStatus>> GetScheduleStatusAsync(
|
||||||
|
string pipelineName,
|
||||||
|
CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets recent execution history for a pipeline.
|
||||||
|
/// </summary>
|
||||||
|
Task<List<PipelineExecution>> GetRecentExecutionsAsync(
|
||||||
|
string pipelineName,
|
||||||
|
int count = 10,
|
||||||
|
CancellationToken cancellationToken = default);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### IDataUpdateRepository (addition)
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
/// <summary>
|
||||||
|
/// Gets the last N execution records for a specific table.
|
||||||
|
/// </summary>
|
||||||
|
Task<List<DataUpdate>> GetRecentUpdatesAsync(
|
||||||
|
string tableName,
|
||||||
|
int count = 10,
|
||||||
|
CancellationToken cancellationToken = default);
|
||||||
|
```
|
||||||
|
|
||||||
|
## SQL Query Modal Component
|
||||||
|
|
||||||
|
### Features
|
||||||
|
- Large modal (80% viewport width)
|
||||||
|
- Monospace font display (`<pre><code>`)
|
||||||
|
- Basic SQL formatting (line breaks at clauses)
|
||||||
|
- Copy to Clipboard button
|
||||||
|
- Dynamic title based on context
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
```razor
|
||||||
|
<SqlQueryModal @bind-Visible="_showModal"
|
||||||
|
Title="Mass Query - WorkOrder_Curr"
|
||||||
|
Sql="@_selectedSql" />
|
||||||
|
```
|
||||||
|
|
||||||
|
## Schedule Section Component
|
||||||
|
|
||||||
|
### Features
|
||||||
|
- Header with schedule type name and Enabled/Disabled badge
|
||||||
|
- Settings table showing:
|
||||||
|
- Interval (with "Default" or "Override" indicator)
|
||||||
|
- Pre-Purge flag
|
||||||
|
- Re-Index flag
|
||||||
|
- Query preview (truncated to ~100 chars)
|
||||||
|
- "View Full Query" button
|
||||||
|
|
||||||
|
### Props
|
||||||
|
```csharp
|
||||||
|
[Parameter] public string ScheduleType { get; set; } // "Mass", "Daily", "Hourly"
|
||||||
|
[Parameter] public ScheduleConfig? Config { get; set; }
|
||||||
|
[Parameter] public ScheduleConfig DefaultConfig { get; set; }
|
||||||
|
[Parameter] public string? Query { get; set; }
|
||||||
|
[Parameter] public EventCallback<string> OnViewQuery { get; set; }
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Method | Route | Description |
|
||||||
|
|--------|-------|-------------|
|
||||||
|
| GET | `/api/pipelines` | Get all pipeline names |
|
||||||
|
| GET | `/api/pipelines/{name}` | Get pipeline configuration |
|
||||||
|
| GET | `/api/pipelines/{name}/status` | Get schedule status summary |
|
||||||
|
| GET | `/api/pipelines/{name}/executions?count=10` | Get recent executions |
|
||||||
|
|
||||||
|
## Implementation Notes
|
||||||
|
|
||||||
|
### Override Detection
|
||||||
|
To show "Default" vs "Override" for schedule settings:
|
||||||
|
```csharp
|
||||||
|
bool IsOverride(int? pipelineValue, int defaultValue) =>
|
||||||
|
pipelineValue.HasValue && pipelineValue.Value != defaultValue;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Next Required Calculation
|
||||||
|
```csharp
|
||||||
|
DateTime? nextRequired = lastSuccessful?.AddMinutes(intervalMinutes);
|
||||||
|
bool isOverdue = nextRequired.HasValue && nextRequired.Value < DateTime.UtcNow;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Query Truncation
|
||||||
|
```csharp
|
||||||
|
string Truncate(string sql, int maxLength = 100) =>
|
||||||
|
sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
|
||||||
|
```
|
||||||
|
|
||||||
|
### Connection Type Badge Colors
|
||||||
|
- JDE: Blue (`BadgeStyle.Info`)
|
||||||
|
- CMS: Green (`BadgeStyle.Success`)
|
||||||
|
- GIW: Orange (`BadgeStyle.Warning`)
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- Radzen.Blazor (already installed)
|
||||||
|
- Existing `IDataUpdateRepository`
|
||||||
|
- Existing `pipelines.json` configuration
|
||||||
|
- Existing `UpdateTypes` enum (Hourly, Daily, Mass)
|
||||||
Reference in New Issue
Block a user