docs: add ETL pipeline viewer component design
Design document for admin monitoring component that visualizes: - Pipeline configuration from pipelines.json - Execution status (last run, next required, overdue) - Recent execution history (last 10 per schedule type) - Schedule sections for Mass/Daily/Hourly with defaults vs overrides
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,319 @@
|
||||
# ETL Pipeline Viewer Component Design
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status.
|
||||
|
||||
**Architecture:** Single-page admin component with pipeline selector, execution status tables, and vertical configuration sections for each schedule type (Mass, Daily, Hourly).
|
||||
|
||||
**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataUpdate repository
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A read-only admin monitoring page that displays:
|
||||
1. Pipeline configuration details from `pipelines.json`
|
||||
2. Execution status (last run, next required, overdue status)
|
||||
3. Recent execution history (last 10 runs per schedule type)
|
||||
|
||||
## Component Structure
|
||||
|
||||
### Page Layout
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ETL Pipeline Configuration Viewer │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Pipeline: [Dropdown - Alphabetical list] │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ SCHEDULE STATUS SUMMARY (Table 1) │
|
||||
│ ┌─────────┬────────────────────┬─────────────────┬────────┐ │
|
||||
│ │ Type │ Last Successful │ Next Required │ Status │ │
|
||||
│ ├─────────┼────────────────────┼─────────────────┼────────┤ │
|
||||
│ │ Mass │ 2026-01-05 02:00 │ 2026-01-12 02:00│ ✓ OK │ │
|
||||
│ │ Daily │ 2026-01-07 01:00 │ 2026-01-08 01:00│ ✓ OK │ │
|
||||
│ │ Hourly │ 2026-01-07 10:00 │ 2026-01-07 11:00│ ⚠ Over │ │
|
||||
│ └─────────┴────────────────────┴─────────────────┴────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ RECENT EXECUTION HISTORY (Table 2 - Last 10 per type) │
|
||||
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
|
||||
│ │ Type │ Start │ End │ Duration│ Rows│ Result│ │
|
||||
│ ├─────────┼────────────┼──────────┼─────────┼─────┼───────┤ │
|
||||
│ │ Hourly │ 10:00 AM │ 10:02 AM │ 2m 15s │1,247│ ✓ │ │
|
||||
│ │ Hourly │ 09:00 AM │ 09:01 AM │ 1m 42s │ 892 │ ✓ │ │
|
||||
│ │ ... │ │ │ │ │ │ │
|
||||
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ COMMON PIPELINE INFO │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Source: [JDE] Connection │ │
|
||||
│ │ Parameters: dateUpdated (jdeJulian), timeUpdated (jdeTime)│
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Destination: WorkOrder_Curr │ │
|
||||
│ │ Operation: Bulk Merge │ │
|
||||
│ │ Match: WorkOrderNumber, BranchCode │ │
|
||||
│ │ Exclude: WorkOrderNumber, BranchCode, LastUpdateDt │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Pre-Scripts: 0 scripts │ │
|
||||
│ │ Post-Scripts: 3 scripts [View] │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ MASS REFRESH [Enabled] │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Schedule Settings │ │
|
||||
│ │ Interval: 10080 min (7 days) [Default] │ │
|
||||
│ │ Pre-Purge: Yes [Default] │ │
|
||||
│ │ Re-Index: Yes [Default] │ │
|
||||
│ ├─────────────────────────────────────────────────────────┤ │
|
||||
│ │ Query: SELECT wo.WADOCO AS Work... [View Full Query] │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ DAILY REFRESH [Enabled] │
|
||||
│ └─ (Same structure as Mass) │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ HOURLY REFRESH [Enabled] │
|
||||
│ └─ (Same structure as Mass) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Files to Create
|
||||
|
||||
### Client Project (`src/JdeScoping.Client/`)
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `Pages/Admin/PipelineViewer.razor` | Main page component |
|
||||
| `Components/Admin/PipelineScheduleSection.razor` | Reusable schedule section (Mass/Daily/Hourly) |
|
||||
| `Components/Admin/SqlQueryModal.razor` | Modal for displaying SQL queries/scripts |
|
||||
| `Services/IPipelineConfigService.cs` | Interface for pipeline config access |
|
||||
| `Services/PipelineConfigService.cs` | Implementation - loads pipelines.json |
|
||||
| `Services/IPipelineStatusService.cs` | Interface for execution status |
|
||||
| `Services/PipelineStatusService.cs` | Implementation - calls API for status |
|
||||
| `Models/PipelineScheduleStatus.cs` | View model for schedule status row |
|
||||
| `Models/PipelineExecution.cs` | View model for execution history row |
|
||||
|
||||
### API Project (`src/JdeScoping.Api/`)
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `Controllers/PipelineController.cs` | API endpoints for pipeline config and status |
|
||||
|
||||
### DataSync Project (`src/JdeScoping.DataSync/`)
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `Contracts/IDataUpdateRepository.cs` | Add `GetRecentUpdatesAsync` method |
|
||||
| `Services/DataUpdateRepository.cs` | Implement new method |
|
||||
|
||||
## Data Models
|
||||
|
||||
### View Models (Client)
|
||||
|
||||
```csharp
|
||||
public record PipelineScheduleStatus(
|
||||
UpdateTypes ScheduleType,
|
||||
DateTime? LastSuccessfulRun,
|
||||
DateTime? NextRequiredRun,
|
||||
bool IsOverdue,
|
||||
int IntervalMinutes);
|
||||
|
||||
public record PipelineExecution(
|
||||
UpdateTypes ScheduleType,
|
||||
DateTime StartTime,
|
||||
DateTime? EndTime,
|
||||
TimeSpan Duration,
|
||||
long RecordCount,
|
||||
bool WasSuccessful);
|
||||
```
|
||||
|
||||
### Config Models (reuse or create in Contracts)
|
||||
|
||||
```csharp
|
||||
public record PipelinesRoot(
|
||||
PipelineSettings? Settings,
|
||||
ScheduleDefaults? ScheduleDefaults,
|
||||
Dictionary<string, PipelineConfig> Pipelines);
|
||||
|
||||
public record ScheduleDefaults(
|
||||
ScheduleConfig Mass,
|
||||
ScheduleConfig Daily,
|
||||
ScheduleConfig Hourly);
|
||||
|
||||
public record PipelineConfig(
|
||||
SourceConfig Source,
|
||||
DestinationConfig Destination,
|
||||
PipelineSchedules? Schedules,
|
||||
List<string>? PreScripts,
|
||||
List<string>? PostScripts);
|
||||
|
||||
public record SourceConfig(
|
||||
string Connection,
|
||||
string Query,
|
||||
string? MassQuery,
|
||||
Dictionary<string, ParameterConfig>? Parameters);
|
||||
|
||||
public record ParameterConfig(
|
||||
string Name,
|
||||
string? Format,
|
||||
string? Source);
|
||||
|
||||
public record DestinationConfig(
|
||||
string Table,
|
||||
string[]? MatchColumns,
|
||||
string[]? ExcludeFromUpdate);
|
||||
|
||||
public record PipelineSchedules(
|
||||
ScheduleConfig? Mass,
|
||||
ScheduleConfig? Daily,
|
||||
ScheduleConfig? Hourly);
|
||||
|
||||
public record ScheduleConfig(
|
||||
bool? Enabled,
|
||||
int? IntervalMinutes,
|
||||
bool? PrePurge,
|
||||
bool? ReIndex);
|
||||
```
|
||||
|
||||
## Service Interfaces
|
||||
|
||||
### IPipelineConfigService
|
||||
|
||||
```csharp
|
||||
public interface IPipelineConfigService
|
||||
{
|
||||
/// <summary>
|
||||
/// Gets the full pipelines configuration.
|
||||
/// </summary>
|
||||
Task<PipelinesRoot> GetPipelinesConfigAsync();
|
||||
|
||||
/// <summary>
|
||||
/// Gets configuration for a specific pipeline.
|
||||
/// </summary>
|
||||
Task<PipelineConfig?> GetPipelineAsync(string pipelineName);
|
||||
|
||||
/// <summary>
|
||||
/// Gets list of all pipeline names (sorted alphabetically).
|
||||
/// </summary>
|
||||
IEnumerable<string> GetPipelineNames();
|
||||
|
||||
/// <summary>
|
||||
/// Gets schedule defaults for computing effective values.
|
||||
/// </summary>
|
||||
ScheduleDefaults GetScheduleDefaults();
|
||||
}
|
||||
```
|
||||
|
||||
### IPipelineStatusService
|
||||
|
||||
```csharp
|
||||
public interface IPipelineStatusService
|
||||
{
|
||||
/// <summary>
|
||||
/// Gets schedule status summary for a pipeline (one row per schedule type).
|
||||
/// </summary>
|
||||
Task<List<PipelineScheduleStatus>> GetScheduleStatusAsync(
|
||||
string pipelineName,
|
||||
CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Gets recent execution history for a pipeline.
|
||||
/// </summary>
|
||||
Task<List<PipelineExecution>> GetRecentExecutionsAsync(
|
||||
string pipelineName,
|
||||
int count = 10,
|
||||
CancellationToken cancellationToken = default);
|
||||
}
|
||||
```
|
||||
|
||||
### IDataUpdateRepository (addition)
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Gets the last N execution records for a specific table.
|
||||
/// </summary>
|
||||
Task<List<DataUpdate>> GetRecentUpdatesAsync(
|
||||
string tableName,
|
||||
int count = 10,
|
||||
CancellationToken cancellationToken = default);
|
||||
```
|
||||
|
||||
## SQL Query Modal Component
|
||||
|
||||
### Features
|
||||
- Large modal (80% viewport width)
|
||||
- Monospace font display (`<pre><code>`)
|
||||
- Basic SQL formatting (line breaks at clauses)
|
||||
- Copy to Clipboard button
|
||||
- Dynamic title based on context
|
||||
|
||||
### Usage
|
||||
```razor
|
||||
<SqlQueryModal @bind-Visible="_showModal"
|
||||
Title="Mass Query - WorkOrder_Curr"
|
||||
Sql="@_selectedSql" />
|
||||
```
|
||||
|
||||
## Schedule Section Component
|
||||
|
||||
### Features
|
||||
- Header with schedule type name and Enabled/Disabled badge
|
||||
- Settings table showing:
|
||||
- Interval (with "Default" or "Override" indicator)
|
||||
- Pre-Purge flag
|
||||
- Re-Index flag
|
||||
- Query preview (truncated to ~100 chars)
|
||||
- "View Full Query" button
|
||||
|
||||
### Props
|
||||
```csharp
|
||||
[Parameter] public string ScheduleType { get; set; } // "Mass", "Daily", "Hourly"
|
||||
[Parameter] public ScheduleConfig? Config { get; set; }
|
||||
[Parameter] public ScheduleConfig DefaultConfig { get; set; }
|
||||
[Parameter] public string? Query { get; set; }
|
||||
[Parameter] public EventCallback<string> OnViewQuery { get; set; }
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Method | Route | Description |
|
||||
|--------|-------|-------------|
|
||||
| GET | `/api/pipelines` | Get all pipeline names |
|
||||
| GET | `/api/pipelines/{name}` | Get pipeline configuration |
|
||||
| GET | `/api/pipelines/{name}/status` | Get schedule status summary |
|
||||
| GET | `/api/pipelines/{name}/executions?count=10` | Get recent executions |
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Override Detection
|
||||
To show "Default" vs "Override" for schedule settings:
|
||||
```csharp
|
||||
bool IsOverride(int? pipelineValue, int defaultValue) =>
|
||||
pipelineValue.HasValue && pipelineValue.Value != defaultValue;
|
||||
```
|
||||
|
||||
### Next Required Calculation
|
||||
```csharp
|
||||
DateTime? nextRequired = lastSuccessful?.AddMinutes(intervalMinutes);
|
||||
bool isOverdue = nextRequired.HasValue && nextRequired.Value < DateTime.UtcNow;
|
||||
```
|
||||
|
||||
### Query Truncation
|
||||
```csharp
|
||||
string Truncate(string sql, int maxLength = 100) =>
|
||||
sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
|
||||
```
|
||||
|
||||
### Connection Type Badge Colors
|
||||
- JDE: Blue (`BadgeStyle.Info`)
|
||||
- CMS: Green (`BadgeStyle.Success`)
|
||||
- GIW: Orange (`BadgeStyle.Warning`)
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Radzen.Blazor (already installed)
|
||||
- Existing `IDataUpdateRepository`
|
||||
- Existing `pipelines.json` configuration
|
||||
- Existing `UpdateTypes` enum (Hourly, Daily, Mass)
|
||||
Reference in New Issue
Block a user