Joseph Doherty
9afecca957
Addressed CLEAN architecture and best practices feedback: - API loads config, returns DTO (not client loading JSON directly) - Reuse existing DataSync Configuration types - Add UpdateType filter to GetRecentUpdatesAsync - Reuse existing IsOverdue() with grace period - Add LastRun + LastRunWasSuccessful (not just LastSuccessfulRun) - Follow ApiRoutes + ApiClientBase patterns - Make Duration nullable for in-progress runs - Use UpdateTypes enum instead of string - Add [Authorize] to page and controller
489 lines
20 KiB
Markdown
489 lines
20 KiB
Markdown
# ETL Pipeline Viewer Component Design
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status.
|
|
|
|
**Architecture:** Server-side API loads pipeline config and exposes DTOs; Blazor client consumes via typed HttpClient following existing ApiClientBase pattern.
|
|
|
|
**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataSync config types, existing DataUpdateRepository
|
|
|
|
---
|
|
|
|
## Codex Review Feedback (Addressed)
|
|
|
|
| Issue | Resolution |
|
|
|-------|------------|
|
|
| Client loading pipelines.json directly breaks CLEAN | API loads config, returns DTO to client |
|
|
| Duplicate config records diverge from DataSync | Reuse existing DataSync Configuration types |
|
|
| GetRecentUpdatesAsync missing UpdateType filter | Add UpdateType parameter to signature |
|
|
| Status/overdue logic diverges from DataSync | Reuse existing `DataUpdateRepository.IsOverdue()` |
|
|
| Only LastSuccessfulRun, hiding failures | Add LastRun + LastRunWasSuccessful |
|
|
| API patterns don't match ApiRoutes + ApiClientBase | Add `ApiRoutes.Pipelines`, implement `PipelineApiClient` |
|
|
| Duration non-nullable for in-progress | Make Duration nullable |
|
|
| ScheduleType as string | Use `UpdateTypes` enum |
|
|
| Missing authorization | Add `[Authorize]` to page and controller |
|
|
|
|
---
|
|
|
|
## Component Structure
|
|
|
|
### Page Layout
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ ETL Pipeline Configuration Viewer [Authorize] │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Pipeline: [Dropdown - Alphabetical list] │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ SCHEDULE STATUS SUMMARY │
|
|
│ ┌─────────┬────────────┬────────────┬─────────────┬───────┐ │
|
|
│ │ Type │ Last Run │ Success? │ Next Req. │Status │ │
|
|
│ ├─────────┼────────────┼────────────┼─────────────┼───────┤ │
|
|
│ │ Mass │ 01/05 02:00│ ✓ │ 01/12 02:00 │ OK │ │
|
|
│ │ Daily │ 01/07 01:00│ ✓ │ 01/08 01:00 │ OK │ │
|
|
│ │ Hourly │ 01/07 10:00│ ✗ │ 01/07 11:00 │ Over │ │
|
|
│ └─────────┴────────────┴────────────┴─────────────┴───────┘ │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ RECENT EXECUTION HISTORY (Last 10 per type) │
|
|
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
|
|
│ │ Type │ Start │ End │ Duration│ Rows│ Result│ │
|
|
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ COMMON PIPELINE INFO │
|
|
│ [Source Card] [Destination Card] [Scripts Card] │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ MASS REFRESH [Enabled] │
|
|
│ [Schedule Settings] [Query Preview + Modal] │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ DAILY REFRESH [Enabled] │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ HOURLY REFRESH [Enabled] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
### CLEAN Architecture Layers
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PRESENTATION (JdeScoping.Client) │
|
|
│ - PipelineViewer.razor (page) │
|
|
│ - PipelineScheduleSection.razor (component) │
|
|
│ - SqlQueryModal.razor (component) │
|
|
│ - PipelineApiClient : ApiClientBase │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│ HTTP via ApiRoutes.Pipelines
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ APPLICATION (JdeScoping.Api) │
|
|
│ - PipelineController │
|
|
│ - Returns DTOs from Core.ApiContracts │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│ Depends on
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ INFRASTRUCTURE (JdeScoping.DataSync) │
|
|
│ - IEtlPipelineFactory (existing - loads pipelines.json) │
|
|
│ - IDataUpdateRepository (add GetRecentUpdatesAsync) │
|
|
│ - Reuse existing Configuration/* types │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│ Depends on
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ DOMAIN (JdeScoping.Core) │
|
|
│ - ApiRoutes.Pipelines (route constants) │
|
|
│ - ApiContracts.Pipelines/* (DTOs) │
|
|
│ - Models.Enums.UpdateTypes (existing) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Files to Create/Modify
|
|
|
|
### Core Project (`src/JdeScoping.Core/`)
|
|
|
|
| File | Action | Description |
|
|
|------|--------|-------------|
|
|
| `ApiContracts/ApiRoutes.cs` | Modify | Add `Pipelines` routes |
|
|
| `ApiContracts/Pipelines/PipelineListResponse.cs` | Create | List of pipeline names |
|
|
| `ApiContracts/Pipelines/PipelineConfigDto.cs` | Create | Pipeline config DTO (sanitized) |
|
|
| `ApiContracts/Pipelines/PipelineStatusDto.cs` | Create | Schedule status DTO |
|
|
| `ApiContracts/Pipelines/PipelineExecutionDto.cs` | Create | Execution history DTO |
|
|
|
|
### DataSync Project (`src/JdeScoping.DataSync/`)
|
|
|
|
| File | Action | Description |
|
|
|------|--------|-------------|
|
|
| `Contracts/IDataUpdateRepository.cs` | Modify | Add `GetRecentUpdatesAsync` |
|
|
| `Services/DataUpdateRepository.cs` | Modify | Implement new method |
|
|
|
|
### API Project (`src/JdeScoping.Api/`)
|
|
|
|
| File | Action | Description |
|
|
|------|--------|-------------|
|
|
| `Controllers/PipelineController.cs` | Create | API endpoints |
|
|
|
|
### Client Project (`src/JdeScoping.Client/`)
|
|
|
|
| File | Action | Description |
|
|
|------|--------|-------------|
|
|
| `Services/IPipelineApiClient.cs` | Create | Client interface |
|
|
| `Services/PipelineApiClient.cs` | Create | Client implementation |
|
|
| `Pages/Admin/PipelineViewer.razor` | Create | Main page |
|
|
| `Components/Admin/PipelineScheduleSection.razor` | Create | Schedule section |
|
|
| `Components/Admin/SqlQueryModal.razor` | Create | SQL modal |
|
|
|
|
---
|
|
|
|
## API Routes (add to `ApiRoutes.cs`)
|
|
|
|
```csharp
|
|
/// <summary>
|
|
/// Routes for pipeline configuration API endpoints.
|
|
/// </summary>
|
|
public static class Pipelines
|
|
{
|
|
/// <summary>Base route for pipeline endpoints.</summary>
|
|
public const string Base = "api/pipelines";
|
|
|
|
/// <summary>Route template for getting a pipeline by name.</summary>
|
|
public const string ByName = "{name}";
|
|
|
|
/// <summary>Route template for getting pipeline status.</summary>
|
|
public const string Status = "{name}/status";
|
|
|
|
/// <summary>Route template for getting pipeline executions.</summary>
|
|
public const string Executions = "{name}/executions";
|
|
|
|
/// <summary>Builds the route to get a specific pipeline config.</summary>
|
|
public static string GetByName(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}";
|
|
|
|
/// <summary>Builds the route to get pipeline status.</summary>
|
|
public static string GetStatus(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}/status";
|
|
|
|
/// <summary>Builds the route to get pipeline executions.</summary>
|
|
public static string GetExecutions(string name, int count = 10) =>
|
|
$"api/pipelines/{Uri.EscapeDataString(name)}/executions?count={count}";
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## DTOs (in `Core/ApiContracts/Pipelines/`)
|
|
|
|
### PipelineListResponse.cs
|
|
|
|
```csharp
|
|
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
|
|
|
/// <summary>
|
|
/// Response containing list of available pipeline names.
|
|
/// </summary>
|
|
public record PipelineListResponse(List<string> PipelineNames);
|
|
```
|
|
|
|
### PipelineConfigDto.cs
|
|
|
|
```csharp
|
|
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
|
|
|
/// <summary>
|
|
/// Pipeline configuration DTO for display (sanitized - no raw SQL exposed by default).
|
|
/// </summary>
|
|
public record PipelineConfigDto(
|
|
string Name,
|
|
PipelineSourceDto Source,
|
|
PipelineDestinationDto Destination,
|
|
PipelineSchedulesDto Schedules,
|
|
int PreScriptCount,
|
|
int PostScriptCount,
|
|
List<string>? PreScripts, // Only populated if user has admin role
|
|
List<string>? PostScripts); // Only populated if user has admin role
|
|
|
|
public record PipelineSourceDto(
|
|
string Connection,
|
|
string? QueryPreview, // First 100 chars
|
|
string? MassQueryPreview, // First 100 chars
|
|
string? Query, // Full query - only if admin
|
|
string? MassQuery, // Full query - only if admin
|
|
List<PipelineParameterDto> Parameters);
|
|
|
|
public record PipelineParameterDto(
|
|
string Name,
|
|
string? Format,
|
|
string Source);
|
|
|
|
public record PipelineDestinationDto(
|
|
string Table,
|
|
string OperationType, // "BulkMerge" or "BulkImport"
|
|
List<string>? MatchColumns,
|
|
List<string>? ExcludeFromUpdate);
|
|
|
|
public record PipelineSchedulesDto(
|
|
PipelineScheduleDto Mass,
|
|
PipelineScheduleDto Daily,
|
|
PipelineScheduleDto Hourly);
|
|
|
|
public record PipelineScheduleDto(
|
|
bool Enabled,
|
|
int IntervalMinutes,
|
|
bool PrePurge,
|
|
bool ReIndex,
|
|
bool IntervalIsOverride,
|
|
bool PrePurgeIsOverride,
|
|
bool ReIndexIsOverride);
|
|
```
|
|
|
|
### PipelineStatusDto.cs
|
|
|
|
```csharp
|
|
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
|
|
|
using JdeScoping.Core.Models.Enums;
|
|
|
|
/// <summary>
|
|
/// Pipeline schedule status for each update type.
|
|
/// </summary>
|
|
public record PipelineStatusResponse(List<PipelineScheduleStatusDto> Statuses);
|
|
|
|
public record PipelineScheduleStatusDto(
|
|
UpdateTypes ScheduleType,
|
|
DateTime? LastRun,
|
|
bool LastRunWasSuccessful,
|
|
DateTime? LastSuccessfulRun,
|
|
DateTime? NextRequiredRun,
|
|
bool IsOverdue,
|
|
int IntervalMinutes);
|
|
```
|
|
|
|
### PipelineExecutionDto.cs
|
|
|
|
```csharp
|
|
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
|
|
|
using JdeScoping.Core.Models.Enums;
|
|
|
|
/// <summary>
|
|
/// Pipeline execution history.
|
|
/// </summary>
|
|
public record PipelineExecutionsResponse(List<PipelineExecutionDto> Executions);
|
|
|
|
public record PipelineExecutionDto(
|
|
UpdateTypes ScheduleType,
|
|
DateTime StartTime,
|
|
DateTime? EndTime,
|
|
TimeSpan? Duration, // Nullable for in-progress runs
|
|
long RecordCount,
|
|
bool WasSuccessful);
|
|
```
|
|
|
|
---
|
|
|
|
## Repository Update (IDataUpdateRepository)
|
|
|
|
```csharp
|
|
/// <summary>
|
|
/// Gets the last N execution records for a specific table and optional update type.
|
|
/// </summary>
|
|
/// <param name="tableName">The table name to filter by.</param>
|
|
/// <param name="updateType">Optional update type filter. If null, returns all types.</param>
|
|
/// <param name="count">Maximum records to return per update type.</param>
|
|
/// <param name="cancellationToken">Cancellation token.</param>
|
|
/// <returns>List of DataUpdate records ordered by StartDt descending.</returns>
|
|
Task<List<DataUpdate>> GetRecentUpdatesAsync(
|
|
string tableName,
|
|
UpdateTypes? updateType = null,
|
|
int count = 10,
|
|
CancellationToken cancellationToken = default);
|
|
|
|
/// <summary>
|
|
/// Gets the last run (successful or not) for each update type for a table.
|
|
/// </summary>
|
|
/// <param name="tableName">The table name.</param>
|
|
/// <param name="cancellationToken">Cancellation token.</param>
|
|
/// <returns>Dictionary keyed by UpdateType.</returns>
|
|
Task<Dictionary<UpdateTypes, DataUpdate>> GetLastRunsAsync(
|
|
string tableName,
|
|
CancellationToken cancellationToken = default);
|
|
```
|
|
|
|
---
|
|
|
|
## API Controller
|
|
|
|
```csharp
|
|
[ApiController]
|
|
[Route(ApiRoutes.Pipelines.Base)]
|
|
[Authorize]
|
|
public class PipelineController : ControllerBase
|
|
{
|
|
private readonly IEtlPipelineFactory _pipelineFactory;
|
|
private readonly IDataUpdateRepository _dataUpdateRepository;
|
|
|
|
[HttpGet]
|
|
public ActionResult<PipelineListResponse> GetPipelineNames()
|
|
{
|
|
var names = _pipelineFactory.GetAvailableTables()
|
|
.OrderBy(n => n)
|
|
.ToList();
|
|
return Ok(new PipelineListResponse(names));
|
|
}
|
|
|
|
[HttpGet(ApiRoutes.Pipelines.ByName)]
|
|
public ActionResult<PipelineConfigDto> GetPipeline(string name)
|
|
{
|
|
// Map from internal config to DTO
|
|
// Truncate queries for preview, include full if admin
|
|
}
|
|
|
|
[HttpGet(ApiRoutes.Pipelines.Status)]
|
|
public async Task<ActionResult<PipelineStatusResponse>> GetStatus(string name)
|
|
{
|
|
// Get last runs, calculate IsOverdue using DataUpdateRepository.IsOverdue()
|
|
}
|
|
|
|
[HttpGet(ApiRoutes.Pipelines.Executions)]
|
|
public async Task<ActionResult<PipelineExecutionsResponse>> GetExecutions(
|
|
string name, [FromQuery] int count = 10)
|
|
{
|
|
// Get recent updates from repository
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Client Implementation
|
|
|
|
### PipelineApiClient.cs
|
|
|
|
```csharp
|
|
public interface IPipelineApiClient
|
|
{
|
|
Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default);
|
|
Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default);
|
|
Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default);
|
|
Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default);
|
|
}
|
|
|
|
public class PipelineApiClient : ApiClientBase, IPipelineApiClient
|
|
{
|
|
public PipelineApiClient(HttpClient httpClient) : base(httpClient) { }
|
|
|
|
public Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default)
|
|
=> GetAsync<PipelineListResponse>(ApiRoutes.Pipelines.Base, ct);
|
|
|
|
public Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default)
|
|
=> GetAsync<PipelineConfigDto>(ApiRoutes.Pipelines.GetByName(name), ct);
|
|
|
|
public Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default)
|
|
=> GetAsync<PipelineStatusResponse>(ApiRoutes.Pipelines.GetStatus(name), ct);
|
|
|
|
public Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default)
|
|
=> GetAsync<PipelineExecutionsResponse>(ApiRoutes.Pipelines.GetExecutions(name, count), ct);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Blazor Components
|
|
|
|
### PipelineViewer.razor Structure
|
|
|
|
```razor
|
|
@page "/admin/pipeline-viewer"
|
|
@attribute [Authorize]
|
|
@inject IPipelineApiClient PipelineApi
|
|
|
|
<PageTitle>Pipeline Configuration Viewer</PageTitle>
|
|
|
|
<RadzenText TextStyle="TextStyle.H4">ETL Pipeline Configuration Viewer</RadzenText>
|
|
|
|
<!-- Pipeline Selector -->
|
|
<RadzenDropDown @bind-Value="_selectedPipeline"
|
|
Data="_pipelineNames"
|
|
Change="OnPipelineChanged" />
|
|
|
|
@if (_config is not null)
|
|
{
|
|
<!-- Status Summary Table -->
|
|
<!-- Execution History Table -->
|
|
<!-- Common Info Cards -->
|
|
<!-- Schedule Sections (Mass, Daily, Hourly) -->
|
|
}
|
|
|
|
<SqlQueryModal @bind-Visible="_showSqlModal" Title="@_sqlModalTitle" Sql="@_sqlModalContent" />
|
|
```
|
|
|
|
### SqlQueryModal.razor
|
|
|
|
- Large modal (80% width)
|
|
- `<pre><code>` with monospace font
|
|
- Copy to clipboard button
|
|
- Basic SQL formatting (line breaks at clauses)
|
|
|
|
### PipelineScheduleSection.razor
|
|
|
|
- Props: ScheduleType (UpdateTypes), Config (PipelineScheduleDto), Query, OnViewQuery callback
|
|
- Shows enabled badge
|
|
- Settings table with Default/Override indicators
|
|
- Query preview with "View Full" button
|
|
|
|
---
|
|
|
|
## Implementation Notes
|
|
|
|
### Override Detection (in API)
|
|
|
|
```csharp
|
|
bool IsOverride<T>(T? pipelineValue, T defaultValue) where T : struct =>
|
|
pipelineValue.HasValue && !pipelineValue.Value.Equals(defaultValue);
|
|
```
|
|
|
|
### Overdue Calculation (reuse existing)
|
|
|
|
```csharp
|
|
// Use DataUpdateRepository.IsOverdue() which includes 50% grace period
|
|
var isOverdue = DataUpdateRepository.IsOverdue(
|
|
lastSuccessfulRun, tableName, updateType, customIntervals);
|
|
```
|
|
|
|
### Connection Type Badge Colors
|
|
|
|
| Connection | Badge Style |
|
|
|------------|-------------|
|
|
| jde | `BadgeStyle.Info` (blue) |
|
|
| cms | `BadgeStyle.Success` (green) |
|
|
| giw | `BadgeStyle.Warning` (orange) |
|
|
|
|
### SQL Query Truncation
|
|
|
|
```csharp
|
|
static string Truncate(string? sql, int maxLength = 100) =>
|
|
sql is null ? "" :
|
|
sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
|
|
```
|
|
|
|
---
|
|
|
|
## Security Considerations
|
|
|
|
1. **Authorization**: Page and API require `[Authorize]`
|
|
2. **SQL Visibility**: Full SQL/scripts only exposed if user has appropriate permissions (consider admin-only)
|
|
3. **No write operations**: This is read-only, no mutations exposed
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
- Radzen.Blazor (existing)
|
|
- Existing `IEtlPipelineFactory` and `IDataUpdateRepository`
|
|
- Existing `pipelines.json` configuration
|
|
- Existing `UpdateTypes` enum
|
|
- Existing `ApiClientBase` pattern
|