docs: update pipeline viewer design based on Codex review
Addressed CLEAN architecture and best practices feedback: - API loads config, returns DTO (not client loading JSON directly) - Reuse existing DataSync Configuration types - Add UpdateType filter to GetRecentUpdatesAsync - Reuse existing IsOverdue() with grace period - Add LastRun + LastRunWasSuccessful (not just LastSuccessfulRun) - Follow ApiRoutes + ApiClientBase patterns - Make Duration nullable for in-progress runs - Use UpdateTypes enum instead of string - Add [Authorize] to page and controller
This commit is contained in:
Joseph Doherty
@@ -4,18 +4,27 @@
|
||||
|
||||
**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status.
|
||||
|
||||
**Architecture:** Single-page admin component with pipeline selector, execution status tables, and vertical configuration sections for each schedule type (Mass, Daily, Hourly).
|
||||
**Architecture:** Server-side API loads pipeline config and exposes DTOs; Blazor client consumes via typed HttpClient following existing ApiClientBase pattern.
|
||||
|
||||
**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataUpdate repository
|
||||
**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataSync config types, existing DataUpdateRepository
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Codex Review Feedback (Addressed)
|
||||
|
||||
A read-only admin monitoring page that displays:
|
||||
1. Pipeline configuration details from `pipelines.json`
|
||||
2. Execution status (last run, next required, overdue status)
|
||||
3. Recent execution history (last 10 runs per schedule type)
|
||||
| Issue | Resolution |
|
||||
|-------|------------|
|
||||
| Client loading pipelines.json directly breaks CLEAN | API loads config, returns DTO to client |
|
||||
| Duplicate config records diverge from DataSync | Reuse existing DataSync Configuration types |
|
||||
| GetRecentUpdatesAsync missing UpdateType filter | Add UpdateType parameter to signature |
|
||||
| Status/overdue logic diverges from DataSync | Reuse existing `DataUpdateRepository.IsOverdue()` |
|
||||
| Only LastSuccessfulRun, hiding failures | Add LastRun + LastRunWasSuccessful |
|
||||
| API patterns don't match ApiRoutes + ApiClientBase | Add `ApiRoutes.Pipelines`, implement `PipelineApiClient` |
|
||||
| Duration non-nullable for in-progress | Make Duration nullable |
|
||||
| ScheduleType as string | Use `UpdateTypes` enum |
|
||||
| Missing authorization | Add `[Authorize]` to page and controller |
|
||||
|
||||
---
|
||||
|
||||
## Component Structure
|
||||
|
||||
@@ -23,297 +32,457 @@ A read-only admin monitoring page that displays:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ETL Pipeline Configuration Viewer │
|
||||
│ ETL Pipeline Configuration Viewer [Authorize] │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Pipeline: [Dropdown - Alphabetical list] │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ SCHEDULE STATUS SUMMARY (Table 1) │
|
||||
│ ┌─────────┬────────────────────┬─────────────────┬────────┐ │
|
||||
│ │ Type │ Last Successful │ Next Required │ Status │ │
|
||||
│ ├─────────┼────────────────────┼─────────────────┼────────┤ │
|
||||
│ │ Mass │ 2026-01-05 02:00 │ 2026-01-12 02:00│ ✓ OK │ │
|
||||
│ │ Daily │ 2026-01-07 01:00 │ 2026-01-08 01:00│ ✓ OK │ │
|
||||
│ │ Hourly │ 2026-01-07 10:00 │ 2026-01-07 11:00│ ⚠ Over │ │
|
||||
│ └─────────┴────────────────────┴─────────────────┴────────┘ │
|
||||
│ SCHEDULE STATUS SUMMARY │
|
||||
│ ┌─────────┬────────────┬────────────┬─────────────┬───────┐ │
|
||||
│ │ Type │ Last Run │ Success? │ Next Req. │Status │ │
|
||||
│ ├─────────┼────────────┼────────────┼─────────────┼───────┤ │
|
||||
│ │ Mass │ 01/05 02:00│ ✓ │ 01/12 02:00 │ OK │ │
|
||||
│ │ Daily │ 01/07 01:00│ ✓ │ 01/08 01:00 │ OK │ │
|
||||
│ │ Hourly │ 01/07 10:00│ ✗ │ 01/07 11:00 │ Over │ │
|
||||
│ └─────────┴────────────┴────────────┴─────────────┴───────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ RECENT EXECUTION HISTORY (Table 2 - Last 10 per type) │
|
||||
│ RECENT EXECUTION HISTORY (Last 10 per type) │
|
||||
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
|
||||
│ │ Type │ Start │ End │ Duration│ Rows│ Result│ │
|
||||
│ ├─────────┼────────────┼──────────┼─────────┼─────┼───────┤ │
|
||||
│ │ Hourly │ 10:00 AM │ 10:02 AM │ 2m 15s │1,247│ ✓ │ │
|
||||
│ │ Hourly │ 09:00 AM │ 09:01 AM │ 1m 42s │ 892 │ ✓ │ │
|
||||
│ │ ... │ │ │ │ │ │ │
|
||||
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ COMMON PIPELINE INFO │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Source: [JDE] Connection │ │
|
||||
│ │ Parameters: dateUpdated (jdeJulian), timeUpdated (jdeTime)│
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Destination: WorkOrder_Curr │ │
|
||||
│ │ Operation: Bulk Merge │ │
|
||||
│ │ Match: WorkOrderNumber, BranchCode │ │
|
||||
│ │ Exclude: WorkOrderNumber, BranchCode, LastUpdateDt │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Pre-Scripts: 0 scripts │ │
|
||||
│ │ Post-Scripts: 3 scripts [View] │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ [Source Card] [Destination Card] [Scripts Card] │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ MASS REFRESH [Enabled] │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Schedule Settings │ │
|
||||
│ │ Interval: 10080 min (7 days) [Default] │ │
|
||||
│ │ Pre-Purge: Yes [Default] │ │
|
||||
│ │ Re-Index: Yes [Default] │ │
|
||||
│ ├─────────────────────────────────────────────────────────┤ │
|
||||
│ │ Query: SELECT wo.WADOCO AS Work... [View Full Query] │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ MASS REFRESH [Enabled] │
|
||||
│ [Schedule Settings] [Query Preview + Modal] │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ DAILY REFRESH [Enabled] │
|
||||
│ └─ (Same structure as Mass) │
|
||||
│ DAILY REFRESH [Enabled] │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ HOURLY REFRESH [Enabled] │
|
||||
│ └─ (Same structure as Mass) │
|
||||
│ HOURLY REFRESH [Enabled] │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Files to Create
|
||||
---
|
||||
|
||||
### Client Project (`src/JdeScoping.Client/`)
|
||||
## Architecture
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `Pages/Admin/PipelineViewer.razor` | Main page component |
|
||||
| `Components/Admin/PipelineScheduleSection.razor` | Reusable schedule section (Mass/Daily/Hourly) |
|
||||
| `Components/Admin/SqlQueryModal.razor` | Modal for displaying SQL queries/scripts |
|
||||
| `Services/IPipelineConfigService.cs` | Interface for pipeline config access |
|
||||
| `Services/PipelineConfigService.cs` | Implementation - loads pipelines.json |
|
||||
| `Services/IPipelineStatusService.cs` | Interface for execution status |
|
||||
| `Services/PipelineStatusService.cs` | Implementation - calls API for status |
|
||||
| `Models/PipelineScheduleStatus.cs` | View model for schedule status row |
|
||||
| `Models/PipelineExecution.cs` | View model for execution history row |
|
||||
### CLEAN Architecture Layers
|
||||
|
||||
### API Project (`src/JdeScoping.Api/`)
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PRESENTATION (JdeScoping.Client) │
|
||||
│ - PipelineViewer.razor (page) │
|
||||
│ - PipelineScheduleSection.razor (component) │
|
||||
│ - SqlQueryModal.razor (component) │
|
||||
│ - PipelineApiClient : ApiClientBase │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│ HTTP via ApiRoutes.Pipelines
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ APPLICATION (JdeScoping.Api) │
|
||||
│ - PipelineController │
|
||||
│ - Returns DTOs from Core.ApiContracts │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│ Depends on
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ INFRASTRUCTURE (JdeScoping.DataSync) │
|
||||
│ - IEtlPipelineFactory (existing - loads pipelines.json) │
|
||||
│ - IDataUpdateRepository (add GetRecentUpdatesAsync) │
|
||||
│ - Reuse existing Configuration/* types │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│ Depends on
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ DOMAIN (JdeScoping.Core) │
|
||||
│ - ApiRoutes.Pipelines (route constants) │
|
||||
│ - ApiContracts.Pipelines/* (DTOs) │
|
||||
│ - Models.Enums.UpdateTypes (existing) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `Controllers/PipelineController.cs` | API endpoints for pipeline config and status |
|
||||
---
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
### Core Project (`src/JdeScoping.Core/`)
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `ApiContracts/ApiRoutes.cs` | Modify | Add `Pipelines` routes |
|
||||
| `ApiContracts/Pipelines/PipelineListResponse.cs` | Create | List of pipeline names |
|
||||
| `ApiContracts/Pipelines/PipelineConfigDto.cs` | Create | Pipeline config DTO (sanitized) |
|
||||
| `ApiContracts/Pipelines/PipelineStatusDto.cs` | Create | Schedule status DTO |
|
||||
| `ApiContracts/Pipelines/PipelineExecutionDto.cs` | Create | Execution history DTO |
|
||||
|
||||
### DataSync Project (`src/JdeScoping.DataSync/`)
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `Contracts/IDataUpdateRepository.cs` | Add `GetRecentUpdatesAsync` method |
|
||||
| `Services/DataUpdateRepository.cs` | Implement new method |
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `Contracts/IDataUpdateRepository.cs` | Modify | Add `GetRecentUpdatesAsync` |
|
||||
| `Services/DataUpdateRepository.cs` | Modify | Implement new method |
|
||||
|
||||
## Data Models
|
||||
### API Project (`src/JdeScoping.Api/`)
|
||||
|
||||
### View Models (Client)
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `Controllers/PipelineController.cs` | Create | API endpoints |
|
||||
|
||||
### Client Project (`src/JdeScoping.Client/`)
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `Services/IPipelineApiClient.cs` | Create | Client interface |
|
||||
| `Services/PipelineApiClient.cs` | Create | Client implementation |
|
||||
| `Pages/Admin/PipelineViewer.razor` | Create | Main page |
|
||||
| `Components/Admin/PipelineScheduleSection.razor` | Create | Schedule section |
|
||||
| `Components/Admin/SqlQueryModal.razor` | Create | SQL modal |
|
||||
|
||||
---
|
||||
|
||||
## API Routes (add to `ApiRoutes.cs`)
|
||||
|
||||
```csharp
|
||||
public record PipelineScheduleStatus(
|
||||
/// <summary>
|
||||
/// Routes for pipeline configuration API endpoints.
|
||||
/// </summary>
|
||||
public static class Pipelines
|
||||
{
|
||||
/// <summary>Base route for pipeline endpoints.</summary>
|
||||
public const string Base = "api/pipelines";
|
||||
|
||||
/// <summary>Route template for getting a pipeline by name.</summary>
|
||||
public const string ByName = "{name}";
|
||||
|
||||
/// <summary>Route template for getting pipeline status.</summary>
|
||||
public const string Status = "{name}/status";
|
||||
|
||||
/// <summary>Route template for getting pipeline executions.</summary>
|
||||
public const string Executions = "{name}/executions";
|
||||
|
||||
/// <summary>Builds the route to get a specific pipeline config.</summary>
|
||||
public static string GetByName(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}";
|
||||
|
||||
/// <summary>Builds the route to get pipeline status.</summary>
|
||||
public static string GetStatus(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}/status";
|
||||
|
||||
/// <summary>Builds the route to get pipeline executions.</summary>
|
||||
public static string GetExecutions(string name, int count = 10) =>
|
||||
$"api/pipelines/{Uri.EscapeDataString(name)}/executions?count={count}";
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## DTOs (in `Core/ApiContracts/Pipelines/`)
|
||||
|
||||
### PipelineListResponse.cs
|
||||
|
||||
```csharp
|
||||
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
||||
|
||||
/// <summary>
|
||||
/// Response containing list of available pipeline names.
|
||||
/// </summary>
|
||||
public record PipelineListResponse(List<string> PipelineNames);
|
||||
```
|
||||
|
||||
### PipelineConfigDto.cs
|
||||
|
||||
```csharp
|
||||
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
||||
|
||||
/// <summary>
|
||||
/// Pipeline configuration DTO for display (sanitized - no raw SQL exposed by default).
|
||||
/// </summary>
|
||||
public record PipelineConfigDto(
|
||||
string Name,
|
||||
PipelineSourceDto Source,
|
||||
PipelineDestinationDto Destination,
|
||||
PipelineSchedulesDto Schedules,
|
||||
int PreScriptCount,
|
||||
int PostScriptCount,
|
||||
List<string>? PreScripts, // Only populated if user has admin role
|
||||
List<string>? PostScripts); // Only populated if user has admin role
|
||||
|
||||
public record PipelineSourceDto(
|
||||
string Connection,
|
||||
string? QueryPreview, // First 100 chars
|
||||
string? MassQueryPreview, // First 100 chars
|
||||
string? Query, // Full query - only if admin
|
||||
string? MassQuery, // Full query - only if admin
|
||||
List<PipelineParameterDto> Parameters);
|
||||
|
||||
public record PipelineParameterDto(
|
||||
string Name,
|
||||
string? Format,
|
||||
string Source);
|
||||
|
||||
public record PipelineDestinationDto(
|
||||
string Table,
|
||||
string OperationType, // "BulkMerge" or "BulkImport"
|
||||
List<string>? MatchColumns,
|
||||
List<string>? ExcludeFromUpdate);
|
||||
|
||||
public record PipelineSchedulesDto(
|
||||
PipelineScheduleDto Mass,
|
||||
PipelineScheduleDto Daily,
|
||||
PipelineScheduleDto Hourly);
|
||||
|
||||
public record PipelineScheduleDto(
|
||||
bool Enabled,
|
||||
int IntervalMinutes,
|
||||
bool PrePurge,
|
||||
bool ReIndex,
|
||||
bool IntervalIsOverride,
|
||||
bool PrePurgeIsOverride,
|
||||
bool ReIndexIsOverride);
|
||||
```
|
||||
|
||||
### PipelineStatusDto.cs
|
||||
|
||||
```csharp
|
||||
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
||||
|
||||
using JdeScoping.Core.Models.Enums;
|
||||
|
||||
/// <summary>
|
||||
/// Pipeline schedule status for each update type.
|
||||
/// </summary>
|
||||
public record PipelineStatusResponse(List<PipelineScheduleStatusDto> Statuses);
|
||||
|
||||
public record PipelineScheduleStatusDto(
|
||||
UpdateTypes ScheduleType,
|
||||
DateTime? LastRun,
|
||||
bool LastRunWasSuccessful,
|
||||
DateTime? LastSuccessfulRun,
|
||||
DateTime? NextRequiredRun,
|
||||
bool IsOverdue,
|
||||
int IntervalMinutes);
|
||||
```
|
||||
|
||||
public record PipelineExecution(
|
||||
### PipelineExecutionDto.cs
|
||||
|
||||
```csharp
|
||||
namespace JdeScoping.Core.ApiContracts.Pipelines;
|
||||
|
||||
using JdeScoping.Core.Models.Enums;
|
||||
|
||||
/// <summary>
|
||||
/// Pipeline execution history.
|
||||
/// </summary>
|
||||
public record PipelineExecutionsResponse(List<PipelineExecutionDto> Executions);
|
||||
|
||||
public record PipelineExecutionDto(
|
||||
UpdateTypes ScheduleType,
|
||||
DateTime StartTime,
|
||||
DateTime? EndTime,
|
||||
TimeSpan Duration,
|
||||
TimeSpan? Duration, // Nullable for in-progress runs
|
||||
long RecordCount,
|
||||
bool WasSuccessful);
|
||||
```
|
||||
|
||||
### Config Models (reuse or create in Contracts)
|
||||
---
|
||||
|
||||
```csharp
|
||||
public record PipelinesRoot(
|
||||
PipelineSettings? Settings,
|
||||
ScheduleDefaults? ScheduleDefaults,
|
||||
Dictionary<string, PipelineConfig> Pipelines);
|
||||
|
||||
public record ScheduleDefaults(
|
||||
ScheduleConfig Mass,
|
||||
ScheduleConfig Daily,
|
||||
ScheduleConfig Hourly);
|
||||
|
||||
public record PipelineConfig(
|
||||
SourceConfig Source,
|
||||
DestinationConfig Destination,
|
||||
PipelineSchedules? Schedules,
|
||||
List<string>? PreScripts,
|
||||
List<string>? PostScripts);
|
||||
|
||||
public record SourceConfig(
|
||||
string Connection,
|
||||
string Query,
|
||||
string? MassQuery,
|
||||
Dictionary<string, ParameterConfig>? Parameters);
|
||||
|
||||
public record ParameterConfig(
|
||||
string Name,
|
||||
string? Format,
|
||||
string? Source);
|
||||
|
||||
public record DestinationConfig(
|
||||
string Table,
|
||||
string[]? MatchColumns,
|
||||
string[]? ExcludeFromUpdate);
|
||||
|
||||
public record PipelineSchedules(
|
||||
ScheduleConfig? Mass,
|
||||
ScheduleConfig? Daily,
|
||||
ScheduleConfig? Hourly);
|
||||
|
||||
public record ScheduleConfig(
|
||||
bool? Enabled,
|
||||
int? IntervalMinutes,
|
||||
bool? PrePurge,
|
||||
bool? ReIndex);
|
||||
```
|
||||
|
||||
## Service Interfaces
|
||||
|
||||
### IPipelineConfigService
|
||||
|
||||
```csharp
|
||||
public interface IPipelineConfigService
|
||||
{
|
||||
/// <summary>
|
||||
/// Gets the full pipelines configuration.
|
||||
/// </summary>
|
||||
Task<PipelinesRoot> GetPipelinesConfigAsync();
|
||||
|
||||
/// <summary>
|
||||
/// Gets configuration for a specific pipeline.
|
||||
/// </summary>
|
||||
Task<PipelineConfig?> GetPipelineAsync(string pipelineName);
|
||||
|
||||
/// <summary>
|
||||
/// Gets list of all pipeline names (sorted alphabetically).
|
||||
/// </summary>
|
||||
IEnumerable<string> GetPipelineNames();
|
||||
|
||||
/// <summary>
|
||||
/// Gets schedule defaults for computing effective values.
|
||||
/// </summary>
|
||||
ScheduleDefaults GetScheduleDefaults();
|
||||
}
|
||||
```
|
||||
|
||||
### IPipelineStatusService
|
||||
|
||||
```csharp
|
||||
public interface IPipelineStatusService
|
||||
{
|
||||
/// <summary>
|
||||
/// Gets schedule status summary for a pipeline (one row per schedule type).
|
||||
/// </summary>
|
||||
Task<List<PipelineScheduleStatus>> GetScheduleStatusAsync(
|
||||
string pipelineName,
|
||||
CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Gets recent execution history for a pipeline.
|
||||
/// </summary>
|
||||
Task<List<PipelineExecution>> GetRecentExecutionsAsync(
|
||||
string pipelineName,
|
||||
int count = 10,
|
||||
CancellationToken cancellationToken = default);
|
||||
}
|
||||
```
|
||||
|
||||
### IDataUpdateRepository (addition)
|
||||
## Repository Update (IDataUpdateRepository)
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Gets the last N execution records for a specific table.
|
||||
/// Gets the last N execution records for a specific table and optional update type.
|
||||
/// </summary>
|
||||
/// <param name="tableName">The table name to filter by.</param>
|
||||
/// <param name="updateType">Optional update type filter. If null, returns all types.</param>
|
||||
/// <param name="count">Maximum records to return per update type.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>List of DataUpdate records ordered by StartDt descending.</returns>
|
||||
Task<List<DataUpdate>> GetRecentUpdatesAsync(
|
||||
string tableName,
|
||||
UpdateTypes? updateType = null,
|
||||
int count = 10,
|
||||
CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Gets the last run (successful or not) for each update type for a table.
|
||||
/// </summary>
|
||||
/// <param name="tableName">The table name.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Dictionary keyed by UpdateType.</returns>
|
||||
Task<Dictionary<UpdateTypes, DataUpdate>> GetLastRunsAsync(
|
||||
string tableName,
|
||||
CancellationToken cancellationToken = default);
|
||||
```
|
||||
|
||||
## SQL Query Modal Component
|
||||
---
|
||||
|
||||
### Features
|
||||
- Large modal (80% viewport width)
|
||||
- Monospace font display (`<pre><code>`)
|
||||
- Basic SQL formatting (line breaks at clauses)
|
||||
- Copy to Clipboard button
|
||||
- Dynamic title based on context
|
||||
## API Controller
|
||||
|
||||
### Usage
|
||||
```razor
|
||||
<SqlQueryModal @bind-Visible="_showModal"
|
||||
Title="Mass Query - WorkOrder_Curr"
|
||||
Sql="@_selectedSql" />
|
||||
```
|
||||
|
||||
## Schedule Section Component
|
||||
|
||||
### Features
|
||||
- Header with schedule type name and Enabled/Disabled badge
|
||||
- Settings table showing:
|
||||
- Interval (with "Default" or "Override" indicator)
|
||||
- Pre-Purge flag
|
||||
- Re-Index flag
|
||||
- Query preview (truncated to ~100 chars)
|
||||
- "View Full Query" button
|
||||
|
||||
### Props
|
||||
```csharp
|
||||
[Parameter] public string ScheduleType { get; set; } // "Mass", "Daily", "Hourly"
|
||||
[Parameter] public ScheduleConfig? Config { get; set; }
|
||||
[Parameter] public ScheduleConfig DefaultConfig { get; set; }
|
||||
[Parameter] public string? Query { get; set; }
|
||||
[Parameter] public EventCallback<string> OnViewQuery { get; set; }
|
||||
[ApiController]
|
||||
[Route(ApiRoutes.Pipelines.Base)]
|
||||
[Authorize]
|
||||
public class PipelineController : ControllerBase
|
||||
{
|
||||
private readonly IEtlPipelineFactory _pipelineFactory;
|
||||
private readonly IDataUpdateRepository _dataUpdateRepository;
|
||||
|
||||
[HttpGet]
|
||||
public ActionResult<PipelineListResponse> GetPipelineNames()
|
||||
{
|
||||
var names = _pipelineFactory.GetAvailableTables()
|
||||
.OrderBy(n => n)
|
||||
.ToList();
|
||||
return Ok(new PipelineListResponse(names));
|
||||
}
|
||||
|
||||
[HttpGet(ApiRoutes.Pipelines.ByName)]
|
||||
public ActionResult<PipelineConfigDto> GetPipeline(string name)
|
||||
{
|
||||
// Map from internal config to DTO
|
||||
// Truncate queries for preview, include full if admin
|
||||
}
|
||||
|
||||
[HttpGet(ApiRoutes.Pipelines.Status)]
|
||||
public async Task<ActionResult<PipelineStatusResponse>> GetStatus(string name)
|
||||
{
|
||||
// Get last runs, calculate IsOverdue using DataUpdateRepository.IsOverdue()
|
||||
}
|
||||
|
||||
[HttpGet(ApiRoutes.Pipelines.Executions)]
|
||||
public async Task<ActionResult<PipelineExecutionsResponse>> GetExecutions(
|
||||
string name, [FromQuery] int count = 10)
|
||||
{
|
||||
// Get recent updates from repository
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
---
|
||||
|
||||
| Method | Route | Description |
|
||||
|--------|-------|-------------|
|
||||
| GET | `/api/pipelines` | Get all pipeline names |
|
||||
| GET | `/api/pipelines/{name}` | Get pipeline configuration |
|
||||
| GET | `/api/pipelines/{name}/status` | Get schedule status summary |
|
||||
| GET | `/api/pipelines/{name}/executions?count=10` | Get recent executions |
|
||||
## Client Implementation
|
||||
|
||||
### PipelineApiClient.cs
|
||||
|
||||
```csharp
|
||||
public interface IPipelineApiClient
|
||||
{
|
||||
Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default);
|
||||
Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default);
|
||||
Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default);
|
||||
Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public class PipelineApiClient : ApiClientBase, IPipelineApiClient
|
||||
{
|
||||
public PipelineApiClient(HttpClient httpClient) : base(httpClient) { }
|
||||
|
||||
public Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default)
|
||||
=> GetAsync<PipelineListResponse>(ApiRoutes.Pipelines.Base, ct);
|
||||
|
||||
public Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default)
|
||||
=> GetAsync<PipelineConfigDto>(ApiRoutes.Pipelines.GetByName(name), ct);
|
||||
|
||||
public Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default)
|
||||
=> GetAsync<PipelineStatusResponse>(ApiRoutes.Pipelines.GetStatus(name), ct);
|
||||
|
||||
public Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default)
|
||||
=> GetAsync<PipelineExecutionsResponse>(ApiRoutes.Pipelines.GetExecutions(name, count), ct);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Blazor Components
|
||||
|
||||
### PipelineViewer.razor Structure
|
||||
|
||||
```razor
|
||||
@page "/admin/pipeline-viewer"
|
||||
@attribute [Authorize]
|
||||
@inject IPipelineApiClient PipelineApi
|
||||
|
||||
<PageTitle>Pipeline Configuration Viewer</PageTitle>
|
||||
|
||||
<RadzenText TextStyle="TextStyle.H4">ETL Pipeline Configuration Viewer</RadzenText>
|
||||
|
||||
<!-- Pipeline Selector -->
|
||||
<RadzenDropDown @bind-Value="_selectedPipeline"
|
||||
Data="_pipelineNames"
|
||||
Change="OnPipelineChanged" />
|
||||
|
||||
@if (_config is not null)
|
||||
{
|
||||
<!-- Status Summary Table -->
|
||||
<!-- Execution History Table -->
|
||||
<!-- Common Info Cards -->
|
||||
<!-- Schedule Sections (Mass, Daily, Hourly) -->
|
||||
}
|
||||
|
||||
<SqlQueryModal @bind-Visible="_showSqlModal" Title="@_sqlModalTitle" Sql="@_sqlModalContent" />
|
||||
```
|
||||
|
||||
### SqlQueryModal.razor
|
||||
|
||||
- Large modal (80% width)
|
||||
- `<pre><code>` with monospace font
|
||||
- Copy to clipboard button
|
||||
- Basic SQL formatting (line breaks at clauses)
|
||||
|
||||
### PipelineScheduleSection.razor
|
||||
|
||||
- Props: ScheduleType (UpdateTypes), Config (PipelineScheduleDto), Query, OnViewQuery callback
|
||||
- Shows enabled badge
|
||||
- Settings table with Default/Override indicators
|
||||
- Query preview with "View Full" button
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Override Detection
|
||||
To show "Default" vs "Override" for schedule settings:
|
||||
### Override Detection (in API)
|
||||
|
||||
```csharp
|
||||
bool IsOverride(int? pipelineValue, int defaultValue) =>
|
||||
pipelineValue.HasValue && pipelineValue.Value != defaultValue;
|
||||
bool IsOverride<T>(T? pipelineValue, T defaultValue) where T : struct =>
|
||||
pipelineValue.HasValue && !pipelineValue.Value.Equals(defaultValue);
|
||||
```
|
||||
|
||||
### Next Required Calculation
|
||||
```csharp
|
||||
DateTime? nextRequired = lastSuccessful?.AddMinutes(intervalMinutes);
|
||||
bool isOverdue = nextRequired.HasValue && nextRequired.Value < DateTime.UtcNow;
|
||||
```
|
||||
### Overdue Calculation (reuse existing)
|
||||
|
||||
### Query Truncation
|
||||
```csharp
|
||||
string Truncate(string sql, int maxLength = 100) =>
|
||||
sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
|
||||
// Use DataUpdateRepository.IsOverdue() which includes 50% grace period
|
||||
var isOverdue = DataUpdateRepository.IsOverdue(
|
||||
lastSuccessfulRun, tableName, updateType, customIntervals);
|
||||
```
|
||||
|
||||
### Connection Type Badge Colors
|
||||
- JDE: Blue (`BadgeStyle.Info`)
|
||||
- CMS: Green (`BadgeStyle.Success`)
|
||||
- GIW: Orange (`BadgeStyle.Warning`)
|
||||
|
||||
| Connection | Badge Style |
|
||||
|------------|-------------|
|
||||
| jde | `BadgeStyle.Info` (blue) |
|
||||
| cms | `BadgeStyle.Success` (green) |
|
||||
| giw | `BadgeStyle.Warning` (orange) |
|
||||
|
||||
### SQL Query Truncation
|
||||
|
||||
```csharp
|
||||
static string Truncate(string? sql, int maxLength = 100) =>
|
||||
sql is null ? "" :
|
||||
sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Authorization**: Page and API require `[Authorize]`
|
||||
2. **SQL Visibility**: Full SQL/scripts only exposed if user has appropriate permissions (consider admin-only)
|
||||
3. **No write operations**: This is read-only, no mutations exposed
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Radzen.Blazor (already installed)
|
||||
- Existing `IDataUpdateRepository`
|
||||
- Radzen.Blazor (existing)
|
||||
- Existing `IEtlPipelineFactory` and `IDataUpdateRepository`
|
||||
- Existing `pipelines.json` configuration
|
||||
- Existing `UpdateTypes` enum (Hourly, Daily, Mass)
|
||||
- Existing `UpdateTypes` enum
|
||||
- Existing `ApiClientBase` pattern
|
||||
|
||||
Reference in New Issue
Block a user