refactor(webui): remove pipeline viewer feature

Remove the read-only pipeline viewer from the web UI:
- Delete PipelineViewer.razor page and supporting components
- Delete PipelineController and PipelineMapper from API
- Delete Pipeline DTOs from Core
- Delete PipelineApiClient from Client
- Remove navigation link and DI registrations
- Delete obsolete plan documents

The ConfigManager utility retains pipeline editing capabilities.
This commit is contained in:
Joseph Doherty
2026-01-21 10:14:43 -05:00
parent 94d5a864e0
commit ceb63bfefb
20 changed files with 329 additions and 2776 deletions
@@ -1,488 +0,0 @@
# ETL Pipeline Viewer Component Design
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Build a Blazor component that visualizes ETL pipeline configuration and execution status.
**Architecture:** Server-side API loads pipeline config and exposes DTOs; Blazor client consumes via typed HttpClient following existing ApiClientBase pattern.
**Tech Stack:** Blazor WebAssembly, Radzen components, existing DataSync config types, existing DataUpdateRepository
---
## Codex Review Feedback (Addressed)
| Issue | Resolution |
|-------|------------|
| Client loading pipelines.json directly breaks CLEAN | API loads config, returns DTO to client |
| Duplicate config records diverge from DataSync | Reuse existing DataSync Configuration types |
| GetRecentUpdatesAsync missing UpdateType filter | Add UpdateType parameter to signature |
| Status/overdue logic diverges from DataSync | Reuse existing `DataUpdateRepository.IsOverdue()` |
| Only LastSuccessfulRun, hiding failures | Add LastRun + LastRunWasSuccessful |
| API patterns don't match ApiRoutes + ApiClientBase | Add `ApiRoutes.Pipelines`, implement `PipelineApiClient` |
| Duration non-nullable for in-progress | Make Duration nullable |
| ScheduleType as string | Use `UpdateTypes` enum |
| Missing authorization | Add `[Authorize]` to page and controller |
---
## Component Structure
### Page Layout
```
┌─────────────────────────────────────────────────────────────┐
│ ETL Pipeline Configuration Viewer [Authorize] │
├─────────────────────────────────────────────────────────────┤
│ Pipeline: [Dropdown - Alphabetical list] │
├─────────────────────────────────────────────────────────────┤
│ SCHEDULE STATUS SUMMARY │
│ ┌─────────┬────────────┬────────────┬─────────────┬───────┐ │
│ │ Type │ Last Run │ Success? │ Next Req. │Status │ │
│ ├─────────┼────────────┼────────────┼─────────────┼───────┤ │
│ │ Mass │ 01/05 02:00│ ✓ │ 01/12 02:00 │ OK │ │
│ │ Daily │ 01/07 01:00│ ✓ │ 01/08 01:00 │ OK │ │
│ │ Hourly │ 01/07 10:00│ ✗ │ 01/07 11:00 │ Over │ │
│ └─────────┴────────────┴────────────┴─────────────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ RECENT EXECUTION HISTORY (Last 10 per type) │
│ ┌─────────┬────────────┬──────────┬─────────┬─────┬───────┐ │
│ │ Type │ Start │ End │ Duration│ Rows│ Result│ │
│ └─────────┴────────────┴──────────┴─────────┴─────┴───────┘ │
├─────────────────────────────────────────────────────────────┤
│ COMMON PIPELINE INFO │
│ [Source Card] [Destination Card] [Scripts Card] │
├─────────────────────────────────────────────────────────────┤
│ MASS REFRESH [Enabled] │
│ [Schedule Settings] [Query Preview + Modal] │
├─────────────────────────────────────────────────────────────┤
│ DAILY REFRESH [Enabled] │
├─────────────────────────────────────────────────────────────┤
│ HOURLY REFRESH [Enabled] │
└─────────────────────────────────────────────────────────────┘
```
---
## Architecture
### CLEAN Architecture Layers
```
┌─────────────────────────────────────────────────────────────┐
│ PRESENTATION (JdeScoping.Client) │
│ - PipelineViewer.razor (page) │
│ - PipelineScheduleSection.razor (component) │
│ - SqlQueryModal.razor (component) │
│ - PipelineApiClient : ApiClientBase │
└─────────────────────────────────────────────────────────────┘
│ HTTP via ApiRoutes.Pipelines
┌─────────────────────────────────────────────────────────────┐
│ APPLICATION (JdeScoping.Api) │
│ - PipelineController │
│ - Returns DTOs from Core.ApiContracts │
└─────────────────────────────────────────────────────────────┘
│ Depends on
┌─────────────────────────────────────────────────────────────┐
│ INFRASTRUCTURE (JdeScoping.DataSync) │
│ - IEtlPipelineFactory (existing - loads pipelines.json) │
│ - IDataUpdateRepository (add GetRecentUpdatesAsync) │
│ - Reuse existing Configuration/* types │
└─────────────────────────────────────────────────────────────┘
│ Depends on
┌─────────────────────────────────────────────────────────────┐
│ DOMAIN (JdeScoping.Core) │
│ - ApiRoutes.Pipelines (route constants) │
│ - ApiContracts.Pipelines/* (DTOs) │
│ - Models.Enums.UpdateTypes (existing) │
└─────────────────────────────────────────────────────────────┘
```
---
## Files to Create/Modify
### Core Project (`src/JdeScoping.Core/`)
| File | Action | Description |
|------|--------|-------------|
| `ApiContracts/ApiRoutes.cs` | Modify | Add `Pipelines` routes |
| `ApiContracts/Pipelines/PipelineListResponse.cs` | Create | List of pipeline names |
| `ApiContracts/Pipelines/PipelineConfigDto.cs` | Create | Pipeline config DTO (sanitized) |
| `ApiContracts/Pipelines/PipelineStatusDto.cs` | Create | Schedule status DTO |
| `ApiContracts/Pipelines/PipelineExecutionDto.cs` | Create | Execution history DTO |
### DataSync Project (`src/JdeScoping.DataSync/`)
| File | Action | Description |
|------|--------|-------------|
| `Contracts/IDataUpdateRepository.cs` | Modify | Add `GetRecentUpdatesAsync` |
| `Services/DataUpdateRepository.cs` | Modify | Implement new method |
### API Project (`src/JdeScoping.Api/`)
| File | Action | Description |
|------|--------|-------------|
| `Controllers/PipelineController.cs` | Create | API endpoints |
### Client Project (`src/JdeScoping.Client/`)
| File | Action | Description |
|------|--------|-------------|
| `Services/IPipelineApiClient.cs` | Create | Client interface |
| `Services/PipelineApiClient.cs` | Create | Client implementation |
| `Pages/Admin/PipelineViewer.razor` | Create | Main page |
| `Components/Admin/PipelineScheduleSection.razor` | Create | Schedule section |
| `Components/Admin/SqlQueryModal.razor` | Create | SQL modal |
---
## API Routes (add to `ApiRoutes.cs`)
```csharp
/// <summary>
/// Routes for pipeline configuration API endpoints.
/// </summary>
public static class Pipelines
{
/// <summary>Base route for pipeline endpoints.</summary>
public const string Base = "api/pipelines";
/// <summary>Route template for getting a pipeline by name.</summary>
public const string ByName = "{name}";
/// <summary>Route template for getting pipeline status.</summary>
public const string Status = "{name}/status";
/// <summary>Route template for getting pipeline executions.</summary>
public const string Executions = "{name}/executions";
/// <summary>Builds the route to get a specific pipeline config.</summary>
public static string GetByName(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}";
/// <summary>Builds the route to get pipeline status.</summary>
public static string GetStatus(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}/status";
/// <summary>Builds the route to get pipeline executions.</summary>
public static string GetExecutions(string name, int count = 10) =>
$"api/pipelines/{Uri.EscapeDataString(name)}/executions?count={count}";
}
```
---
## DTOs (in `Core/ApiContracts/Pipelines/`)
### PipelineListResponse.cs
```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;
/// <summary>
/// Response containing list of available pipeline names.
/// </summary>
public record PipelineListResponse(List<string> PipelineNames);
```
### PipelineConfigDto.cs
```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;
/// <summary>
/// Pipeline configuration DTO for display (sanitized - no raw SQL exposed by default).
/// </summary>
public record PipelineConfigDto(
string Name,
PipelineSourceDto Source,
PipelineDestinationDto Destination,
PipelineSchedulesDto Schedules,
int PreScriptCount,
int PostScriptCount,
List<string>? PreScripts, // Only populated if user has admin role
List<string>? PostScripts); // Only populated if user has admin role
public record PipelineSourceDto(
string Connection,
string? QueryPreview, // First 100 chars
string? MassQueryPreview, // First 100 chars
string? Query, // Full query - only if admin
string? MassQuery, // Full query - only if admin
List<PipelineParameterDto> Parameters);
public record PipelineParameterDto(
string Name,
string? Format,
string Source);
public record PipelineDestinationDto(
string Table,
string OperationType, // "BulkMerge" or "BulkImport"
List<string>? MatchColumns,
List<string>? ExcludeFromUpdate);
public record PipelineSchedulesDto(
PipelineScheduleDto Mass,
PipelineScheduleDto Daily,
PipelineScheduleDto Hourly);
public record PipelineScheduleDto(
bool Enabled,
int IntervalMinutes,
bool PrePurge,
bool ReIndex,
bool IntervalIsOverride,
bool PrePurgeIsOverride,
bool ReIndexIsOverride);
```
### PipelineStatusDto.cs
```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;
using JdeScoping.Core.Models.Enums;
/// <summary>
/// Pipeline schedule status for each update type.
/// </summary>
public record PipelineStatusResponse(List<PipelineScheduleStatusDto> Statuses);
public record PipelineScheduleStatusDto(
UpdateTypes ScheduleType,
DateTime? LastRun,
bool LastRunWasSuccessful,
DateTime? LastSuccessfulRun,
DateTime? NextRequiredRun,
bool IsOverdue,
int IntervalMinutes);
```
### PipelineExecutionDto.cs
```csharp
namespace JdeScoping.Core.ApiContracts.Pipelines;
using JdeScoping.Core.Models.Enums;
/// <summary>
/// Pipeline execution history.
/// </summary>
public record PipelineExecutionsResponse(List<PipelineExecutionDto> Executions);
public record PipelineExecutionDto(
UpdateTypes ScheduleType,
DateTime StartTime,
DateTime? EndTime,
TimeSpan? Duration, // Nullable for in-progress runs
long RecordCount,
bool WasSuccessful);
```
---
## Repository Update (IDataUpdateRepository)
```csharp
/// <summary>
/// Gets the last N execution records for a specific table and optional update type.
/// </summary>
/// <param name="tableName">The table name to filter by.</param>
/// <param name="updateType">Optional update type filter. If null, returns all types.</param>
/// <param name="count">Maximum records to return per update type.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>List of DataUpdate records ordered by StartDt descending.</returns>
Task<List<DataUpdate>> GetRecentUpdatesAsync(
string tableName,
UpdateTypes? updateType = null,
int count = 10,
CancellationToken cancellationToken = default);
/// <summary>
/// Gets the last run (successful or not) for each update type for a table.
/// </summary>
/// <param name="tableName">The table name.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>Dictionary keyed by UpdateType.</returns>
Task<Dictionary<UpdateTypes, DataUpdate>> GetLastRunsAsync(
string tableName,
CancellationToken cancellationToken = default);
```
---
## API Controller
```csharp
[ApiController]
[Route(ApiRoutes.Pipelines.Base)]
[Authorize]
public class PipelineController : ControllerBase
{
private readonly IEtlPipelineFactory _pipelineFactory;
private readonly IDataUpdateRepository _dataUpdateRepository;
[HttpGet]
public ActionResult<PipelineListResponse> GetPipelineNames()
{
var names = _pipelineFactory.GetAvailableTables()
.OrderBy(n => n)
.ToList();
return Ok(new PipelineListResponse(names));
}
[HttpGet(ApiRoutes.Pipelines.ByName)]
public ActionResult<PipelineConfigDto> GetPipeline(string name)
{
// Map from internal config to DTO
// Truncate queries for preview, include full if admin
}
[HttpGet(ApiRoutes.Pipelines.Status)]
public async Task<ActionResult<PipelineStatusResponse>> GetStatus(string name)
{
// Get last runs, calculate IsOverdue using DataUpdateRepository.IsOverdue()
}
[HttpGet(ApiRoutes.Pipelines.Executions)]
public async Task<ActionResult<PipelineExecutionsResponse>> GetExecutions(
string name, [FromQuery] int count = 10)
{
// Get recent updates from repository
}
}
```
---
## Client Implementation
### PipelineApiClient.cs
```csharp
public interface IPipelineApiClient
{
Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default);
Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default);
Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default);
Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default);
}
public class PipelineApiClient : ApiClientBase, IPipelineApiClient
{
public PipelineApiClient(HttpClient httpClient) : base(httpClient) { }
public Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default)
=> GetAsync<PipelineListResponse>(ApiRoutes.Pipelines.Base, ct);
public Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default)
=> GetAsync<PipelineConfigDto>(ApiRoutes.Pipelines.GetByName(name), ct);
public Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default)
=> GetAsync<PipelineStatusResponse>(ApiRoutes.Pipelines.GetStatus(name), ct);
public Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 10, CancellationToken ct = default)
=> GetAsync<PipelineExecutionsResponse>(ApiRoutes.Pipelines.GetExecutions(name, count), ct);
}
```
---
## Blazor Components
### PipelineViewer.razor Structure
```razor
@page "/admin/pipeline-viewer"
@attribute [Authorize]
@inject IPipelineApiClient PipelineApi
<PageTitle>Pipeline Configuration Viewer</PageTitle>
<RadzenText TextStyle="TextStyle.H4">ETL Pipeline Configuration Viewer</RadzenText>
<!-- Pipeline Selector -->
<RadzenDropDown @bind-Value="_selectedPipeline"
Data="_pipelineNames"
Change="OnPipelineChanged" />
@if (_config is not null)
{
<!-- Status Summary Table -->
<!-- Execution History Table -->
<!-- Common Info Cards -->
<!-- Schedule Sections (Mass, Daily, Hourly) -->
}
<SqlQueryModal @bind-Visible="_showSqlModal" Title="@_sqlModalTitle" Sql="@_sqlModalContent" />
```
### SqlQueryModal.razor
- Large modal (80% width)
- `<pre><code>` with monospace font
- Copy to clipboard button
- Basic SQL formatting (line breaks at clauses)
### PipelineScheduleSection.razor
- Props: ScheduleType (UpdateTypes), Config (PipelineScheduleDto), Query, OnViewQuery callback
- Shows enabled badge
- Settings table with Default/Override indicators
- Query preview with "View Full" button
---
## Implementation Notes
### Override Detection (in API)
```csharp
bool IsOverride<T>(T? pipelineValue, T defaultValue) where T : struct =>
pipelineValue.HasValue && !pipelineValue.Value.Equals(defaultValue);
```
### Overdue Calculation (reuse existing)
```csharp
// Use DataUpdateRepository.IsOverdue() which includes 50% grace period
var isOverdue = DataUpdateRepository.IsOverdue(
lastSuccessfulRun, tableName, updateType, customIntervals);
```
### Connection Type Badge Colors
| Connection | Badge Style |
|------------|-------------|
| jde | `BadgeStyle.Info` (blue) |
| cms | `BadgeStyle.Success` (green) |
| giw | `BadgeStyle.Warning` (orange) |
### SQL Query Truncation
```csharp
static string Truncate(string? sql, int maxLength = 100) =>
sql is null ? "" :
sql.Length <= maxLength ? sql : sql[..maxLength] + "...";
```
---
## Security Considerations
1. **Authorization**: Page and API require `[Authorize]`
2. **SQL Visibility**: Full SQL/scripts only exposed if user has appropriate permissions (consider admin-only)
3. **No write operations**: This is read-only, no mutations exposed
---
## Dependencies
- Radzen.Blazor (existing)
- Existing `IEtlPipelineFactory` and `IDataUpdateRepository`
- Existing `pipelines.json` configuration
- Existing `UpdateTypes` enum
- Existing `ApiClientBase` pattern
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,329 @@
# Remove Pipeline Viewer Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Remove the pipeline viewer feature from the web UI and clean up all supporting code (controllers, DTOs, client services, routes).
**Architecture:** Delete files in dependency order (DTOs/interfaces first would break compilation), so we delete leaf components first (views, client), then API layer, then shared contracts/DTOs. Modify DI registrations and navigation.
**Tech Stack:** Blazor WebAssembly, ASP.NET Core API, C#
---
## Task 1: Remove Client Components (Blazor Pages)
**Files:**
- Delete: `src/JdeScoping.Client/Pages/Admin/PipelineViewer.razor`
- Delete: `src/JdeScoping.Client/Components/Admin/PipelineScheduleSection.razor`
- Delete: `src/JdeScoping.Client/Components/Admin/SqlQueryModal.razor`
**Step 1: Delete PipelineViewer.razor**
```bash
rm src/JdeScoping.Client/Pages/Admin/PipelineViewer.razor
```
**Step 2: Delete PipelineScheduleSection.razor**
```bash
rm src/JdeScoping.Client/Components/Admin/PipelineScheduleSection.razor
```
**Step 3: Delete SqlQueryModal.razor (if exists)**
```bash
rm -f src/JdeScoping.Client/Components/Admin/SqlQueryModal.razor
```
**Step 4: Verify files deleted**
```bash
ls src/JdeScoping.Client/Pages/Admin/ | grep -i pipeline || echo "No pipeline pages remain"
ls src/JdeScoping.Client/Components/Admin/ | grep -i pipeline || echo "No pipeline components remain"
```
Expected: No pipeline-related files in Admin folders.
---
## Task 2: Remove Navigation Link
**Files:**
- Modify: `src/JdeScoping.Client/Layout/MainLayout.razor:16`
**Step 1: Remove the pipeline viewer nav link**
In `MainLayout.razor`, delete this line (around line 16):
```razor
<NavLink class="nav-link" href="/admin/pipeline-viewer">Pipeline Viewer</NavLink>
```
**Step 2: Verify the change**
```bash
grep -n "pipeline-viewer" src/JdeScoping.Client/Layout/MainLayout.razor || echo "Nav link removed"
```
Expected: "Nav link removed"
---
## Task 3: Remove Client Service
**Files:**
- Delete: `src/JdeScoping.Client/Services/PipelineApiClient.cs`
- Modify: `src/JdeScoping.Client/Program.cs:58`
**Step 1: Delete PipelineApiClient.cs**
```bash
rm src/JdeScoping.Client/Services/PipelineApiClient.cs
```
**Step 2: Remove DI registration from Program.cs**
In `src/JdeScoping.Client/Program.cs`, delete this line (around line 58):
```csharp
builder.Services.AddScoped<IPipelineApiClient, PipelineApiClient>();
```
**Step 3: Verify the changes**
```bash
grep -n "PipelineApiClient" src/JdeScoping.Client/Program.cs || echo "DI registration removed"
ls src/JdeScoping.Client/Services/PipelineApiClient.cs 2>/dev/null || echo "File deleted"
```
Expected: Both checks show removal confirmed.
---
## Task 4: Remove API Controller and Mapper
**Files:**
- Delete: `src/JdeScoping.Api/Controllers/PipelineController.cs`
- Delete: `src/JdeScoping.Api/Mapping/PipelineMapper.cs`
- Delete: `src/JdeScoping.Api/Mapping/IPipelineMapper.cs`
- Modify: `src/JdeScoping.Api/DependencyInjection.cs:43`
**Step 1: Delete PipelineController.cs**
```bash
rm src/JdeScoping.Api/Controllers/PipelineController.cs
```
**Step 2: Delete PipelineMapper.cs**
```bash
rm src/JdeScoping.Api/Mapping/PipelineMapper.cs
```
**Step 3: Delete IPipelineMapper.cs**
```bash
rm src/JdeScoping.Api/Mapping/IPipelineMapper.cs
```
**Step 4: Remove DI registration from DependencyInjection.cs**
In `src/JdeScoping.Api/DependencyInjection.cs`, delete this line (around line 43):
```csharp
services.AddSingleton<IPipelineMapper, PipelineMapper>();
```
**Step 5: Verify the changes**
```bash
ls src/JdeScoping.Api/Controllers/PipelineController.cs 2>/dev/null || echo "Controller deleted"
ls src/JdeScoping.Api/Mapping/*Pipeline* 2>/dev/null || echo "Mapper files deleted"
grep -n "PipelineMapper" src/JdeScoping.Api/DependencyInjection.cs || echo "DI registration removed"
```
Expected: All three checks confirm removal.
---
## Task 5: Remove Core DTOs and Interface
**Files:**
- Delete: `src/JdeScoping.Core/Models/Pipelines/` (entire folder)
- Delete: `src/JdeScoping.Core/ApiContracts/IPipelineApiClient.cs`
**Step 1: Delete the Pipelines DTO folder**
```bash
rm -rf src/JdeScoping.Core/Models/Pipelines/
```
**Step 2: Delete IPipelineApiClient.cs**
```bash
rm src/JdeScoping.Core/ApiContracts/IPipelineApiClient.cs
```
**Step 3: Verify deletions**
```bash
ls src/JdeScoping.Core/Models/Pipelines/ 2>/dev/null || echo "Pipelines folder deleted"
ls src/JdeScoping.Core/ApiContracts/IPipelineApiClient.cs 2>/dev/null || echo "Interface deleted"
```
Expected: Both checks confirm deletion.
---
## Task 6: Remove API Routes
**Files:**
- Modify: `src/JdeScoping.Core/ApiContracts/ApiRoutes.cs:155-188`
**Step 1: Remove the Pipelines class from ApiRoutes.cs**
Delete the entire `Pipelines` static class (lines 155-188):
```csharp
/// <summary>
/// Routes for pipeline configuration API endpoints.
/// </summary>
public static class Pipelines
{
/// <summary>Base route for pipeline endpoints.</summary>
public const string Base = "api/pipelines";
/// <summary>Route template for getting a pipeline by name.</summary>
public const string ByName = "{name}";
/// <summary>Route template for getting pipeline status.</summary>
public const string Status = "{name}/status";
/// <summary>Route template for getting pipeline executions.</summary>
public const string Executions = "{name}/executions";
/// <summary>Builds the route to get a specific pipeline config.</summary>
/// <param name="name">The pipeline name to URL-encode.</param>
/// <returns>The formatted route.</returns>
public static string GetByName(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}";
/// <summary>Builds the route to get pipeline status.</summary>
/// <param name="name">The pipeline name to URL-encode.</param>
/// <returns>The formatted route.</returns>
public static string GetStatus(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}/status";
/// <summary>Builds the route to get pipeline executions.</summary>
/// <param name="name">The pipeline name to URL-encode.</param>
/// <param name="count">The number of recent executions to retrieve.</param>
/// <returns>The formatted route.</returns>
public static string GetExecutions(string name, int count = 10) =>
$"api/pipelines/{Uri.EscapeDataString(name)}/executions?count={count}";
}
```
**Step 2: Verify the change**
```bash
grep -n "Pipelines" src/JdeScoping.Core/ApiContracts/ApiRoutes.cs || echo "Pipelines routes removed"
```
Expected: "Pipelines routes removed"
---
## Task 7: Delete Plan Documents
**Files:**
- Delete: `docs/plans/2026-01-07-pipeline-viewer-design.md`
- Delete: `docs/plans/2026-01-07-pipeline-viewer-implementation.md`
**Step 1: Delete design document**
```bash
rm docs/plans/2026-01-07-pipeline-viewer-design.md
```
**Step 2: Delete implementation plan**
```bash
rm docs/plans/2026-01-07-pipeline-viewer-implementation.md
```
**Step 3: Verify deletions**
```bash
ls docs/plans/*pipeline-viewer* 2>/dev/null || echo "Plan documents deleted"
```
Expected: "Plan documents deleted"
---
## Task 8: Build and Verify
**Step 1: Build the solution**
```bash
cd /Users/dohertj2/Desktop/JdeScopingTool/NEW && dotnet build
```
Expected: Build succeeds with no errors.
**Step 2: Search for any remaining pipeline viewer references**
```bash
grep -r "PipelineViewer\|PipelineApiClient\|IPipelineApiClient\|PipelineController\|IPipelineMapper\|PipelineMapper\|pipeline-viewer" src/ --include="*.cs" --include="*.razor" || echo "No references remain"
```
Expected: "No references remain"
**Step 3: Run tests**
```bash
dotnet test
```
Expected: All tests pass.
---
## Task 9: Commit
**Step 1: Stage all changes**
```bash
git add -A
```
**Step 2: Commit**
```bash
git commit -m "$(cat <<'EOF'
refactor(webui): remove pipeline viewer feature
Remove the read-only pipeline viewer from the web UI:
- Delete PipelineViewer.razor page and supporting components
- Delete PipelineController and PipelineMapper from API
- Delete Pipeline DTOs from Core
- Delete PipelineApiClient from Client
- Remove navigation link and DI registrations
- Delete obsolete plan documents
The ConfigManager utility retains pipeline editing capabilities.
EOF
)"
```
---
## Summary
| Task | Action | Files Affected |
|------|--------|----------------|
| 1 | Delete Blazor components | 3 files deleted |
| 2 | Remove nav link | 1 file modified |
| 3 | Remove client service | 1 deleted, 1 modified |
| 4 | Remove API layer | 3 deleted, 1 modified |
| 5 | Remove Core DTOs | 5 files deleted |
| 6 | Remove API routes | 1 file modified |
| 7 | Delete plan docs | 2 files deleted |
| 8 | Build and verify | - |
| 9 | Commit | - |
**Total:** 14 files deleted, 4 files modified
@@ -1,144 +0,0 @@
using JdeScoping.Api.Mapping;
using JdeScoping.Core.ApiContracts;
using JdeScoping.Core.Models.Pipelines;
using JdeScoping.Core.Models.Enums;
using JdeScoping.DataSync.Contracts;
using JdeScoping.DataSync.Services;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
namespace JdeScoping.Api.Controllers;
/// <summary>
/// API endpoints for pipeline configuration and status.
/// </summary>
[Route(ApiRoutes.Pipelines.Base)]
[Authorize]
public class PipelineController : ApiControllerBase
{
private readonly IEtlPipelineFactory _pipelineFactory;
private readonly IDataUpdateRepository _dataUpdateRepository;
private readonly IPipelineMapper _mapper;
/// <summary>
/// Initializes a new instance of the <see cref="PipelineController"/> class.
/// </summary>
/// <param name="pipelineFactory">The ETL pipeline factory.</param>
/// <param name="dataUpdateRepository">The data update repository.</param>
/// <param name="mapper">The pipeline mapper.</param>
public PipelineController(
IEtlPipelineFactory pipelineFactory,
IDataUpdateRepository dataUpdateRepository,
IPipelineMapper mapper)
{
_pipelineFactory = pipelineFactory;
_dataUpdateRepository = dataUpdateRepository;
_mapper = mapper;
}
/// <summary>
/// Gets list of all available pipeline names.
/// </summary>
[HttpGet]
public ActionResult<PipelineListResponse> GetPipelineNames()
{
var names = _pipelineFactory.GetAvailableTables()
.OrderBy(n => n)
.ToList();
return Ok(new PipelineListResponse(names));
}
/// <summary>
/// Gets configuration for a specific pipeline.
/// </summary>
/// <param name="name">The pipeline name.</param>
[HttpGet(ApiRoutes.Pipelines.ByName)]
public ActionResult<PipelineConfigDto> GetPipeline(string name)
{
var config = _pipelineFactory.GetPipelineConfig(name);
if (config is null)
return NotFound();
var defaults = _pipelineFactory.GetScheduleDefaults();
var dto = _mapper.MapToDto(name, config, defaults);
return Ok(dto);
}
/// <summary>
/// Gets schedule status for a pipeline.
/// </summary>
/// <param name="name">The pipeline name.</param>
/// <param name="cancellationToken">The cancellation token.</param>
[HttpGet(ApiRoutes.Pipelines.Status)]
public async Task<ActionResult<PipelineStatusResponse>> GetStatus(
string name,
CancellationToken cancellationToken)
{
var config = _pipelineFactory.GetPipelineConfig(name);
if (config is null)
return NotFound();
var tableName = config.Destination.Table;
var lastRuns = await _dataUpdateRepository.GetLastRunsAsync(tableName, cancellationToken);
var lastSuccessful = await _dataUpdateRepository.GetLastDataUpdatesAsync(cancellationToken);
var defaults = _pipelineFactory.GetScheduleDefaults();
var statuses = new List<PipelineScheduleStatusDto>();
foreach (var updateType in new[] { UpdateTypes.Mass, UpdateTypes.Daily, UpdateTypes.Hourly })
{
var scheduleConfig = _mapper.GetScheduleConfig(config, updateType);
var interval = _mapper.GetEffectiveInterval(scheduleConfig, defaults, updateType);
lastRuns.TryGetValue(updateType, out var lastRun);
var successKey = $"{tableName}_{(int)updateType}";
lastSuccessful.TryGetValue(successKey, out var lastSuccess);
var nextRequired = lastSuccess?.EndDt?.AddMinutes(interval);
var isOverdue = DataUpdateRepository.IsOverdue(
lastSuccess?.EndDt, tableName, updateType, null);
statuses.Add(new PipelineScheduleStatusDto(
updateType,
lastRun?.StartDt,
lastRun?.WasSuccessful ?? false,
lastSuccess?.EndDt,
nextRequired,
isOverdue,
interval));
}
return Ok(new PipelineStatusResponse(statuses));
}
/// <summary>
/// Gets recent execution history for a pipeline.
/// </summary>
/// <param name="name">The pipeline name.</param>
/// <param name="count">The maximum number of recent executions to retrieve.</param>
/// <param name="cancellationToken">The cancellation token.</param>
[HttpGet(ApiRoutes.Pipelines.Executions)]
public async Task<ActionResult<PipelineExecutionsResponse>> GetExecutions(
string name,
[FromQuery] int count = 30,
CancellationToken cancellationToken = default)
{
var config = _pipelineFactory.GetPipelineConfig(name);
if (config is null)
return NotFound();
var tableName = config.Destination.Table;
var updates = await _dataUpdateRepository.GetRecentUpdatesAsync(
tableName, null, count, cancellationToken);
var executions = updates.Select(u => new PipelineExecutionDto(
u.UpdateType,
u.StartDt,
u.EndDt == default ? null : u.EndDt,
u.EndDt == default ? null : u.EndDt - u.StartDt,
u.NumberRecords,
u.WasSuccessful
)).ToList();
return Ok(new PipelineExecutionsResponse(executions));
}
}
@@ -1,6 +1,5 @@
using System.Text.Json.Serialization;
using JdeScoping.Api.Hubs;
using JdeScoping.Api.Mapping;
using JdeScoping.Api.Options;
using JdeScoping.Api.Services;
using JdeScoping.Core.Interfaces;
@@ -39,9 +38,6 @@ public static class ApiDependencyInjection
// Register TimeProvider for testability (allows mocking DateTime.UtcNow)
services.AddSingleton(TimeProvider.System);
// Register mappers
services.AddSingleton<IPipelineMapper, PipelineMapper>();
// Configure SignalR
services.AddSignalR();
@@ -1,34 +0,0 @@
using JdeScoping.Core.Models.Enums;
using JdeScoping.Core.Models.Pipelines;
using JdeScoping.DataSync.Configuration;
namespace JdeScoping.Api.Mapping;
/// <summary>
/// Mapper for pipeline configuration to DTOs.
/// </summary>
public interface IPipelineMapper
{
/// <summary>
/// Maps a pipeline configuration to its DTO representation.
/// </summary>
/// <param name="name">The pipeline name.</param>
/// <param name="config">The pipeline configuration.</param>
/// <param name="defaults">The default schedule settings.</param>
PipelineConfigDto MapToDto(string name, PipelineConfig config, ScheduleDefaults defaults);
/// <summary>
/// Gets the effective interval for a schedule, applying defaults if not specified.
/// </summary>
/// <param name="config">The schedule configuration, or null to use defaults.</param>
/// <param name="defaults">The default schedule settings.</param>
/// <param name="updateType">The type of update to get the interval for.</param>
int GetEffectiveInterval(ScheduleConfig? config, ScheduleDefaults defaults, UpdateTypes updateType);
/// <summary>
/// Gets the schedule configuration for a specific update type.
/// </summary>
/// <param name="config">The pipeline configuration.</param>
/// <param name="updateType">The type of update to get the configuration for.</param>
ScheduleConfig? GetScheduleConfig(PipelineConfig config, UpdateTypes updateType);
}
@@ -1,108 +0,0 @@
using JdeScoping.Core.Models.Enums;
using JdeScoping.Core.Models.Pipelines;
using JdeScoping.DataSync.Configuration;
namespace JdeScoping.Api.Mapping;
/// <summary>
/// Maps pipeline configuration to DTOs.
/// </summary>
public class PipelineMapper : IPipelineMapper
{
/// <inheritdoc />
public PipelineConfigDto MapToDto(
string name,
PipelineConfig config,
ScheduleDefaults defaults)
{
var source = new PipelineSourceDto(
config.Source.Connection,
Truncate(config.Source.Query),
Truncate(config.Source.MassQuery),
config.Source.Query,
config.Source.MassQuery,
config.Source.Parameters?.Select(p => new PipelineParameterDto(
p.Key, p.Value.Format, p.Value.Source)).ToList() ?? []);
var matchCols = config.Destination.MatchColumns?.ToList();
var destination = new PipelineDestinationDto(
config.Destination.Table,
matchCols?.Count > 0 ? "BulkMerge" : "BulkImport",
matchCols,
config.Destination.ExcludeFromUpdate?.ToList());
// Mass uses massQuery with no parameters; Daily/Hourly use query with parameters
var parameters = config.Source.Parameters?.Select(p => new PipelineParameterDto(
p.Key, p.Value.Format, p.Value.Source)).ToList() ?? [];
var schedules = new PipelineSchedulesDto(
MapSchedule(config.Schedules?.Mass, defaults.Mass, config.Source.MassQuery, [], config.PreScripts, config.PostScripts),
MapSchedule(config.Schedules?.Daily, defaults.Daily, config.Source.Query, parameters, config.PreScripts, config.PostScripts),
MapSchedule(config.Schedules?.Hourly, defaults.Hourly, config.Source.Query, parameters, config.PreScripts, config.PostScripts));
return new PipelineConfigDto(
name,
source,
destination,
schedules,
config.PreScripts?.Count ?? 0,
config.PostScripts?.Count ?? 0,
config.PreScripts,
config.PostScripts);
}
/// <inheritdoc />
public ScheduleConfig? GetScheduleConfig(
PipelineConfig config,
UpdateTypes updateType) => updateType switch
{
UpdateTypes.Mass => config.Schedules?.Mass,
UpdateTypes.Daily => config.Schedules?.Daily,
UpdateTypes.Hourly => config.Schedules?.Hourly,
_ => null
};
/// <inheritdoc />
public int GetEffectiveInterval(
ScheduleConfig? config,
ScheduleDefaults defaults,
UpdateTypes updateType)
{
if (config?.IntervalMinutes > 0)
return config.IntervalMinutes;
return updateType switch
{
UpdateTypes.Mass => defaults.Mass.IntervalMinutes,
UpdateTypes.Daily => defaults.Daily.IntervalMinutes,
UpdateTypes.Hourly => defaults.Hourly.IntervalMinutes,
_ => 60
};
}
private static PipelineScheduleDto MapSchedule(
ScheduleConfig? config,
ScheduleConfig defaults,
string? query,
List<PipelineParameterDto> parameters,
List<string>? preScripts,
List<string>? postScripts)
{
return new PipelineScheduleDto(
config?.Enabled ?? defaults.Enabled,
config?.IntervalMinutes > 0 ? config.IntervalMinutes : defaults.IntervalMinutes,
config?.PrePurge ?? defaults.PrePurge,
config?.ReIndex ?? defaults.ReIndex,
config?.IntervalMinutes > 0 && config.IntervalMinutes != defaults.IntervalMinutes,
config?.PrePurge != null && config.PrePurge != defaults.PrePurge,
config?.ReIndex != null && config.ReIndex != defaults.ReIndex,
query,
parameters,
preScripts,
postScripts);
}
private static string? Truncate(string? value, int maxLength = 100) =>
value is null ? null :
value.Length <= maxLength ? value : value[..maxLength] + "...";
}
@@ -1,116 +0,0 @@
@namespace JdeScoping.Client.Components.Admin
@using JdeScoping.Core.Models.Pipelines
@using JdeScoping.Core.Models.Enums
@using JdeScoping.Client.Helpers
<RadzenCard class="rz-mb-4">
<RadzenRow AlignItems="AlignItems.Center" class="rz-mb-3">
<RadzenColumn>
<RadzenText TextStyle="TextStyle.H5" class="rz-m-0">
@GetScheduleTypeName(ScheduleType) Refresh
</RadzenText>
</RadzenColumn>
<RadzenColumn Size="2" class="rz-text-align-end">
@if (Config?.Enabled == true)
{
<RadzenBadge BadgeStyle="BadgeStyle.Success" Text="Enabled" />
}
else
{
<RadzenBadge BadgeStyle="BadgeStyle.Light" Text="Disabled" />
}
</RadzenColumn>
</RadzenRow>
@if (Config is not null)
{
<RadzenText TextStyle="TextStyle.Subtitle1" class="rz-mb-2">Schedule Settings</RadzenText>
<table class="rz-mb-4" style="width: 100%; border-collapse: collapse;">
<thead>
<tr style="border-bottom: 1px solid #dee2e6;">
<th style="text-align: left; padding: 0.5rem;">Setting</th>
<th style="text-align: left; padding: 0.5rem;">Value</th>
<th style="text-align: left; padding: 0.5rem;">Source</th>
</tr>
</thead>
<tbody>
<tr style="border-bottom: 1px solid #dee2e6;">
<td style="padding: 0.5rem;">Interval</td>
<td style="padding: 0.5rem;">@FormatInterval(Config.IntervalMinutes)</td>
<td style="padding: 0.5rem;">@(Config.IntervalIsOverride ? "Override" : "Default")</td>
</tr>
<tr style="border-bottom: 1px solid #dee2e6;">
<td style="padding: 0.5rem;">Pre-Purge</td>
<td style="padding: 0.5rem;">@(Config.PrePurge ? "Yes" : "No")</td>
<td style="padding: 0.5rem;">@(Config.PrePurgeIsOverride ? "Override" : "Default")</td>
</tr>
<tr style="border-bottom: 1px solid #dee2e6;">
<td style="padding: 0.5rem;">Re-Index</td>
<td style="padding: 0.5rem;">@(Config.ReIndex ? "Yes" : "No")</td>
<td style="padding: 0.5rem;">@(Config.ReIndexIsOverride ? "Override" : "Default")</td>
</tr>
</tbody>
</table>
@if (Config.Parameters?.Count > 0)
{
<RadzenText TextStyle="TextStyle.Subtitle1" class="rz-mb-2">Parameters</RadzenText>
<ul class="rz-mb-4">
@foreach (var param in Config.Parameters)
{
<li><strong>@param.Name</strong>: @(param.Format ?? "default") (source: @param.Source)</li>
}
</ul>
}
@if (!string.IsNullOrWhiteSpace(Config.Query))
{
<RadzenText TextStyle="TextStyle.Subtitle1" class="rz-mb-2">Query</RadzenText>
<pre style="background: #f8f9fa; padding: 1rem; border-radius: 4px; font-family: 'Consolas', 'Monaco', 'Courier New', monospace; font-size: 0.875rem; line-height: 1.5; overflow-x: auto; white-space: pre-wrap; word-break: break-word; max-height: 400px; overflow-y: auto;">@SqlFormatHelper.FormatSql(Config.Query)</pre>
}
@if (Config.PreScripts?.Count > 0)
{
<RadzenText TextStyle="TextStyle.Subtitle1" class="rz-mt-4 rz-mb-2">Pre-Scripts (@Config.PreScripts.Count)</RadzenText>
@for (int i = 0; i < Config.PreScripts.Count; i++)
{
var script = Config.PreScripts[i];
<RadzenText TextStyle="TextStyle.Body2" class="rz-mb-1"><strong>Script @(i + 1):</strong></RadzenText>
<pre style="background: #fff3cd; padding: 0.75rem; border-radius: 4px; font-family: 'Consolas', 'Monaco', 'Courier New', monospace; font-size: 0.8rem; line-height: 1.4; overflow-x: auto; white-space: pre-wrap; word-break: break-word; max-height: 200px; overflow-y: auto; margin-bottom: 0.5rem;">@SqlFormatHelper.FormatSql(script)</pre>
}
}
@if (Config.PostScripts?.Count > 0)
{
<RadzenText TextStyle="TextStyle.Subtitle1" class="rz-mt-4 rz-mb-2">Post-Scripts (@Config.PostScripts.Count)</RadzenText>
@for (int i = 0; i < Config.PostScripts.Count; i++)
{
var script = Config.PostScripts[i];
<RadzenText TextStyle="TextStyle.Body2" class="rz-mb-1"><strong>Script @(i + 1):</strong></RadzenText>
<pre style="background: #d1ecf1; padding: 0.75rem; border-radius: 4px; font-family: 'Consolas', 'Monaco', 'Courier New', monospace; font-size: 0.8rem; line-height: 1.4; overflow-x: auto; white-space: pre-wrap; word-break: break-word; max-height: 200px; overflow-y: auto; margin-bottom: 0.5rem;">@SqlFormatHelper.FormatSql(script)</pre>
}
}
}
</RadzenCard>
@code {
[Parameter] public UpdateTypes ScheduleType { get; set; }
[Parameter] public PipelineScheduleDto? Config { get; set; }
private static string GetScheduleTypeName(UpdateTypes type) => type switch
{
UpdateTypes.Mass => "Mass",
UpdateTypes.Daily => "Daily",
UpdateTypes.Hourly => "Hourly",
_ => type.ToString()
};
private static string FormatInterval(int minutes)
{
if (minutes >= 1440)
return $"{minutes / 1440} day(s) ({minutes} min)";
if (minutes >= 60)
return $"{minutes / 60} hour(s) ({minutes} min)";
return $"{minutes} minutes";
}
}
@@ -1,101 +0,0 @@
@namespace JdeScoping.Client.Components.Admin
@using JdeScoping.Client.Helpers
@inject IJSRuntime JS
@if (Visible)
{
<div class="sql-modal-overlay" @onclick="Close">
<div class="sql-modal-content" @onclick:stopPropagation="true">
<div class="sql-modal-header">
<RadzenText TextStyle="TextStyle.H5" class="rz-m-0">@Title</RadzenText>
<RadzenButton Icon="close" ButtonStyle="ButtonStyle.Light" Size="ButtonSize.Small" Click="Close" />
</div>
<div class="sql-modal-body">
<pre>@FormattedSql</pre>
</div>
<div class="sql-modal-footer">
<RadzenButton Text="Copy to Clipboard" Icon="content_copy" ButtonStyle="ButtonStyle.Secondary" Click="CopyToClipboard" class="rz-mr-2" />
<RadzenButton Text="Close" Click="Close" />
</div>
</div>
</div>
}
<style>
.sql-modal-overlay {
position: fixed;
top: 0;
left: 0;
width: 100%;
height: 100%;
background-color: rgba(0, 0, 0, 0.5);
display: flex;
justify-content: center;
align-items: center;
z-index: 1000;
}
.sql-modal-content {
background: white;
border-radius: 8px;
width: 80vw;
max-width: 1200px;
max-height: 80vh;
display: flex;
flex-direction: column;
box-shadow: 0 4px 20px rgba(0, 0, 0, 0.2);
}
.sql-modal-header {
display: flex;
justify-content: space-between;
align-items: center;
padding: 1rem 1.5rem;
border-bottom: 1px solid #dee2e6;
}
.sql-modal-body {
flex: 1;
overflow: auto;
padding: 1.5rem;
background: #f8f9fa;
}
.sql-modal-body pre {
margin: 0;
white-space: pre-wrap;
word-break: break-word;
font-family: 'Consolas', 'Monaco', 'Courier New', monospace;
font-size: 0.875rem;
line-height: 1.5;
}
.sql-modal-footer {
display: flex;
justify-content: flex-end;
padding: 1rem 1.5rem;
border-top: 1px solid #dee2e6;
}
</style>
@code {
[Parameter] public bool Visible { get; set; }
[Parameter] public EventCallback<bool> VisibleChanged { get; set; }
[Parameter] public string? Title { get; set; }
[Parameter] public string? Sql { get; set; }
private string FormattedSql => SqlFormatHelper.FormatSql(Sql);
private async Task CopyToClipboard()
{
if (!string.IsNullOrWhiteSpace(Sql))
{
await JS.InvokeVoidAsync("navigator.clipboard.writeText", Sql);
}
}
private async Task Close()
{
await VisibleChanged.InvokeAsync(false);
}
}
@@ -13,7 +13,6 @@
<NavLink class="nav-link" href="/search">New Search</NavLink>
<NavLink class="nav-link" href="/search/queue">Search Queue</NavLink>
<NavLink class="nav-link" href="/refresh-status">Refresh Status</NavLink>
<NavLink class="nav-link" href="/admin/pipeline-viewer">Pipeline Viewer</NavLink>
</nav>
</div>
<div class="navbar-right">
@@ -1,243 +0,0 @@
@*
PipelineViewer.razor - ETL pipeline configuration viewer (admin).
Displays all configured data sync pipelines with their schedules, queries, and mappings.
Read-only view for inspecting pipeline configuration without modifying.
*@
@page "/admin/pipeline-viewer"
@attribute [Authorize]
@using JdeScoping.Core.ApiContracts
@using JdeScoping.Core.Models.Pipelines
@using JdeScoping.Core.Models.Enums
@inject IPipelineApiClient PipelineApi
<PageTitle>Pipeline Configuration Viewer - JDE Scoping Tool</PageTitle>
<RadzenText TextStyle="TextStyle.H4" class="rz-mb-4">ETL Pipeline Configuration Viewer</RadzenText>
<!-- Pipeline Selector -->
<RadzenCard class="rz-mb-4">
<RadzenFormField Text="Select Pipeline" Style="width: 400px;">
<RadzenDropDown @bind-Value="_selectedPipeline" Data="@_pipelineNames" Style="width: 100%;"
Change="@OnPipelineChanged" Placeholder="Select a pipeline..." />
</RadzenFormField>
</RadzenCard>
@if (_isLoading)
{
<LoadingIndicator Message="Loading pipeline data..." />
}
else if (_config is not null)
{
<!-- Status Summary Table -->
<RadzenCard class="rz-mb-4">
<RadzenText TextStyle="TextStyle.H5" class="rz-mb-3">Schedule Status Summary</RadzenText>
<RadzenDataGrid Data="@_statuses" TItem="PipelineScheduleStatusDto">
<Columns>
<RadzenDataGridColumn TItem="PipelineScheduleStatusDto" Property="ScheduleType" Title="Type" Width="100px">
<Template Context="item">@item.ScheduleType.ToString()</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineScheduleStatusDto" Title="Last Run" Width="160px">
<Template Context="item">@(item.LastRun?.ToString("MM/dd/yyyy hh:mm tt") ?? "Never")</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineScheduleStatusDto" Title="Success?" Width="80px" TextAlign="TextAlign.Center">
<Template Context="item">
@if (item.LastRun.HasValue)
{
@if (item.LastRunWasSuccessful)
{
<RadzenIcon Icon="check_circle" Style="color: green;" />
}
else
{
<RadzenIcon Icon="cancel" Style="color: red;" />
}
}
</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineScheduleStatusDto" Title="Next Required" Width="160px">
<Template Context="item">@(item.NextRequiredRun?.ToString("MM/dd/yyyy hh:mm tt") ?? "N/A")</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineScheduleStatusDto" Title="Status" Width="100px" TextAlign="TextAlign.Center">
<Template Context="item">
@if (item.IsOverdue)
{
<RadzenBadge BadgeStyle="BadgeStyle.Warning" Text="Overdue" />
}
else
{
<RadzenBadge BadgeStyle="BadgeStyle.Success" Text="OK" />
}
</Template>
</RadzenDataGridColumn>
</Columns>
</RadzenDataGrid>
</RadzenCard>
<!-- Execution History Table -->
<RadzenCard class="rz-mb-4">
<RadzenText TextStyle="TextStyle.H5" class="rz-mb-3">Recent Execution History</RadzenText>
<RadzenDataGrid Data="@_executions" TItem="PipelineExecutionDto" AllowSorting="true" PageSize="10" AllowPaging="true">
<Columns>
<RadzenDataGridColumn TItem="PipelineExecutionDto" Property="ScheduleType" Title="Type" Width="100px">
<Template Context="item">@item.ScheduleType.ToString()</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineExecutionDto" Title="Start" Width="160px">
<Template Context="item">@item.StartTime.ToString("MM/dd/yyyy hh:mm tt")</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineExecutionDto" Title="End" Width="160px">
<Template Context="item">@(item.EndTime?.ToString("MM/dd/yyyy hh:mm tt") ?? "In Progress")</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineExecutionDto" Title="Duration" Width="100px">
<Template Context="item">@FormatDuration(item.Duration)</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineExecutionDto" Property="RecordCount" Title="Records" Width="100px" TextAlign="TextAlign.Right">
<Template Context="item">@item.RecordCount.ToString("N0")</Template>
</RadzenDataGridColumn>
<RadzenDataGridColumn TItem="PipelineExecutionDto" Title="Result" Width="100px" TextAlign="TextAlign.Center">
<Template Context="item">
@if (item.WasSuccessful)
{
<RadzenBadge BadgeStyle="BadgeStyle.Success" Text="Success" />
}
else
{
<RadzenBadge BadgeStyle="BadgeStyle.Danger" Text="Failed" />
}
</Template>
</RadzenDataGridColumn>
</Columns>
</RadzenDataGrid>
</RadzenCard>
<!-- Common Pipeline Info -->
<RadzenRow Gap="1rem" class="rz-mb-4">
<!-- Source Card -->
<RadzenColumn Size="6">
<RadzenCard Style="height: 100%;">
<RadzenText TextStyle="TextStyle.H6" class="rz-mb-2">Source</RadzenText>
<p><strong>Connection:</strong>
@switch (_config.Source.Connection.ToLower())
{
case "jde":
<RadzenBadge BadgeStyle="BadgeStyle.Info" Text="JDE" />
break;
case "cms":
<RadzenBadge BadgeStyle="BadgeStyle.Success" Text="CMS" />
break;
case "giw":
<RadzenBadge BadgeStyle="BadgeStyle.Warning" Text="GIW" />
break;
default:
<RadzenBadge Text="@_config.Source.Connection" />
break;
}
</p>
</RadzenCard>
</RadzenColumn>
<!-- Destination Card -->
<RadzenColumn Size="6">
<RadzenCard Style="height: 100%;">
<RadzenText TextStyle="TextStyle.H6" class="rz-mb-2">Destination</RadzenText>
<p><strong>Table:</strong> @_config.Destination.Table</p>
<p><strong>Operation:</strong>
<RadzenBadge BadgeStyle="@(_config.Destination.OperationType == "BulkMerge" ? BadgeStyle.Primary : BadgeStyle.Secondary)"
Text="@_config.Destination.OperationType" />
</p>
@if (_config.Destination.MatchColumns?.Count > 0)
{
<p><strong>Match Columns:</strong></p>
<ul>
@foreach (var col in _config.Destination.MatchColumns)
{
<li>@col</li>
}
</ul>
}
@if (_config.Destination.ExcludeFromUpdate?.Count > 0)
{
<p><strong>Exclude:</strong></p>
<ul>
@foreach (var col in _config.Destination.ExcludeFromUpdate)
{
<li>@col</li>
}
</ul>
}
</RadzenCard>
</RadzenColumn>
</RadzenRow>
<!-- Schedule Sections -->
<PipelineScheduleSection ScheduleType="UpdateTypes.Mass" Config="@_config.Schedules.Mass" />
<PipelineScheduleSection ScheduleType="UpdateTypes.Daily" Config="@_config.Schedules.Daily" />
<PipelineScheduleSection ScheduleType="UpdateTypes.Hourly" Config="@_config.Schedules.Hourly" />
}
@code {
private List<string> _pipelineNames = [];
private string? _selectedPipeline;
private bool _isLoading;
private PipelineConfigDto? _config;
private List<PipelineScheduleStatusDto> _statuses = [];
private List<PipelineExecutionDto> _executions = [];
protected override async Task OnInitializedAsync()
{
var result = await PipelineApi.GetPipelineNamesAsync();
if (result.IsSuccess)
{
_pipelineNames = result.Value.PipelineNames;
}
}
private async Task OnPipelineChanged()
{
if (string.IsNullOrWhiteSpace(_selectedPipeline))
{
_config = null;
_statuses = [];
_executions = [];
return;
}
_isLoading = true;
try
{
var configTask = PipelineApi.GetPipelineAsync(_selectedPipeline);
var statusTask = PipelineApi.GetStatusAsync(_selectedPipeline);
var executionsTask = PipelineApi.GetExecutionsAsync(_selectedPipeline);
await Task.WhenAll(configTask, statusTask, executionsTask);
var configResult = configTask.Result;
var statusResult = statusTask.Result;
var executionsResult = executionsTask.Result;
if (configResult.IsSuccess)
_config = configResult.Value;
if (statusResult.IsSuccess)
_statuses = statusResult.Value.Statuses;
if (executionsResult.IsSuccess)
_executions = executionsResult.Value.Executions;
}
finally
{
_isLoading = false;
}
}
private static string FormatDuration(TimeSpan? duration)
{
if (!duration.HasValue) return "-";
var d = duration.Value;
if (d.TotalHours >= 1) return $"{d.Hours}h {d.Minutes}m";
if (d.TotalMinutes >= 1) return $"{d.Minutes}m {d.Seconds}s";
return $"{d.Seconds}s";
}
}
-1
View File
@@ -55,7 +55,6 @@ builder.Services.AddScoped<ISearchApiClient, SearchApiClient>();
builder.Services.AddScoped<ILookupApiClient, LookupApiClient>();
builder.Services.AddScoped<IAuthApiClient, AuthApiClient>();
builder.Services.AddScoped<IFileApiClient, FileApiClient>();
builder.Services.AddScoped<IPipelineApiClient, PipelineApiClient>();
// Search services
builder.Services.AddScoped<ISearchValidationService, SearchValidationService>();
@@ -1,53 +0,0 @@
using JdeScoping.Core.ApiContracts;
using JdeScoping.Core.Models.Pipelines;
using JdeScoping.Core.ApiContracts.Results;
namespace JdeScoping.Client.Services;
/// <summary>
/// HTTP client implementation for pipeline configuration API.
/// </summary>
public class PipelineApiClient : ApiClientBase, IPipelineApiClient
{
/// <summary>
/// Initializes a new instance of the <see cref="PipelineApiClient"/> class.
/// </summary>
/// <param name="httpClient">The HTTP client for API communication.</param>
public PipelineApiClient(HttpClient httpClient) : base(httpClient) { }
/// <summary>
/// Gets the list of available pipeline names from the API.
/// </summary>
/// <param name="ct">Cancellation token for the request.</param>
/// <returns>The API result containing the pipeline list.</returns>
public Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default)
=> GetAsync<PipelineListResponse>(ApiRoutes.Pipelines.Base, ct);
/// <summary>
/// Gets the configuration for a specific pipeline by name.
/// </summary>
/// <param name="name">The pipeline name.</param>
/// <param name="ct">Cancellation token for the request.</param>
/// <returns>The API result containing the pipeline configuration.</returns>
public Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default)
=> GetAsync<PipelineConfigDto>(ApiRoutes.Pipelines.GetByName(name), ct);
/// <summary>
/// Gets the current execution status of a pipeline.
/// </summary>
/// <param name="name">The pipeline name.</param>
/// <param name="ct">Cancellation token for the request.</param>
/// <returns>The API result containing the pipeline status.</returns>
public Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default)
=> GetAsync<PipelineStatusResponse>(ApiRoutes.Pipelines.GetStatus(name), ct);
/// <summary>
/// Gets the recent execution history for a pipeline.
/// </summary>
/// <param name="name">The pipeline name.</param>
/// <param name="count">The maximum number of execution records to retrieve.</param>
/// <param name="ct">Cancellation token for the request.</param>
/// <returns>The API result containing the execution history.</returns>
public Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 30, CancellationToken ct = default)
=> GetAsync<PipelineExecutionsResponse>(ApiRoutes.Pipelines.GetExecutions(name, count), ct);
}
-2
View File
@@ -13,7 +13,6 @@
@using Radzen.Blazor
@using JdeScoping.Client
@using JdeScoping.Client.Auth
@using JdeScoping.Client.Components.Admin
@using JdeScoping.Client.Components.FilterPanels
@using JdeScoping.Client.Extensions
@using JdeScoping.Client.Components.Search
@@ -25,6 +24,5 @@
@using JdeScoping.Core.ApiContracts
@using JdeScoping.Core.Models.Auth
@using JdeScoping.Core.Models.Infrastructure
@using JdeScoping.Core.Models.Pipelines
@using JdeScoping.Core.ViewModels
@using ClientSearchViewModel = JdeScoping.Client.Models.SearchViewModel
@@ -152,38 +152,4 @@ public static class ApiRoutes
$"api/refresh-status?minDT={minDt:yyyy-MM-dd}&maxDT={maxDt:yyyy-MM-dd}";
}
/// <summary>
/// Routes for pipeline configuration API endpoints.
/// </summary>
public static class Pipelines
{
/// <summary>Base route for pipeline endpoints.</summary>
public const string Base = "api/pipelines";
/// <summary>Route template for getting a pipeline by name.</summary>
public const string ByName = "{name}";
/// <summary>Route template for getting pipeline status.</summary>
public const string Status = "{name}/status";
/// <summary>Route template for getting pipeline executions.</summary>
public const string Executions = "{name}/executions";
/// <summary>Builds the route to get a specific pipeline config.</summary>
/// <param name="name">The pipeline name to URL-encode.</param>
/// <returns>The formatted route.</returns>
public static string GetByName(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}";
/// <summary>Builds the route to get pipeline status.</summary>
/// <param name="name">The pipeline name to URL-encode.</param>
/// <returns>The formatted route.</returns>
public static string GetStatus(string name) => $"api/pipelines/{Uri.EscapeDataString(name)}/status";
/// <summary>Builds the route to get pipeline executions.</summary>
/// <param name="name">The pipeline name to URL-encode.</param>
/// <param name="count">The number of recent executions to retrieve.</param>
/// <returns>The formatted route.</returns>
public static string GetExecutions(string name, int count = 10) =>
$"api/pipelines/{Uri.EscapeDataString(name)}/executions?count={count}";
}
}
@@ -1,30 +0,0 @@
using JdeScoping.Core.Models.Pipelines;
using JdeScoping.Core.ApiContracts.Results;
namespace JdeScoping.Core.ApiContracts;
/// <summary>
/// Client contract for pipeline configuration API operations.
/// </summary>
public interface IPipelineApiClient
{
/// <summary>Gets list of all available pipeline names.</summary>
/// <param name="ct">Cancellation token.</param>
Task<ApiResult<PipelineListResponse>> GetPipelineNamesAsync(CancellationToken ct = default);
/// <summary>Gets configuration for a specific pipeline.</summary>
/// <param name="name">The pipeline name.</param>
/// <param name="ct">Cancellation token.</param>
Task<ApiResult<PipelineConfigDto>> GetPipelineAsync(string name, CancellationToken ct = default);
/// <summary>Gets schedule status for a pipeline.</summary>
/// <param name="name">The pipeline name.</param>
/// <param name="ct">Cancellation token.</param>
Task<ApiResult<PipelineStatusResponse>> GetStatusAsync(string name, CancellationToken ct = default);
/// <summary>Gets recent execution history for a pipeline.</summary>
/// <param name="name">The pipeline name.</param>
/// <param name="count">The maximum number of executions to return.</param>
/// <param name="ct">Cancellation token.</param>
Task<ApiResult<PipelineExecutionsResponse>> GetExecutionsAsync(string name, int count = 30, CancellationToken ct = default);
}
@@ -1,51 +0,0 @@
namespace JdeScoping.Core.Models.Pipelines;
/// <summary>
/// Pipeline configuration DTO for display.
/// </summary>
public record PipelineConfigDto(
string Name,
PipelineSourceDto Source,
PipelineDestinationDto Destination,
PipelineSchedulesDto Schedules,
int PreScriptCount,
int PostScriptCount,
List<string>? PreScripts,
List<string>? PostScripts);
public record PipelineSourceDto(
string Connection,
string? QueryPreview,
string? MassQueryPreview,
string? Query,
string? MassQuery,
List<PipelineParameterDto> Parameters);
public record PipelineParameterDto(
string Name,
string? Format,
string Source);
public record PipelineDestinationDto(
string Table,
string OperationType,
List<string>? MatchColumns,
List<string>? ExcludeFromUpdate);
public record PipelineSchedulesDto(
PipelineScheduleDto Mass,
PipelineScheduleDto Daily,
PipelineScheduleDto Hourly);
public record PipelineScheduleDto(
bool Enabled,
int IntervalMinutes,
bool PrePurge,
bool ReIndex,
bool IntervalIsOverride,
bool PrePurgeIsOverride,
bool ReIndexIsOverride,
string? Query,
List<PipelineParameterDto> Parameters,
List<string>? PreScripts,
List<string>? PostScripts);
@@ -1,16 +0,0 @@
namespace JdeScoping.Core.Models.Pipelines;
using JdeScoping.Core.Models.Enums;
/// <summary>
/// Pipeline execution history.
/// </summary>
public record PipelineExecutionsResponse(List<PipelineExecutionDto> Executions);
public record PipelineExecutionDto(
UpdateTypes ScheduleType,
DateTime StartTime,
DateTime? EndTime,
TimeSpan? Duration,
long RecordCount,
bool WasSuccessful);
@@ -1,6 +0,0 @@
namespace JdeScoping.Core.Models.Pipelines;
/// <summary>
/// Response containing list of available pipeline names.
/// </summary>
public record PipelineListResponse(List<string> PipelineNames);
@@ -1,17 +0,0 @@
namespace JdeScoping.Core.Models.Pipelines;
using JdeScoping.Core.Models.Enums;
/// <summary>
/// Pipeline schedule status for each update type.
/// </summary>
public record PipelineStatusResponse(List<PipelineScheduleStatusDto> Statuses);
public record PipelineScheduleStatusDto(
UpdateTypes ScheduleType,
DateTime? LastRun,
bool LastRunWasSuccessful,
DateTime? LastSuccessfulRun,
DateTime? NextRequiredRun,
bool IsOverdue,
int IntervalMinutes);