# Phase 4: Host Health, Metrics & Status Server — Implementation Plan

**Date**: 2026-03-21
**Prerequisites**: Phase 3 complete and passing (gRPC server, Security, Configuration, Service Hosting all functional)
**Working Directory**: The lmxproxy repo is on windev at `C:\src\lmxproxy`

## Guardrails

1. **This is a v2 rebuild** — do not copy code from the v1 reference in `src-reference/`. Write fresh implementations guided by the design docs and the reference code's structure.
2. **Host targets .NET Framework 4.8, x86** — all code must use C# 9.0 language features maximum (`LangVersion` is `9.0` in the csproj). No file-scoped namespaces, no `required` keyword, no collection expressions in Host code.
3. **No new NuGet packages** — all required packages are already in the Host `.csproj` (`Microsoft.Extensions.Diagnostics.HealthChecks`, `Serilog`, `System.Threading.Channels`, `System.Text.Json` via framework).
4. **Namespace**: `ZB.MOM.WW.LmxProxy.Host` with sub-namespaces matching folder structure (e.g., `ZB.MOM.WW.LmxProxy.Host.Health`, `ZB.MOM.WW.LmxProxy.Host.Metrics`, `ZB.MOM.WW.LmxProxy.Host.Status`).
5. **All COM operations are on the STA thread** — health checks that read test tags must go through `MxAccessClient.ReadAsync()`, never directly touching COM objects.
6. **Build must pass after each step**: `dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86`
7. **Tests run on windev**: `dotnet test tests/ZB.MOM.WW.LmxProxy.Host.Tests --platform x86`

## Step 1: Create PerformanceMetrics

**File**: `src/ZB.MOM.WW.LmxProxy.Host/Metrics/PerformanceMetrics.cs`

Create the `PerformanceMetrics` class in namespace `ZB.MOM.WW.LmxProxy.Host.Metrics`.

### 1.1 OperationMetrics (nested or separate class in same file)

```csharp
namespace ZB.MOM.WW.LmxProxy.Host.Metrics
{
    public class OperationMetrics
    {
        private readonly List<double> _durations = new List<double>();
        private readonly object _lock = new object();
        private long _totalCount;
        private long _successCount;
        private double _totalMilliseconds;
        private double _minMilliseconds = double.MaxValue;
        private double _maxMilliseconds;

        public void Record(TimeSpan duration, bool success) { ... }
        public MetricsStatistics GetStatistics() { ... }
    }
}
```

Implementation details:
- `Record(TimeSpan duration, bool success)`: Inside `lock (_lock)`, increment `_totalCount`, conditionally increment `_successCount`, add `duration.TotalMilliseconds` to `_durations` list, update `_totalMilliseconds`, `_minMilliseconds`, `_maxMilliseconds`. If `_durations.Count > 1000`, call `_durations.RemoveAt(0)` to maintain rolling buffer.
- `GetStatistics()`: Inside `lock (_lock)`, return early with empty `MetricsStatistics` if `_totalCount == 0`. Otherwise sort `_durations`, compute p95 index as `(int)Math.Ceiling(sortedDurations.Count * 0.95) - 1`, clamp to `Math.Max(0, p95Index)`.

### 1.2 MetricsStatistics

```csharp
public class MetricsStatistics
{
    public long TotalCount { get; set; }
    public long SuccessCount { get; set; }
    public double SuccessRate { get; set; }
    public double AverageMilliseconds { get; set; }
    public double MinMilliseconds { get; set; }
    public double MaxMilliseconds { get; set; }
    public double Percentile95Milliseconds { get; set; }
}
```

### 1.3 ITimingScope interface and TimingScope implementation

```csharp
public interface ITimingScope : IDisposable
{
    void SetSuccess(bool success);
}
```

`TimingScope` is a private nested class inside `PerformanceMetrics`:
- Constructor takes `PerformanceMetrics metrics, string operationName`, starts a `Stopwatch`.
- `SetSuccess(bool success)` stores the flag (default `true`).
- `Dispose()`: stops stopwatch, calls `_metrics.RecordOperation(_operationName, _stopwatch.Elapsed, _success)`. Guard against double-dispose with `_disposed` flag.

### 1.4 PerformanceMetrics class

```csharp
public class PerformanceMetrics : IDisposable
{
    private static readonly ILogger Logger = Log.ForContext<PerformanceMetrics>();
    private readonly ConcurrentDictionary<string, OperationMetrics> _metrics = new ConcurrentDictionary<string, OperationMetrics>();
    private readonly Timer _reportingTimer;
    private bool _disposed;

    public PerformanceMetrics()
    {
        _reportingTimer = new Timer(ReportMetrics, null, TimeSpan.FromSeconds(60), TimeSpan.FromSeconds(60));
    }

    public void RecordOperation(string operationName, TimeSpan duration, bool success = true) { ... }
    public ITimingScope BeginOperation(string operationName) => new TimingScope(this, operationName);
    public OperationMetrics? GetMetrics(string operationName) { ... }
    public IReadOnlyDictionary<string, OperationMetrics> GetAllMetrics() { ... }
    public Dictionary<string, MetricsStatistics> GetStatistics() { ... }

    private void ReportMetrics(object? state) { ... } // Log each operation's stats at Information level
    public void Dispose() { ... } // Dispose timer, call ReportMetrics one final time
}
```

`ReportMetrics` iterates `_metrics`, calls `GetStatistics()` on each, logs via Serilog structured logging with properties: `Operation`, `Count`, `SuccessRate`, `AverageMs`, `MinMs`, `MaxMs`, `P95Ms`.

### 1.5 Verify build

```bash
ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"
```

## Step 2: Create HealthCheckService

**File**: `src/ZB.MOM.WW.LmxProxy.Host/Health/HealthCheckService.cs`

Namespace: `ZB.MOM.WW.LmxProxy.Host.Health`

### 2.1 Basic HealthCheckService

```csharp
public class HealthCheckService : IHealthCheck
{
    private static readonly ILogger Logger = Log.ForContext<HealthCheckService>();
    private readonly IScadaClient _scadaClient;
    private readonly SubscriptionManager _subscriptionManager;
    private readonly PerformanceMetrics _performanceMetrics;

    public HealthCheckService(
        IScadaClient scadaClient,
        SubscriptionManager subscriptionManager,
        PerformanceMetrics performanceMetrics) { ... }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default) { ... }
}
```

Dependencies imported:
- `ZB.MOM.WW.LmxProxy.Host.Domain` for `IScadaClient`, `ConnectionState`
- `ZB.MOM.WW.LmxProxy.Host.Services` for `SubscriptionManager` (if still in that namespace after Phase 2/3; adjust import to match actual location)
- `ZB.MOM.WW.LmxProxy.Host.Metrics` for `PerformanceMetrics`
- `Microsoft.Extensions.Diagnostics.HealthChecks` for `IHealthCheck`, `HealthCheckResult`, `HealthCheckContext`

`CheckHealthAsync` logic:
1. Create `Dictionary<string, object> data`.
2. Read `_scadaClient.IsConnected` and `_scadaClient.ConnectionState` into `data["scada_connected"]` and `data["scada_connection_state"]`.
3. Get subscription stats via `_subscriptionManager.GetSubscriptionStats()` — store `TotalClients`, `TotalTags` in data.
4. Iterate `_performanceMetrics.GetAllMetrics()` to compute `totalOperations` and `averageSuccessRate`.
5. Store `total_operations` and `average_success_rate` in data.
6. Decision tree:
   - If `!isConnected` → `HealthCheckResult.Unhealthy("SCADA client is not connected", data: data)`
   - If `averageSuccessRate < 0.5 && totalOperations > 100` → `HealthCheckResult.Degraded(...)`
   - If `subscriptionStats.TotalClients > 100` → `HealthCheckResult.Degraded(...)`
   - Otherwise → `HealthCheckResult.Healthy("LmxProxy is healthy", data)`
7. Wrap everything in try/catch — on exception return `Unhealthy` with exception details.

### 2.2 DetailedHealthCheckService

In the same file or a separate file `src/ZB.MOM.WW.LmxProxy.Host/Health/DetailedHealthCheckService.cs`:

```csharp
public class DetailedHealthCheckService : IHealthCheck
{
    private static readonly ILogger Logger = Log.ForContext<DetailedHealthCheckService>();
    private readonly IScadaClient _scadaClient;
    private readonly string _testTagAddress;

    public DetailedHealthCheckService(IScadaClient scadaClient, string testTagAddress = "TestChildObject.TestBool") { ... }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default) { ... }
}
```

`CheckHealthAsync` logic:
1. If `!_scadaClient.IsConnected` → return `Unhealthy`.
2. Try `Vtq vtq = await _scadaClient.ReadAsync(_testTagAddress, cancellationToken)`.
3. If `vtq.Quality != Quality.Good` → return `Degraded` with quality info.
4. If `DateTime.UtcNow - vtq.Timestamp > TimeSpan.FromMinutes(5)` → return `Degraded` (stale data).
5. Otherwise → `Healthy`.
6. Catch read exceptions → return `Degraded("Could not read test tag")`.
7. Catch all exceptions → return `Unhealthy`.

### 2.3 Verify build

```bash
ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"
```

## Step 3: Create StatusReportService

**File**: `src/ZB.MOM.WW.LmxProxy.Host/Status/StatusReportService.cs`

Namespace: `ZB.MOM.WW.LmxProxy.Host.Status`

### 3.1 Data model classes

Define in the same file (or a separate `StatusModels.cs` in the same folder):

```csharp
public class StatusData
{
    public DateTime Timestamp { get; set; }
    public string ServiceName { get; set; } = "";
    public string Version { get; set; } = "";
    public ConnectionStatus Connection { get; set; } = new ConnectionStatus();
    public SubscriptionStatus Subscriptions { get; set; } = new SubscriptionStatus();
    public PerformanceStatus Performance { get; set; } = new PerformanceStatus();
    public HealthInfo Health { get; set; } = new HealthInfo();
    public HealthInfo? DetailedHealth { get; set; }
}

public class ConnectionStatus
{
    public bool IsConnected { get; set; }
    public string State { get; set; } = "";
    public string NodeName { get; set; } = "";
    public string GalaxyName { get; set; } = "";
}

public class SubscriptionStatus
{
    public int TotalClients { get; set; }
    public int TotalTags { get; set; }
    public int ActiveSubscriptions { get; set; }
}

public class PerformanceStatus
{
    public long TotalOperations { get; set; }
    public double AverageSuccessRate { get; set; }
    public Dictionary<string, OperationStatus> Operations { get; set; } = new Dictionary<string, OperationStatus>();
}

public class OperationStatus
{
    public long TotalCount { get; set; }
    public double SuccessRate { get; set; }
    public double AverageMilliseconds { get; set; }
    public double MinMilliseconds { get; set; }
    public double MaxMilliseconds { get; set; }
    public double Percentile95Milliseconds { get; set; }
}

public class HealthInfo
{
    public string Status { get; set; } = "";
    public string Description { get; set; } = "";
    public Dictionary<string, string> Data { get; set; } = new Dictionary<string, string>();
}
```

### 3.2 StatusReportService

```csharp
public class StatusReportService
{
    private static readonly ILogger Logger = Log.ForContext<StatusReportService>();
    private readonly IScadaClient _scadaClient;
    private readonly SubscriptionManager _subscriptionManager;
    private readonly PerformanceMetrics _performanceMetrics;
    private readonly HealthCheckService _healthCheckService;
    private readonly DetailedHealthCheckService? _detailedHealthCheckService;

    public StatusReportService(
        IScadaClient scadaClient,
        SubscriptionManager subscriptionManager,
        PerformanceMetrics performanceMetrics,
        HealthCheckService healthCheckService,
        DetailedHealthCheckService? detailedHealthCheckService = null) { ... }

    public async Task<string> GenerateHtmlReportAsync() { ... }
    public async Task<string> GenerateJsonReportAsync() { ... }
    public async Task<bool> IsHealthyAsync() { ... }
    private async Task<StatusData> CollectStatusDataAsync() { ... }
    private static string GenerateHtmlFromStatusData(StatusData statusData) { ... }
    private static string GenerateErrorHtml(Exception ex) { ... }
}
```

`CollectStatusDataAsync`:
- Populate `StatusData.Timestamp = DateTime.UtcNow`, `ServiceName = "ZB.MOM.WW.LmxProxy.Host"`, `Version` from `Assembly.GetExecutingAssembly().GetName().Version`.
- Connection info from `_scadaClient.IsConnected`, `_scadaClient.ConnectionState`.
- Subscription stats from `_subscriptionManager.GetSubscriptionStats()`.
- Performance stats from `_performanceMetrics.GetStatistics()` — include P95 in the `OperationStatus`.
- Health from `_healthCheckService.CheckHealthAsync(new HealthCheckContext())`.
- Detailed health from `_detailedHealthCheckService?.CheckHealthAsync(new HealthCheckContext())` if not null.

`GenerateJsonReportAsync`:
- Use `System.Text.Json.JsonSerializer.Serialize(statusData, new JsonSerializerOptions { WriteIndented = true, PropertyNamingPolicy = JsonNamingPolicy.CamelCase })`.

`GenerateHtmlFromStatusData`:
- Use `StringBuilder` to generate self-contained HTML.
- Include inline CSS (Bootstrap-like grid, status cards with color-coded left borders).
- Color coding: green (#28a745) for Healthy/Connected, yellow (#ffc107) for Degraded, red (#dc3545) for Unhealthy/Disconnected.
- Operations table with columns: Operation, Count, Success Rate, Avg (ms), Min (ms), Max (ms), P95 (ms).
- `<meta http-equiv="refresh" content="30">` for auto-refresh.
- Last updated timestamp at the bottom.

`IsHealthyAsync`:
- Run basic health check, return `result.Status == HealthStatus.Healthy`.

### 3.3 Verify build

```bash
ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"
```

## Step 4: Create StatusWebServer

**File**: `src/ZB.MOM.WW.LmxProxy.Host/Status/StatusWebServer.cs`

Namespace: `ZB.MOM.WW.LmxProxy.Host.Status`

```csharp
public class StatusWebServer : IDisposable
{
    private static readonly ILogger Logger = Log.ForContext<StatusWebServer>();
    private readonly WebServerConfiguration _configuration;
    private readonly StatusReportService _statusReportService;
    private HttpListener? _httpListener;
    private CancellationTokenSource? _cancellationTokenSource;
    private Task? _listenerTask;
    private bool _disposed;

    public StatusWebServer(WebServerConfiguration configuration, StatusReportService statusReportService) { ... }

    public bool Start() { ... }
    public bool Stop() { ... }
    public void Dispose() { ... }

    private async Task HandleRequestsAsync(CancellationToken cancellationToken) { ... }
    private async Task HandleRequestAsync(HttpListenerContext context) { ... }
    private async Task HandleStatusPageAsync(HttpListenerResponse response) { ... }
    private async Task HandleStatusApiAsync(HttpListenerResponse response) { ... }
    private async Task HandleHealthApiAsync(HttpListenerResponse response) { ... }
    private static async Task WriteResponseAsync(HttpListenerResponse response, string content, string contentType) { ... }
}
```

### 4.1 Start()

1. If `!_configuration.Enabled`, log info and return `true`.
2. Create `HttpListener`, add prefix `_configuration.Prefix ?? $"http://+:{_configuration.Port}/"` (ensure trailing `/`).
3. Call `_httpListener.Start()`.
4. Create `_cancellationTokenSource = new CancellationTokenSource()`.
5. Start `_listenerTask = Task.Run(() => HandleRequestsAsync(_cancellationTokenSource.Token))`.
6. On exception, log error and return `false`.

### 4.2 Stop()

1. If not enabled or listener is null, return `true`.
2. Cancel `_cancellationTokenSource`.
3. Wait for `_listenerTask` with 5-second timeout.
4. Stop and close `_httpListener`.

### 4.3 HandleRequestsAsync

- Loop while not cancelled and listener is listening.
- `await _httpListener.GetContextAsync()` — on success, spawn `Task.Run` to handle.
- Catch `ObjectDisposedException` and `HttpListenerException(995)` as expected shutdown signals.
- On other errors, log and delay 1 second before continuing.

### 4.4 HandleRequestAsync routing

| Path (lowered) | Handler |
|---|---|
| `/` | `HandleStatusPageAsync` — calls `_statusReportService.GenerateHtmlReportAsync()`, content type `text/html; charset=utf-8` |
| `/api/status` | `HandleStatusApiAsync` — calls `_statusReportService.GenerateJsonReportAsync()`, content type `application/json; charset=utf-8` |
| `/api/health` | `HandleHealthApiAsync` — calls `_statusReportService.IsHealthyAsync()`, returns `"OK"` (200) or `"UNHEALTHY"` (503) as `text/plain` |
| Non-GET method | Return 405 Method Not Allowed |
| Unknown path | Return 404 Not Found |
| Exception | Return 500 Internal Server Error |

### 4.5 WriteResponseAsync

- Set `Content-Type`, add `Cache-Control: no-cache, no-store, must-revalidate`, `Pragma: no-cache`, `Expires: 0`.
- Convert content to UTF-8 bytes, set `ContentLength64`, write to `response.OutputStream`.

### 4.6 Dispose

- Guard with `_disposed` flag. Call `Stop()`. Dispose `_cancellationTokenSource` and close `_httpListener`.

### 4.7 Verify build

```bash
ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"
```

## Step 5: Wire into LmxProxyService

**File**: `src/ZB.MOM.WW.LmxProxy.Host/LmxProxyService.cs`

This file already exists. Modify the `Start()` method to create and wire the new components. The v2 rebuild should create these fresh, but the wiring pattern follows the same order as the reference.

### 5.1 Add using directives

```csharp
using ZB.MOM.WW.LmxProxy.Host.Health;
using ZB.MOM.WW.LmxProxy.Host.Metrics;
using ZB.MOM.WW.LmxProxy.Host.Status;
```

### 5.2 Add fields

```csharp
private PerformanceMetrics? _performanceMetrics;
private HealthCheckService? _healthCheckService;
private DetailedHealthCheckService? _detailedHealthCheckService;
private StatusReportService? _statusReportService;
private StatusWebServer? _statusWebServer;
```

### 5.3 In Start(), after SessionManager and SubscriptionManager creation

```csharp
// Create performance metrics
_performanceMetrics = new PerformanceMetrics();

// Create health check services
_healthCheckService = new HealthCheckService(_scadaClient, _subscriptionManager, _performanceMetrics);
_detailedHealthCheckService = new DetailedHealthCheckService(_scadaClient);

// Create status report service
_statusReportService = new StatusReportService(
    _scadaClient, _subscriptionManager, _performanceMetrics,
    _healthCheckService, _detailedHealthCheckService);

// Start status web server
_statusWebServer = new StatusWebServer(_configuration.WebServer, _statusReportService);
if (!_statusWebServer.Start())
{
    Logger.Warning("Status web server failed to start — continuing without it");
}
```

### 5.4 In Stop(), before gRPC server shutdown

```csharp
// Stop status web server
_statusWebServer?.Stop();

// Dispose performance metrics
_performanceMetrics?.Dispose();
```

### 5.5 Pass _performanceMetrics to ScadaGrpcService constructor

Ensure `ScadaGrpcService` receives `_performanceMetrics` so it can record timings on each RPC call. The gRPC service should call `_performanceMetrics.BeginOperation("Read")` (etc.) and dispose the timing scope at the end of each RPC handler.

### 5.6 Verify build

```bash
ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"
```

## Step 6: Unit Tests

**Project**: `tests/ZB.MOM.WW.LmxProxy.Host.Tests/`

If this project does not exist yet, create it:

```bash
ssh windev "cd C:\src\lmxproxy && dotnet new xunit -n ZB.MOM.WW.LmxProxy.Host.Tests -o tests/ZB.MOM.WW.LmxProxy.Host.Tests --framework net48"
```

**Csproj adjustments** for `tests/ZB.MOM.WW.LmxProxy.Host.Tests/ZB.MOM.WW.LmxProxy.Host.Tests.csproj`:
- `<TargetFramework>net48</TargetFramework>`
- `<PlatformTarget>x86</PlatformTarget>`
- `<LangVersion>9.0</LangVersion>`
- Add `<ProjectReference Include="..\..\src\ZB.MOM.WW.LmxProxy.Host\ZB.MOM.WW.LmxProxy.Host.csproj" />`
- Add `<PackageReference Include="xunit" Version="2.9.3" />`
- Add `<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />`
- Add `<PackageReference Include="NSubstitute" Version="5.3.0" />` (for mocking IScadaClient)
- Add `<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />`

**Also add to solution** in `ZB.MOM.WW.LmxProxy.slnx`:
```xml
<Folder Name="/tests/">
    <Project Path="tests/ZB.MOM.WW.LmxProxy.Host.Tests/ZB.MOM.WW.LmxProxy.Host.Tests.csproj" />
</Folder>
```

### 6.1 PerformanceMetrics Tests

**File**: `tests/ZB.MOM.WW.LmxProxy.Host.Tests/Metrics/PerformanceMetricsTests.cs`

```csharp
namespace ZB.MOM.WW.LmxProxy.Host.Tests.Metrics
{
    public class PerformanceMetricsTests
    {
        [Fact]
        public void RecordOperation_TracksCountAndDuration()
        // Record 5 operations, verify GetStatistics returns TotalCount=5

        [Fact]
        public void RecordOperation_TracksSuccessAndFailure()
        // Record 3 success + 2 failure, verify SuccessRate == 0.6

        [Fact]
        public void GetStatistics_CalculatesP95Correctly()
        // Record 100 operations with known durations (1ms through 100ms)
        // Verify P95 is approximately 95ms

        [Fact]
        public void RollingBuffer_CapsAt1000Samples()
        // Record 1500 operations, verify _durations list doesn't exceed 1000
        // (test via GetStatistics behavior — TotalCount is 1500 but percentile computed from 1000)

        [Fact]
        public void BeginOperation_RecordsDurationOnDispose()
        // Use BeginOperation, await Task.Delay(50), dispose scope
        // Verify recorded duration >= 50ms

        [Fact]
        public void TimingScope_DefaultsToSuccess()
        // BeginOperation + dispose without calling SetSuccess
        // Verify SuccessCount == 1

        [Fact]
        public void TimingScope_RespectsSetSuccessFalse()
        // BeginOperation, SetSuccess(false), dispose
        // Verify SuccessCount == 0, TotalCount == 1

        [Fact]
        public void GetMetrics_ReturnsNullForUnknownOperation()

        [Fact]
        public void GetAllMetrics_ReturnsAllTrackedOperations()
    }
}
```

### 6.2 HealthCheckService Tests

**File**: `tests/ZB.MOM.WW.LmxProxy.Host.Tests/Health/HealthCheckServiceTests.cs`

Use NSubstitute to mock `IScadaClient`. Create a real `PerformanceMetrics` instance and a real or mock `SubscriptionManager` (depends on Phase 2/3 implementation — if `SubscriptionManager` has an interface, mock it; if not, use the `GetSubscriptionStats()` approach with a concrete instance).

```csharp
namespace ZB.MOM.WW.LmxProxy.Host.Tests.Health
{
    public class HealthCheckServiceTests
    {
        [Fact]
        public async Task ReturnsHealthy_WhenConnectedAndNormalMetrics()
        // Mock: IsConnected=true, ConnectionState=Connected
        // SubscriptionStats: TotalClients=5, TotalTags=10
        // PerformanceMetrics: record some successes
        // Assert: HealthStatus.Healthy

        [Fact]
        public async Task ReturnsUnhealthy_WhenNotConnected()
        // Mock: IsConnected=false
        // Assert: HealthStatus.Unhealthy, description contains "not connected"

        [Fact]
        public async Task ReturnsDegraded_WhenSuccessRateBelow50Percent()
        // Mock: IsConnected=true
        // Record 200 operations with 40% success rate
        // Assert: HealthStatus.Degraded

        [Fact]
        public async Task ReturnsDegraded_WhenClientCountOver100()
        // Mock: IsConnected=true, SubscriptionStats.TotalClients=150
        // Assert: HealthStatus.Degraded

        [Fact]
        public async Task DoesNotFlagLowSuccessRate_Under100Operations()
        // Record 50 operations with 0% success rate
        // Assert: still Healthy (threshold is > 100 total ops)
    }

    public class DetailedHealthCheckServiceTests
    {
        [Fact]
        public async Task ReturnsUnhealthy_WhenNotConnected()

        [Fact]
        public async Task ReturnsHealthy_WhenTestTagGoodAndRecent()
        // Mock ReadAsync returns Good quality with recent timestamp
        // Assert: Healthy

        [Fact]
        public async Task ReturnsDegraded_WhenTestTagQualityNotGood()
        // Mock ReadAsync returns Uncertain quality
        // Assert: Degraded

        [Fact]
        public async Task ReturnsDegraded_WhenTestTagTimestampStale()
        // Mock ReadAsync returns Good quality but timestamp 10 minutes ago
        // Assert: Degraded

        [Fact]
        public async Task ReturnsDegraded_WhenTestTagReadThrows()
        // Mock ReadAsync throws exception
        // Assert: Degraded
    }
}
```

### 6.3 StatusReportService Tests

**File**: `tests/ZB.MOM.WW.LmxProxy.Host.Tests/Status/StatusReportServiceTests.cs`

```csharp
namespace ZB.MOM.WW.LmxProxy.Host.Tests.Status
{
    public class StatusReportServiceTests
    {
        [Fact]
        public async Task GenerateJsonReportAsync_ReturnsCamelCaseJson()
        // Verify JSON contains "serviceName", "connection", "isConnected" (camelCase)

        [Fact]
        public async Task GenerateHtmlReportAsync_ContainsAutoRefresh()
        // Verify HTML contains <meta http-equiv="refresh" content="30">

        [Fact]
        public async Task IsHealthyAsync_ReturnsTrueWhenHealthy()

        [Fact]
        public async Task IsHealthyAsync_ReturnsFalseWhenUnhealthy()

        [Fact]
        public async Task GenerateJsonReportAsync_IncludesPerformanceMetrics()
        // Record some operations, verify JSON includes operation names and stats
    }
}
```

### 6.4 Run tests

```bash
ssh windev "cd C:\src\lmxproxy && dotnet test tests/ZB.MOM.WW.LmxProxy.Host.Tests --platform x86 --verbosity normal"
```

## Step 7: Build Verification

Run full solution build and tests:

```bash
ssh windev "cd C:\src\lmxproxy && dotnet build ZB.MOM.WW.LmxProxy.slnx && dotnet test --verbosity normal"
```

If the test project is .NET 4.8 x86, you may need:
```bash
ssh windev "cd C:\src\lmxproxy && dotnet build ZB.MOM.WW.LmxProxy.slnx --platform x86 && dotnet test tests/ZB.MOM.WW.LmxProxy.Host.Tests --platform x86"
```

## Completion Criteria

- [ ] `PerformanceMetrics` class with `OperationMetrics`, `MetricsStatistics`, `ITimingScope` in `src/ZB.MOM.WW.LmxProxy.Host/Metrics/`
- [ ] `HealthCheckService` and `DetailedHealthCheckService` in `src/ZB.MOM.WW.LmxProxy.Host/Health/`
- [ ] `StatusReportService` with data model classes in `src/ZB.MOM.WW.LmxProxy.Host/Status/`
- [ ] `StatusWebServer` with HTML dashboard, JSON status, and health endpoints in `src/ZB.MOM.WW.LmxProxy.Host/Status/`
- [ ] All components wired into `LmxProxyService.Start()` / `Stop()`
- [ ] `ScadaGrpcService` uses `PerformanceMetrics.BeginOperation()` for Read, ReadBatch, Write, WriteBatch RPCs
- [ ] Unit tests for PerformanceMetrics (recording, percentile, rolling buffer, timing scope)
- [ ] Unit tests for HealthCheckService (healthy, unhealthy, degraded transitions)
- [ ] Unit tests for DetailedHealthCheckService (connected, quality, staleness)
- [ ] Unit tests for StatusReportService (JSON format, HTML format, health aggregation)
- [ ] Solution builds without errors: `dotnet build ZB.MOM.WW.LmxProxy.slnx`
- [ ] All tests pass: `dotnet test`