scadalink-design/lmxproxy/docs/plans/phase-4-host-health-metrics.md

Phase 4: Host Health, Metrics & Status Server — Implementation Plan

Date: 2026-03-21 Prerequisites: Phase 3 complete and passing (gRPC server, Security, Configuration, Service Hosting all functional) Working Directory: The lmxproxy repo is on windev at C:\src\lmxproxy

Guardrails

1. This is a v2 rebuild — do not copy code from the v1 reference in src-reference/. Write fresh implementations guided by the design docs and the reference code's structure.
2. Host targets .NET Framework 4.8, x86 — all code must use C# 9.0 language features maximum (LangVersion is 9.0 in the csproj). No file-scoped namespaces, no required keyword, no collection expressions in Host code.
3. No new NuGet packages — all required packages are already in the Host .csproj (Microsoft.Extensions.Diagnostics.HealthChecks, Serilog, System.Threading.Channels, System.Text.Json via framework).
4. Namespace: ZB.MOM.WW.LmxProxy.Host with sub-namespaces matching folder structure (e.g., ZB.MOM.WW.LmxProxy.Host.Health, ZB.MOM.WW.LmxProxy.Host.Metrics, ZB.MOM.WW.LmxProxy.Host.Status).
5. All COM operations are on the STA thread — health checks that read test tags must go through MxAccessClient.ReadAsync(), never directly touching COM objects.
6. Build must pass after each step: dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86
7. Tests run on windev: dotnet test tests/ZB.MOM.WW.LmxProxy.Host.Tests --platform x86

Step 1: Create PerformanceMetrics

File: src/ZB.MOM.WW.LmxProxy.Host/Metrics/PerformanceMetrics.cs

Create the PerformanceMetrics class in namespace ZB.MOM.WW.LmxProxy.Host.Metrics.

1.1 OperationMetrics (nested or separate class in same file)

namespace ZB.MOM.WW.LmxProxy.Host.Metrics
{
    public class OperationMetrics
    {
        private readonly List<double> _durations = new List<double>();
        private readonly object _lock = new object();
        private long _totalCount;
        private long _successCount;
        private double _totalMilliseconds;
        private double _minMilliseconds = double.MaxValue;
        private double _maxMilliseconds;

        public void Record(TimeSpan duration, bool success) { ... }
        public MetricsStatistics GetStatistics() { ... }
    }
}

Implementation details:

• Record(TimeSpan duration, bool success): Inside lock (_lock), increment _totalCount, conditionally increment _successCount, add duration.TotalMilliseconds to _durations list, update _totalMilliseconds, _minMilliseconds, _maxMilliseconds. If _durations.Count > 1000, call _durations.RemoveAt(0) to maintain rolling buffer.
• GetStatistics(): Inside lock (_lock), return early with empty MetricsStatistics if _totalCount == 0. Otherwise sort _durations, compute p95 index as (int)Math.Ceiling(sortedDurations.Count * 0.95) - 1, clamp to Math.Max(0, p95Index).

1.2 MetricsStatistics

public class MetricsStatistics
{
    public long TotalCount { get; set; }
    public long SuccessCount { get; set; }
    public double SuccessRate { get; set; }
    public double AverageMilliseconds { get; set; }
    public double MinMilliseconds { get; set; }
    public double MaxMilliseconds { get; set; }
    public double Percentile95Milliseconds { get; set; }
}

1.3 ITimingScope interface and TimingScope implementation

public interface ITimingScope : IDisposable
{
    void SetSuccess(bool success);
}

TimingScope is a private nested class inside PerformanceMetrics:

• Constructor takes PerformanceMetrics metrics, string operationName, starts a Stopwatch.
• SetSuccess(bool success) stores the flag (default true).
• Dispose(): stops stopwatch, calls _metrics.RecordOperation(_operationName, _stopwatch.Elapsed, _success). Guard against double-dispose with _disposed flag.

1.4 PerformanceMetrics class

public class PerformanceMetrics : IDisposable
{
    private static readonly ILogger Logger = Log.ForContext<PerformanceMetrics>();
    private readonly ConcurrentDictionary<string, OperationMetrics> _metrics = new ConcurrentDictionary<string, OperationMetrics>();
    private readonly Timer _reportingTimer;
    private bool _disposed;

    public PerformanceMetrics()
    {
        _reportingTimer = new Timer(ReportMetrics, null, TimeSpan.FromSeconds(60), TimeSpan.FromSeconds(60));
    }

    public void RecordOperation(string operationName, TimeSpan duration, bool success = true) { ... }
    public ITimingScope BeginOperation(string operationName) => new TimingScope(this, operationName);
    public OperationMetrics? GetMetrics(string operationName) { ... }
    public IReadOnlyDictionary<string, OperationMetrics> GetAllMetrics() { ... }
    public Dictionary<string, MetricsStatistics> GetStatistics() { ... }

    private void ReportMetrics(object? state) { ... } // Log each operation's stats at Information level
    public void Dispose() { ... } // Dispose timer, call ReportMetrics one final time
}

ReportMetrics iterates _metrics, calls GetStatistics() on each, logs via Serilog structured logging with properties: Operation, Count, SuccessRate, AverageMs, MinMs, MaxMs, P95Ms.

1.5 Verify build

ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"

Step 2: Create HealthCheckService

File: src/ZB.MOM.WW.LmxProxy.Host/Health/HealthCheckService.cs

Namespace: ZB.MOM.WW.LmxProxy.Host.Health

2.1 Basic HealthCheckService

public class HealthCheckService : IHealthCheck
{
    private static readonly ILogger Logger = Log.ForContext<HealthCheckService>();
    private readonly IScadaClient _scadaClient;
    private readonly SubscriptionManager _subscriptionManager;
    private readonly PerformanceMetrics _performanceMetrics;

    public HealthCheckService(
        IScadaClient scadaClient,
        SubscriptionManager subscriptionManager,
        PerformanceMetrics performanceMetrics) { ... }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default) { ... }
}

Dependencies imported:

• ZB.MOM.WW.LmxProxy.Host.Domain for IScadaClient, ConnectionState
• ZB.MOM.WW.LmxProxy.Host.Services for SubscriptionManager (if still in that namespace after Phase 2/3; adjust import to match actual location)
• ZB.MOM.WW.LmxProxy.Host.Metrics for PerformanceMetrics
• Microsoft.Extensions.Diagnostics.HealthChecks for IHealthCheck, HealthCheckResult, HealthCheckContext

CheckHealthAsync logic:

1. Create Dictionary<string, object> data.
2. Read _scadaClient.IsConnected and _scadaClient.ConnectionState into data["scada_connected"] and data["scada_connection_state"].
3. Get subscription stats via _subscriptionManager.GetSubscriptionStats() — store TotalClients, TotalTags in data.
4. Iterate _performanceMetrics.GetAllMetrics() to compute totalOperations and averageSuccessRate.
5. Store total_operations and average_success_rate in data.
6. Decision tree:
   • If !isConnected → HealthCheckResult.Unhealthy("SCADA client is not connected", data: data)
   • If averageSuccessRate < 0.5 && totalOperations > 100 → HealthCheckResult.Degraded(...)
   • If subscriptionStats.TotalClients > 100 → HealthCheckResult.Degraded(...)
   • Otherwise → HealthCheckResult.Healthy("LmxProxy is healthy", data)
7. Wrap everything in try/catch — on exception return Unhealthy with exception details.

2.2 DetailedHealthCheckService

In the same file or a separate file src/ZB.MOM.WW.LmxProxy.Host/Health/DetailedHealthCheckService.cs:

public class DetailedHealthCheckService : IHealthCheck
{
    private static readonly ILogger Logger = Log.ForContext<DetailedHealthCheckService>();
    private readonly IScadaClient _scadaClient;
    private readonly string _testTagAddress;

    public DetailedHealthCheckService(IScadaClient scadaClient, string testTagAddress = "TestChildObject.TestBool") { ... }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default) { ... }
}

CheckHealthAsync logic:

1. If !_scadaClient.IsConnected → return Unhealthy.
2. Try Vtq vtq = await _scadaClient.ReadAsync(_testTagAddress, cancellationToken).
3. If vtq.Quality != Quality.Good → return Degraded with quality info.
4. If DateTime.UtcNow - vtq.Timestamp > TimeSpan.FromMinutes(5) → return Degraded (stale data).
5. Otherwise → Healthy.
6. Catch read exceptions → return Degraded("Could not read test tag").
7. Catch all exceptions → return Unhealthy.

2.3 Verify build

ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"

Step 3: Create StatusReportService

File: src/ZB.MOM.WW.LmxProxy.Host/Status/StatusReportService.cs

Namespace: ZB.MOM.WW.LmxProxy.Host.Status

3.1 Data model classes

Define in the same file (or a separate StatusModels.cs in the same folder):

public class StatusData
{
    public DateTime Timestamp { get; set; }
    public string ServiceName { get; set; } = "";
    public string Version { get; set; } = "";
    public ConnectionStatus Connection { get; set; } = new ConnectionStatus();
    public SubscriptionStatus Subscriptions { get; set; } = new SubscriptionStatus();
    public PerformanceStatus Performance { get; set; } = new PerformanceStatus();
    public HealthInfo Health { get; set; } = new HealthInfo();
    public HealthInfo? DetailedHealth { get; set; }
}

public class ConnectionStatus
{
    public bool IsConnected { get; set; }
    public string State { get; set; } = "";
    public string NodeName { get; set; } = "";
    public string GalaxyName { get; set; } = "";
}

public class SubscriptionStatus
{
    public int TotalClients { get; set; }
    public int TotalTags { get; set; }
    public int ActiveSubscriptions { get; set; }
}

public class PerformanceStatus
{
    public long TotalOperations { get; set; }
    public double AverageSuccessRate { get; set; }
    public Dictionary<string, OperationStatus> Operations { get; set; } = new Dictionary<string, OperationStatus>();
}

public class OperationStatus
{
    public long TotalCount { get; set; }
    public double SuccessRate { get; set; }
    public double AverageMilliseconds { get; set; }
    public double MinMilliseconds { get; set; }
    public double MaxMilliseconds { get; set; }
    public double Percentile95Milliseconds { get; set; }
}

public class HealthInfo
{
    public string Status { get; set; } = "";
    public string Description { get; set; } = "";
    public Dictionary<string, string> Data { get; set; } = new Dictionary<string, string>();
}

3.2 StatusReportService

public class StatusReportService
{
    private static readonly ILogger Logger = Log.ForContext<StatusReportService>();
    private readonly IScadaClient _scadaClient;
    private readonly SubscriptionManager _subscriptionManager;
    private readonly PerformanceMetrics _performanceMetrics;
    private readonly HealthCheckService _healthCheckService;
    private readonly DetailedHealthCheckService? _detailedHealthCheckService;

    public StatusReportService(
        IScadaClient scadaClient,
        SubscriptionManager subscriptionManager,
        PerformanceMetrics performanceMetrics,
        HealthCheckService healthCheckService,
        DetailedHealthCheckService? detailedHealthCheckService = null) { ... }

    public async Task<string> GenerateHtmlReportAsync() { ... }
    public async Task<string> GenerateJsonReportAsync() { ... }
    public async Task<bool> IsHealthyAsync() { ... }
    private async Task<StatusData> CollectStatusDataAsync() { ... }
    private static string GenerateHtmlFromStatusData(StatusData statusData) { ... }
    private static string GenerateErrorHtml(Exception ex) { ... }
}

CollectStatusDataAsync:

• Populate StatusData.Timestamp = DateTime.UtcNow, ServiceName = "ZB.MOM.WW.LmxProxy.Host", Version from Assembly.GetExecutingAssembly().GetName().Version.
• Connection info from _scadaClient.IsConnected, _scadaClient.ConnectionState.
• Subscription stats from _subscriptionManager.GetSubscriptionStats().
• Performance stats from _performanceMetrics.GetStatistics() — include P95 in the OperationStatus.
• Health from _healthCheckService.CheckHealthAsync(new HealthCheckContext()).
• Detailed health from _detailedHealthCheckService?.CheckHealthAsync(new HealthCheckContext()) if not null.

GenerateJsonReportAsync:

• Use System.Text.Json.JsonSerializer.Serialize(statusData, new JsonSerializerOptions { WriteIndented = true, PropertyNamingPolicy = JsonNamingPolicy.CamelCase }).

GenerateHtmlFromStatusData:

• Use StringBuilder to generate self-contained HTML.
• Include inline CSS (Bootstrap-like grid, status cards with color-coded left borders).
• Color coding: green (#28a745) for Healthy/Connected, yellow (#ffc107) for Degraded, red (#dc3545) for Unhealthy/Disconnected.
• Operations table with columns: Operation, Count, Success Rate, Avg (ms), Min (ms), Max (ms), P95 (ms).
• <meta http-equiv="refresh" content="30"> for auto-refresh.
• Last updated timestamp at the bottom.

IsHealthyAsync:

• Run basic health check, return result.Status == HealthStatus.Healthy.

3.3 Verify build

ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"

Step 4: Create StatusWebServer

File: src/ZB.MOM.WW.LmxProxy.Host/Status/StatusWebServer.cs

Namespace: ZB.MOM.WW.LmxProxy.Host.Status

public class StatusWebServer : IDisposable
{
    private static readonly ILogger Logger = Log.ForContext<StatusWebServer>();
    private readonly WebServerConfiguration _configuration;
    private readonly StatusReportService _statusReportService;
    private HttpListener? _httpListener;
    private CancellationTokenSource? _cancellationTokenSource;
    private Task? _listenerTask;
    private bool _disposed;

    public StatusWebServer(WebServerConfiguration configuration, StatusReportService statusReportService) { ... }

    public bool Start() { ... }
    public bool Stop() { ... }
    public void Dispose() { ... }

    private async Task HandleRequestsAsync(CancellationToken cancellationToken) { ... }
    private async Task HandleRequestAsync(HttpListenerContext context) { ... }
    private async Task HandleStatusPageAsync(HttpListenerResponse response) { ... }
    private async Task HandleStatusApiAsync(HttpListenerResponse response) { ... }
    private async Task HandleHealthApiAsync(HttpListenerResponse response) { ... }
    private static async Task WriteResponseAsync(HttpListenerResponse response, string content, string contentType) { ... }
}

4.1 Start()

1. If !_configuration.Enabled, log info and return true.
2. Create HttpListener, add prefix _configuration.Prefix ?? $"http://+:{_configuration.Port}/" (ensure trailing /).
3. Call _httpListener.Start().
4. Create _cancellationTokenSource = new CancellationTokenSource().
5. Start _listenerTask = Task.Run(() => HandleRequestsAsync(_cancellationTokenSource.Token)).
6. On exception, log error and return false.

4.2 Stop()

1. If not enabled or listener is null, return true.
2. Cancel _cancellationTokenSource.
3. Wait for _listenerTask with 5-second timeout.
4. Stop and close _httpListener.

4.3 HandleRequestsAsync

• Loop while not cancelled and listener is listening.
• await _httpListener.GetContextAsync() — on success, spawn Task.Run to handle.
• Catch ObjectDisposedException and HttpListenerException(995) as expected shutdown signals.
• On other errors, log and delay 1 second before continuing.

4.4 HandleRequestAsync routing

Path (lowered)	Handler
/	HandleStatusPageAsync — calls _statusReportService.GenerateHtmlReportAsync(), content type text/html; charset=utf-8
/api/status	HandleStatusApiAsync — calls _statusReportService.GenerateJsonReportAsync(), content type application/json; charset=utf-8
/api/health	HandleHealthApiAsync — calls _statusReportService.IsHealthyAsync(), returns "OK" (200) or "UNHEALTHY" (503) as text/plain
Non-GET method	Return 405 Method Not Allowed
Unknown path	Return 404 Not Found
Exception	Return 500 Internal Server Error

4.5 WriteResponseAsync

• Set Content-Type, add Cache-Control: no-cache, no-store, must-revalidate, Pragma: no-cache, Expires: 0.
• Convert content to UTF-8 bytes, set ContentLength64, write to response.OutputStream.

4.6 Dispose

• Guard with _disposed flag. Call Stop(). Dispose _cancellationTokenSource and close _httpListener.

4.7 Verify build

ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"

Step 5: Wire into LmxProxyService

File: src/ZB.MOM.WW.LmxProxy.Host/LmxProxyService.cs

This file already exists. Modify the Start() method to create and wire the new components. The v2 rebuild should create these fresh, but the wiring pattern follows the same order as the reference.

5.1 Add using directives

using ZB.MOM.WW.LmxProxy.Host.Health;
using ZB.MOM.WW.LmxProxy.Host.Metrics;
using ZB.MOM.WW.LmxProxy.Host.Status;

5.2 Add fields

private PerformanceMetrics? _performanceMetrics;
private HealthCheckService? _healthCheckService;
private DetailedHealthCheckService? _detailedHealthCheckService;
private StatusReportService? _statusReportService;
private StatusWebServer? _statusWebServer;

5.3 In Start(), after SessionManager and SubscriptionManager creation

// Create performance metrics
_performanceMetrics = new PerformanceMetrics();

// Create health check services
_healthCheckService = new HealthCheckService(_scadaClient, _subscriptionManager, _performanceMetrics);
_detailedHealthCheckService = new DetailedHealthCheckService(_scadaClient);

// Create status report service
_statusReportService = new StatusReportService(
    _scadaClient, _subscriptionManager, _performanceMetrics,
    _healthCheckService, _detailedHealthCheckService);

// Start status web server
_statusWebServer = new StatusWebServer(_configuration.WebServer, _statusReportService);
if (!_statusWebServer.Start())
{
    Logger.Warning("Status web server failed to start — continuing without it");
}

5.4 In Stop(), before gRPC server shutdown

// Stop status web server
_statusWebServer?.Stop();

// Dispose performance metrics
_performanceMetrics?.Dispose();

5.5 Pass _performanceMetrics to ScadaGrpcService constructor

Ensure ScadaGrpcService receives _performanceMetrics so it can record timings on each RPC call. The gRPC service should call _performanceMetrics.BeginOperation("Read") (etc.) and dispose the timing scope at the end of each RPC handler.

5.6 Verify build

ssh windev "cd C:\src\lmxproxy && dotnet build src/ZB.MOM.WW.LmxProxy.Host --platform x86"

Step 6: Unit Tests

Project: tests/ZB.MOM.WW.LmxProxy.Host.Tests/

If this project does not exist yet, create it:

ssh windev "cd C:\src\lmxproxy && dotnet new xunit -n ZB.MOM.WW.LmxProxy.Host.Tests -o tests/ZB.MOM.WW.LmxProxy.Host.Tests --framework net48"

Csproj adjustments for tests/ZB.MOM.WW.LmxProxy.Host.Tests/ZB.MOM.WW.LmxProxy.Host.Tests.csproj:

• <TargetFramework>net48</TargetFramework>
• <PlatformTarget>x86</PlatformTarget>
• <LangVersion>9.0</LangVersion>
• Add <ProjectReference Include="..\..\src\ZB.MOM.WW.LmxProxy.Host\ZB.MOM.WW.LmxProxy.Host.csproj" />
• Add <PackageReference Include="xunit" Version="2.9.3" />
• Add <PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />
• Add <PackageReference Include="NSubstitute" Version="5.3.0" /> (for mocking IScadaClient)
• Add <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />

Also add to solution in ZB.MOM.WW.LmxProxy.slnx:

<Folder Name="/tests/">
    <Project Path="tests/ZB.MOM.WW.LmxProxy.Host.Tests/ZB.MOM.WW.LmxProxy.Host.Tests.csproj" />
</Folder>

6.1 PerformanceMetrics Tests

File: tests/ZB.MOM.WW.LmxProxy.Host.Tests/Metrics/PerformanceMetricsTests.cs

namespace ZB.MOM.WW.LmxProxy.Host.Tests.Metrics
{
    public class PerformanceMetricsTests
    {
        [Fact]
        public void RecordOperation_TracksCountAndDuration()
        // Record 5 operations, verify GetStatistics returns TotalCount=5

        [Fact]
        public void RecordOperation_TracksSuccessAndFailure()
        // Record 3 success + 2 failure, verify SuccessRate == 0.6

        [Fact]
        public void GetStatistics_CalculatesP95Correctly()
        // Record 100 operations with known durations (1ms through 100ms)
        // Verify P95 is approximately 95ms

        [Fact]
        public void RollingBuffer_CapsAt1000Samples()
        // Record 1500 operations, verify _durations list doesn't exceed 1000
        // (test via GetStatistics behavior — TotalCount is 1500 but percentile computed from 1000)

        [Fact]
        public void BeginOperation_RecordsDurationOnDispose()
        // Use BeginOperation, await Task.Delay(50), dispose scope
        // Verify recorded duration >= 50ms

        [Fact]
        public void TimingScope_DefaultsToSuccess()
        // BeginOperation + dispose without calling SetSuccess
        // Verify SuccessCount == 1

        [Fact]
        public void TimingScope_RespectsSetSuccessFalse()
        // BeginOperation, SetSuccess(false), dispose
        // Verify SuccessCount == 0, TotalCount == 1

        [Fact]
        public void GetMetrics_ReturnsNullForUnknownOperation()

        [Fact]
        public void GetAllMetrics_ReturnsAllTrackedOperations()
    }
}

6.2 HealthCheckService Tests

File: tests/ZB.MOM.WW.LmxProxy.Host.Tests/Health/HealthCheckServiceTests.cs

Use NSubstitute to mock IScadaClient. Create a real PerformanceMetrics instance and a real or mock SubscriptionManager (depends on Phase 2/3 implementation — if SubscriptionManager has an interface, mock it; if not, use the GetSubscriptionStats() approach with a concrete instance).

namespace ZB.MOM.WW.LmxProxy.Host.Tests.Health
{
    public class HealthCheckServiceTests
    {
        [Fact]
        public async Task ReturnsHealthy_WhenConnectedAndNormalMetrics()
        // Mock: IsConnected=true, ConnectionState=Connected
        // SubscriptionStats: TotalClients=5, TotalTags=10
        // PerformanceMetrics: record some successes
        // Assert: HealthStatus.Healthy

        [Fact]
        public async Task ReturnsUnhealthy_WhenNotConnected()
        // Mock: IsConnected=false
        // Assert: HealthStatus.Unhealthy, description contains "not connected"

        [Fact]
        public async Task ReturnsDegraded_WhenSuccessRateBelow50Percent()
        // Mock: IsConnected=true
        // Record 200 operations with 40% success rate
        // Assert: HealthStatus.Degraded

        [Fact]
        public async Task ReturnsDegraded_WhenClientCountOver100()
        // Mock: IsConnected=true, SubscriptionStats.TotalClients=150
        // Assert: HealthStatus.Degraded

        [Fact]
        public async Task DoesNotFlagLowSuccessRate_Under100Operations()
        // Record 50 operations with 0% success rate
        // Assert: still Healthy (threshold is > 100 total ops)
    }

    public class DetailedHealthCheckServiceTests
    {
        [Fact]
        public async Task ReturnsUnhealthy_WhenNotConnected()

        [Fact]
        public async Task ReturnsHealthy_WhenTestTagGoodAndRecent()
        // Mock ReadAsync returns Good quality with recent timestamp
        // Assert: Healthy

        [Fact]
        public async Task ReturnsDegraded_WhenTestTagQualityNotGood()
        // Mock ReadAsync returns Uncertain quality
        // Assert: Degraded

        [Fact]
        public async Task ReturnsDegraded_WhenTestTagTimestampStale()
        // Mock ReadAsync returns Good quality but timestamp 10 minutes ago
        // Assert: Degraded

        [Fact]
        public async Task ReturnsDegraded_WhenTestTagReadThrows()
        // Mock ReadAsync throws exception
        // Assert: Degraded
    }
}

6.3 StatusReportService Tests

File: tests/ZB.MOM.WW.LmxProxy.Host.Tests/Status/StatusReportServiceTests.cs

namespace ZB.MOM.WW.LmxProxy.Host.Tests.Status
{
    public class StatusReportServiceTests
    {
        [Fact]
        public async Task GenerateJsonReportAsync_ReturnsCamelCaseJson()
        // Verify JSON contains "serviceName", "connection", "isConnected" (camelCase)

        [Fact]
        public async Task GenerateHtmlReportAsync_ContainsAutoRefresh()
        // Verify HTML contains <meta http-equiv="refresh" content="30">

        [Fact]
        public async Task IsHealthyAsync_ReturnsTrueWhenHealthy()

        [Fact]
        public async Task IsHealthyAsync_ReturnsFalseWhenUnhealthy()

        [Fact]
        public async Task GenerateJsonReportAsync_IncludesPerformanceMetrics()
        // Record some operations, verify JSON includes operation names and stats
    }
}

6.4 Run tests

ssh windev "cd C:\src\lmxproxy && dotnet test tests/ZB.MOM.WW.LmxProxy.Host.Tests --platform x86 --verbosity normal"

Step 7: Build Verification

Run full solution build and tests:

ssh windev "cd C:\src\lmxproxy && dotnet build ZB.MOM.WW.LmxProxy.slnx && dotnet test --verbosity normal"

If the test project is .NET 4.8 x86, you may need:

ssh windev "cd C:\src\lmxproxy && dotnet build ZB.MOM.WW.LmxProxy.slnx --platform x86 && dotnet test tests/ZB.MOM.WW.LmxProxy.Host.Tests --platform x86"

Completion Criteria

• PerformanceMetrics class with OperationMetrics, MetricsStatistics, ITimingScope in src/ZB.MOM.WW.LmxProxy.Host/Metrics/
• HealthCheckService and DetailedHealthCheckService in src/ZB.MOM.WW.LmxProxy.Host/Health/
• StatusReportService with data model classes in src/ZB.MOM.WW.LmxProxy.Host/Status/
• StatusWebServer with HTML dashboard, JSON status, and health endpoints in src/ZB.MOM.WW.LmxProxy.Host/Status/
• All components wired into LmxProxyService.Start() / Stop()
• ScadaGrpcService uses PerformanceMetrics.BeginOperation() for Read, ReadBatch, Write, WriteBatch RPCs
• Unit tests for PerformanceMetrics (recording, percentile, rolling buffer, timing scope)
• Unit tests for HealthCheckService (healthy, unhealthy, degraded transitions)
• Unit tests for DetailedHealthCheckService (connected, quality, staleness)
• Unit tests for StatusReportService (JSON format, HTML format, health aggregation)
• Solution builds without errors: dotnet build ZB.MOM.WW.LmxProxy.slnx
• All tests pass: dotnet test