Phase 6.1 Stream C — health endpoints on :4841 + LogContextEnricher + Serilog JSON sink + CapabilityInvoker enrichment

Closes Stream C per docs/v2/implementation/phase-6-1-resilience-and-observability.md.

Core.Observability (new namespace):
- DriverHealthReport — pure-function aggregation over DriverHealthSnapshot list.
  Empty fleet = Healthy. Any Faulted = Faulted. Any Unknown/Initializing (no
  Faulted) = NotReady. Any Degraded or Reconnecting (no Faulted, no NotReady)
  = Degraded. Else Healthy. HttpStatus(verdict) maps to the Stream C.1 state
  matrix: Healthy/Degraded → 200, NotReady/Faulted → 503.
- LogContextEnricher — Serilog LogContext wrapper. Push(id, type, capability,
  correlationId) returns an IDisposable scope; inner log calls carry
  DriverInstanceId / DriverType / CapabilityName / CorrelationId structured
  properties automatically. NewCorrelationId = 12-hex-char GUID slice for
  cases where no OPC UA RequestHeader.RequestHandle is in flight.

CapabilityInvoker — now threads LogContextEnricher around every ExecuteAsync /
ExecuteWriteAsync call site. OtOpcUaServer passes driver.DriverType through
so logs correlate to the driver type too. Every capability call emits
structured fields per the Stream C.4 compliance check.

Server.Observability:
- HealthEndpointsHost — standalone HttpListener on http://localhost:4841/
  (loopback avoids Windows URL-ACL elevation; remote probing via reverse
  proxy or explicit netsh urlacl grant). Routes:
    /healthz → 200 when (configDbReachable OR usingStaleConfig); 503 otherwise.
      Body: status, uptimeSeconds, configDbReachable, usingStaleConfig.
    /readyz  → DriverHealthReport.Aggregate + HttpStatus mapping.
      Body: verdict, drivers[], degradedDrivers[], uptimeSeconds.
    anything else → 404.
  Disposal cooperative with the HttpListener shutdown.
- OpcUaApplicationHost starts the health host after the OPC UA server comes up
  and disposes it on shutdown. New OpcUaServerOptions knobs:
  HealthEndpointsEnabled (default true), HealthEndpointsPrefix (default
  http://localhost:4841/).

Program.cs:
- Serilog pipeline adds Enrich.FromLogContext + opt-in JSON file sink via
  `Serilog:WriteJson = true` appsetting. Uses Serilog.Formatting.Compact's
  CompactJsonFormatter (one JSON object per line — SIEMs like Splunk,
  Datadog, Graylog ingest without a regex parser).

Server.Tests:
- Existing 3 OpcUaApplicationHost integration tests now set
  HealthEndpointsEnabled=false to avoid port :4841 collisions under parallel
  execution.
- New HealthEndpointsHostTests (9): /healthz healthy empty fleet; stale-config
  returns 200 with flag; unreachable+no-cache returns 503; /readyz empty/
  Healthy/Faulted/Degraded/Initializing drivers return correct status and
  bodies; unknown path → 404. Uses ephemeral ports via Interlocked counter.

Core.Tests:
- DriverHealthReportTests (8): empty fleet, all-healthy, any-Faulted trumps,
  any-NotReady without Faulted, Degraded without Faulted/NotReady, HttpStatus
  per-verdict theory.
- LogContextEnricherTests (8): all 4 properties attach; scope disposes cleanly;
  NewCorrelationId shape; null/whitespace driverInstanceId throws.
- CapabilityInvokerEnrichmentTests (2): inner logs carry structured
  properties; no context leak outside the call site.

Full solution dotnet test: 1016 passing (baseline 906, +110 for Phase 6.1 so
far across Streams A+B+C). Pre-existing Client.CLI Subscribe flake unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

src/ZB.MOM.WW.OtOpcUa.Core/Observability/DriverHealthReport.cs

@@ -0,0 +1,86 @@
+using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Core.Observability;
+
+/// <summary>
+///     Domain-layer health aggregation for Phase 6.1 Stream C. Pure functions over the driver
+///     fleet — given each driver's <see cref="DriverState"/>, produce a <see cref="ReadinessVerdict"/>
+///     that maps to HTTP status codes at the endpoint layer.
+/// </summary>
+/// <remarks>
+///     State matrix per <c>docs/v2/implementation/phase-6-1-resilience-and-observability.md</c>
+///     §Stream C.1:
+///     <list type="bullet">
+///         <item><see cref="DriverState.Unknown"/> / <see cref="DriverState.Initializing"/>
+///             → /readyz 503 (not yet ready).</item>
+///         <item><see cref="DriverState.Healthy"/> → /readyz 200.</item>
+///         <item><see cref="DriverState.Degraded"/> → /readyz 200 with flagged driver IDs.</item>
+///         <item><see cref="DriverState.Faulted"/> → /readyz 503.</item>
+///     </list>
+///     The overall verdict is computed across the fleet: any Faulted → Faulted; any
+///     Unknown/Initializing → NotReady; any Degraded → Degraded; else Healthy. An empty fleet
+///     is Healthy (nothing to degrade).
+/// </remarks>
+public static class DriverHealthReport
+{
+    /// <summary>Compute the fleet-wide readiness verdict from per-driver states.</summary>
+    public static ReadinessVerdict Aggregate(IReadOnlyList<DriverHealthSnapshot> drivers)
+    {
+        ArgumentNullException.ThrowIfNull(drivers);
+        if (drivers.Count == 0) return ReadinessVerdict.Healthy;
+
+        var anyFaulted = drivers.Any(d => d.State == DriverState.Faulted);
+        if (anyFaulted) return ReadinessVerdict.Faulted;
+
+        var anyInitializing = drivers.Any(d =>
+            d.State == DriverState.Unknown || d.State == DriverState.Initializing);
+        if (anyInitializing) return ReadinessVerdict.NotReady;
+
+        // Reconnecting = driver alive but not serving live data; report as Degraded so /readyz
+        // stays 200 (the fleet can still serve cached / last-good data) while operators see the
+        // affected driver in the body.
+        var anyDegraded = drivers.Any(d =>
+            d.State == DriverState.Degraded || d.State == DriverState.Reconnecting);
+        if (anyDegraded) return ReadinessVerdict.Degraded;
+
+        return ReadinessVerdict.Healthy;
+    }
+
+    /// <summary>
+    ///     Map a <see cref="ReadinessVerdict"/> to the HTTP status the /readyz endpoint should
+    ///     return per the Stream C.1 state matrix.
+    /// </summary>
+    public static int HttpStatus(ReadinessVerdict verdict) => verdict switch
+    {
+        ReadinessVerdict.Healthy => 200,
+        ReadinessVerdict.Degraded => 200,
+        ReadinessVerdict.NotReady => 503,
+        ReadinessVerdict.Faulted => 503,
+        _ => 500,
+    };
+}
+
+/// <summary>Per-driver snapshot fed into <see cref="DriverHealthReport.Aggregate"/>.</summary>
+/// <param name="DriverInstanceId">Driver instance identifier (from <c>IDriver.DriverInstanceId</c>).</param>
+/// <param name="State">Current <see cref="DriverState"/> from <c>IDriver.GetHealth</c>.</param>
+/// <param name="DetailMessage">Optional driver-supplied detail (e.g. "primary PLC unreachable").</param>
+public sealed record DriverHealthSnapshot(
+    string DriverInstanceId,
+    DriverState State,
+    string? DetailMessage = null);
+
+/// <summary>Overall fleet readiness — derived from driver states by <see cref="DriverHealthReport.Aggregate"/>.</summary>
+public enum ReadinessVerdict
+{
+    /// <summary>All drivers Healthy (or fleet is empty).</summary>
+    Healthy,
+
+    /// <summary>At least one driver Degraded; none Faulted / NotReady.</summary>
+    Degraded,
+
+    /// <summary>At least one driver Unknown / Initializing; none Faulted.</summary>
+    NotReady,
+
+    /// <summary>At least one driver Faulted.</summary>
+    Faulted,
+}

src/ZB.MOM.WW.OtOpcUa.Core/Observability/LogContextEnricher.cs

@@ -0,0 +1,53 @@
+using Serilog.Context;
+using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Core.Observability;
+
+/// <summary>
+///     Convenience wrapper around Serilog <see cref="LogContext"/> — attaches the set of
+///     structured properties a capability call should carry (DriverInstanceId, DriverType,
+///     CapabilityName, CorrelationId). Callers wrap their call-site body in a <c>using</c>
+///     block; inner <c>Log.Information</c> / <c>Log.Warning</c> calls emit the context
+///     automatically via the Serilog enricher chain.
+/// </summary>
+/// <remarks>
+///     Per <c>docs/v2/implementation/phase-6-1-resilience-and-observability.md</c> §Stream C.2.
+///     The correlation ID should be the OPC UA <c>RequestHeader.RequestHandle</c> when in-flight;
+///     otherwise a short random GUID. Callers supply whichever is available.
+/// </remarks>
+public static class LogContextEnricher
+{
+    /// <summary>Attach the capability-call property set. Dispose the returned scope to pop.</summary>
+    public static IDisposable Push(string driverInstanceId, string driverType, DriverCapability capability, string correlationId)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
+        ArgumentException.ThrowIfNullOrWhiteSpace(driverType);
+        ArgumentException.ThrowIfNullOrWhiteSpace(correlationId);
+
+        var a = LogContext.PushProperty("DriverInstanceId", driverInstanceId);
+        var b = LogContext.PushProperty("DriverType", driverType);
+        var c = LogContext.PushProperty("CapabilityName", capability.ToString());
+        var d = LogContext.PushProperty("CorrelationId", correlationId);
+        return new CompositeScope(a, b, c, d);
+    }
+
+    /// <summary>
+    ///     Generate a short correlation ID when no OPC UA RequestHandle is available.
+    ///     12-hex-char slice of a GUID — long enough for log correlation, short enough to
+    ///     scan visually.
+    /// </summary>
+    public static string NewCorrelationId() => Guid.NewGuid().ToString("N")[..12];
+
+    private sealed class CompositeScope : IDisposable
+    {
+        private readonly IDisposable[] _inner;
+        public CompositeScope(params IDisposable[] inner) => _inner = inner;
+
+        public void Dispose()
+        {
+            // Reverse-order disposal matches Serilog's stack semantics.
+            for (var i = _inner.Length - 1; i >= 0; i--)
+                _inner[i].Dispose();
+        }
+    }
+}

src/ZB.MOM.WW.OtOpcUa.Core/Resilience/CapabilityInvoker.cs

@@ -1,5 +1,6 @@
 using Polly;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
+using ZB.MOM.WW.OtOpcUa.Core.Observability;
 
 namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
 
@@ -19,6 +20,7 @@ public sealed class CapabilityInvoker
 {
     private readonly DriverResiliencePipelineBuilder _builder;
     private readonly string _driverInstanceId;
+    private readonly string _driverType;
     private readonly Func<DriverResilienceOptions> _optionsAccessor;
 
     /// <summary>
@@ -30,16 +32,19 @@ public sealed class CapabilityInvoker
     ///     Snapshot accessor for the current resilience options. Invoked per call so Admin-edit +
     ///     pipeline-invalidate can take effect without restarting the invoker.
     /// </param>
+    /// <param name="driverType">Driver type name for structured-log enrichment (e.g. <c>"Modbus"</c>).</param>
     public CapabilityInvoker(
         DriverResiliencePipelineBuilder builder,
         string driverInstanceId,
-        Func<DriverResilienceOptions> optionsAccessor)
+        Func<DriverResilienceOptions> optionsAccessor,
+        string driverType = "Unknown")
     {
         ArgumentNullException.ThrowIfNull(builder);
         ArgumentNullException.ThrowIfNull(optionsAccessor);
 
         _builder = builder;
         _driverInstanceId = driverInstanceId;
+        _driverType = driverType;
         _optionsAccessor = optionsAccessor;
     }
 
@@ -54,7 +59,10 @@ public sealed class CapabilityInvoker
         ArgumentNullException.ThrowIfNull(callSite);
 
         var pipeline = ResolvePipeline(capability, hostName);
-        return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
+        using (LogContextEnricher.Push(_driverInstanceId, _driverType, capability, LogContextEnricher.NewCorrelationId()))
+        {
+            return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
+        }
     }
 
     /// <summary>Execute a void-returning capability call, honoring the per-capability pipeline.</summary>
@@ -67,7 +75,10 @@ public sealed class CapabilityInvoker
         ArgumentNullException.ThrowIfNull(callSite);
 
         var pipeline = ResolvePipeline(capability, hostName);
-        await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
+        using (LogContextEnricher.Push(_driverInstanceId, _driverType, capability, LogContextEnricher.NewCorrelationId()))
+        {
+            await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
+        }
     }
 
     /// <summary>
@@ -95,7 +106,10 @@ public sealed class CapabilityInvoker
                 },
             };
             var pipeline = _builder.GetOrCreate(_driverInstanceId, $"{hostName}::non-idempotent", DriverCapability.Write, noRetryOptions);
-            return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
+            using (LogContextEnricher.Push(_driverInstanceId, _driverType, DriverCapability.Write, LogContextEnricher.NewCorrelationId()))
+            {
+                return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
+            }
         }
 
         return await ExecuteAsync(DriverCapability.Write, hostName, callSite, cancellationToken).ConfigureAwait(false);

src/ZB.MOM.WW.OtOpcUa.Core/ZB.MOM.WW.OtOpcUa.Core.csproj

@@ -18,6 +18,7 @@
 
     <ItemGroup>
         <PackageReference Include="Polly.Core" Version="8.6.6"/>
+        <PackageReference Include="Serilog" Version="4.3.0"/>
     </ItemGroup>
 
     <ItemGroup>