fix(archreview #10): wire CapabilityInvoker into prod dispatch via a Polly-free seam
Remediates RESILIENCE-DISPATCH-GAP (surfaced by the 07/C-1 analyzer): the Phase 6.1 CapabilityInvoker resilience pipeline (retry / breaker / bulkhead / telemetry) was constructed only in tests — the production dispatch layer called driver-capability methods directly, bypassing it entirely. The filed plan (thread CapabilityInvoker directly into DriverInstanceActor) is INFEASIBLE: Runtime is deliberately Polly-free (references Core.Abstractions, not Core — the same boundary IDriverFactory documents). So this introduces a seam, mirroring IDriverFactory exactly: - Core.Abstractions: IDriverCapabilityInvoker (+ NullDriverCapabilityInvoker pass-through) and IDriverCapabilityInvokerFactory (+ null factory). - Core: CapabilityInvoker now implements the interface; new DriverCapabilityInvokerFactory builds a per-instance invoker over the process-singleton pipeline builder + status tracker + tier resolver. - Runtime: DriverInstanceActor takes an IDriverCapabilityInvoker (default = pass-through) and routes all 6 dispatch sites (write, alarm-ack, subscribe, unsubscribe, alarm-subscribe, discover) through it; per-host key resolved via IPerCallHostResolver for single-ref calls, driver-instance key for bulk calls. DriverHostActor builds + injects the real invoker per spawned driver. - Host: DriverFactoryBootstrap registers the tracker, pipeline builder, and concrete factory (needs DriverFactoryRegistry.GetTier). Runtime SCE resolves the factory from DI like IDriverFactory; pass-through on nodes without it bound. - Analyzer: IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync are now recognized wrapper homes (interface-typed invoker calls must suppress OTOPCUA0001 too). All 6 RESILIENCE-DISPATCH-GAP pragmas removed — the analyzer (an error in Runtime) is now the standing regression guard. Also adds retry / breaker-open / breaker-close LOGGING to the pipeline builder — the operator-facing observability surface (the Admin /hosts reader, Phase 6.1 Stream E.2/E.3, was never built, so the pipeline otherwise runs silently). Scope notes: - Tier-DEFAULT policy only: DriverInstanceSpec (the deploy artifact) does not carry the per-instance ResilienceConfig JSON, so overrides aren't applied yet (tracked follow-up: plumb ResilienceConfig through the composer/artifact). - GenericDriverNodeManager's call is NOT wired: that class is test-only scaffolding (zero production references — the Server has its own address-space path); its pragma is re-annotated accordingly, not left as a "tracked gap". Verification (unit + analyzer; live behavioral gate still pending — see FOLLOWUP-10): - Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001. - New Runtime guard (recording invoker): write routes via ExecuteWriteAsync with the IPerCallHostResolver host; subscribe via ExecuteAsync — proves runtime routing. - New analyzer test: the interface is a valid wrapper home. - New pipeline-builder test: retry events are logged. - Full solution builds clean (0 errors); Runtime.Tests 357, Core.Tests 238, Analyzers.Tests 32 all green; pass-through default keeps existing dispatch tests byte-for-byte unchanged.
This commit is contained in:
Joseph Doherty
@@ -1,4 +1,5 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Polly;
|
||||
using Polly.CircuitBreaker;
|
||||
using Polly.Retry;
|
||||
@@ -25,6 +26,7 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
private readonly ConcurrentDictionary<PipelineKey, ResiliencePipeline> _pipelines = new();
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly DriverResilienceStatusTracker? _statusTracker;
|
||||
private readonly ILogger? _logger;
|
||||
|
||||
/// <summary>Construct with the ambient clock (use <see cref="TimeProvider.System"/> in prod).</summary>
|
||||
/// <param name="timeProvider">Clock source for pipeline timeouts + breaker sampling. Defaults to system.</param>
|
||||
@@ -33,12 +35,19 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
/// <c>LastBreakerOpenUtc</c>, breaker-close resets failures. Feeds Admin <c>/hosts</c> +
|
||||
/// the Polly bulkhead-depth column. Absent tracker means no telemetry (unit tests +
|
||||
/// deployments that don't care about resilience observability).</param>
|
||||
/// <param name="logger">When non-null, retry / circuit-breaker-open / breaker-close events are
|
||||
/// logged (instance + host + capability + attempt/exception). This is the operator-facing
|
||||
/// observability surface for the resilience pipeline until the Admin <c>/hosts</c> reader
|
||||
/// (Phase 6.1 Stream E.2/E.3) ships — without it the pipeline runs silently. Absent logger =
|
||||
/// no resilience logging (unit tests).</param>
|
||||
public DriverResiliencePipelineBuilder(
|
||||
TimeProvider? timeProvider = null,
|
||||
DriverResilienceStatusTracker? statusTracker = null)
|
||||
DriverResilienceStatusTracker? statusTracker = null,
|
||||
ILogger? logger = null)
|
||||
{
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
_statusTracker = statusTracker;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -66,8 +75,8 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
|
||||
var key = new PipelineKey(driverInstanceId, hostName, capability);
|
||||
return _pipelines.GetOrAdd(key, static (k, state) => Build(
|
||||
k.DriverInstanceId, k.HostName, state.capability, state.options, state.timeProvider, state.tracker),
|
||||
(capability, options, timeProvider: _timeProvider, tracker: _statusTracker));
|
||||
k.DriverInstanceId, k.HostName, state.capability, state.options, state.timeProvider, state.tracker, state.logger),
|
||||
(capability, options, timeProvider: _timeProvider, tracker: _statusTracker, logger: _logger));
|
||||
}
|
||||
|
||||
/// <summary>Drop cached pipelines for one driver instance (e.g. on ResilienceConfig change). Test + Admin-reload use.</summary>
|
||||
@@ -93,7 +102,8 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
DriverCapability capability,
|
||||
DriverResilienceOptions options,
|
||||
TimeProvider timeProvider,
|
||||
DriverResilienceStatusTracker? tracker)
|
||||
DriverResilienceStatusTracker? tracker,
|
||||
ILogger? logger)
|
||||
{
|
||||
var policy = options.Resolve(capability);
|
||||
var builder = new ResiliencePipelineBuilder { TimeProvider = timeProvider };
|
||||
@@ -114,11 +124,14 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
MaxDelay = TimeSpan.FromSeconds(5),
|
||||
ShouldHandle = new PredicateBuilder().Handle<Exception>(ex => ex is not OperationCanceledException),
|
||||
};
|
||||
if (tracker is not null)
|
||||
if (tracker is not null || logger is not null)
|
||||
{
|
||||
retryOptions.OnRetry = args =>
|
||||
{
|
||||
tracker.RecordFailure(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
tracker?.RecordFailure(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
logger?.LogWarning(
|
||||
"Driver resilience retry: instance={DriverInstanceId} host={HostName} capability={Capability} attempt={Attempt}: {Reason}",
|
||||
driverInstanceId, hostName, capability, args.AttemptNumber, args.Outcome.Exception?.Message ?? "unknown");
|
||||
return default;
|
||||
};
|
||||
}
|
||||
@@ -135,18 +148,24 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
BreakDuration = TimeSpan.FromSeconds(15),
|
||||
ShouldHandle = new PredicateBuilder().Handle<Exception>(ex => ex is not OperationCanceledException),
|
||||
};
|
||||
if (tracker is not null)
|
||||
if (tracker is not null || logger is not null)
|
||||
{
|
||||
breakerOptions.OnOpened = args =>
|
||||
{
|
||||
tracker.RecordBreakerOpen(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
tracker?.RecordBreakerOpen(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
logger?.LogWarning(
|
||||
"Driver resilience circuit-breaker OPENED: instance={DriverInstanceId} host={HostName} capability={Capability} breakDuration={BreakDuration}: {Reason}",
|
||||
driverInstanceId, hostName, capability, args.BreakDuration, args.Outcome.Exception?.Message ?? "unknown");
|
||||
return default;
|
||||
};
|
||||
breakerOptions.OnClosed = args =>
|
||||
{
|
||||
// Closing the breaker means the target recovered — reset the consecutive-
|
||||
// failure counter so Admin UI stops flashing red for this host.
|
||||
tracker.RecordSuccess(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
tracker?.RecordSuccess(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
logger?.LogInformation(
|
||||
"Driver resilience circuit-breaker closed (recovered): instance={DriverInstanceId} host={HostName} capability={Capability}",
|
||||
driverInstanceId, hostName, capability);
|
||||
return default;
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user