fix(archreview #10): wire CapabilityInvoker into prod dispatch via a Polly-free seam
Remediates RESILIENCE-DISPATCH-GAP (surfaced by the 07/C-1 analyzer): the Phase 6.1 CapabilityInvoker resilience pipeline (retry / breaker / bulkhead / telemetry) was constructed only in tests — the production dispatch layer called driver-capability methods directly, bypassing it entirely. The filed plan (thread CapabilityInvoker directly into DriverInstanceActor) is INFEASIBLE: Runtime is deliberately Polly-free (references Core.Abstractions, not Core — the same boundary IDriverFactory documents). So this introduces a seam, mirroring IDriverFactory exactly: - Core.Abstractions: IDriverCapabilityInvoker (+ NullDriverCapabilityInvoker pass-through) and IDriverCapabilityInvokerFactory (+ null factory). - Core: CapabilityInvoker now implements the interface; new DriverCapabilityInvokerFactory builds a per-instance invoker over the process-singleton pipeline builder + status tracker + tier resolver. - Runtime: DriverInstanceActor takes an IDriverCapabilityInvoker (default = pass-through) and routes all 6 dispatch sites (write, alarm-ack, subscribe, unsubscribe, alarm-subscribe, discover) through it; per-host key resolved via IPerCallHostResolver for single-ref calls, driver-instance key for bulk calls. DriverHostActor builds + injects the real invoker per spawned driver. - Host: DriverFactoryBootstrap registers the tracker, pipeline builder, and concrete factory (needs DriverFactoryRegistry.GetTier). Runtime SCE resolves the factory from DI like IDriverFactory; pass-through on nodes without it bound. - Analyzer: IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync are now recognized wrapper homes (interface-typed invoker calls must suppress OTOPCUA0001 too). All 6 RESILIENCE-DISPATCH-GAP pragmas removed — the analyzer (an error in Runtime) is now the standing regression guard. Also adds retry / breaker-open / breaker-close LOGGING to the pipeline builder — the operator-facing observability surface (the Admin /hosts reader, Phase 6.1 Stream E.2/E.3, was never built, so the pipeline otherwise runs silently). Scope notes: - Tier-DEFAULT policy only: DriverInstanceSpec (the deploy artifact) does not carry the per-instance ResilienceConfig JSON, so overrides aren't applied yet (tracked follow-up: plumb ResilienceConfig through the composer/artifact). - GenericDriverNodeManager's call is NOT wired: that class is test-only scaffolding (zero production references — the Server has its own address-space path); its pragma is re-annotated accordingly, not left as a "tracked gap". Verification (unit + analyzer; live behavioral gate still pending — see FOLLOWUP-10): - Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001. - New Runtime guard (recording invoker): write routes via ExecuteWriteAsync with the IPerCallHostResolver host; subscribe via ExecuteAsync — proves runtime routing. - New analyzer test: the interface is a valid wrapper home. - New pipeline-builder test: retry events are logged. - Full solution builds clean (0 errors); Runtime.Tests 357, Core.Tests 238, Analyzers.Tests 32 all green; pass-through default keeps existing dispatch tests byte-for-byte unchanged.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,99 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Abstraction over the Phase 6.1 resilience pipeline that wraps a single driver instance's
|
||||
/// capability calls (retry / circuit-breaker / bulkhead / tracker telemetry). The Runtime
|
||||
/// dispatch layer (<c>DriverInstanceActor</c>) consumes this interface instead of the concrete
|
||||
/// <c>CapabilityInvoker</c> so the actor assembly does NOT pull in
|
||||
/// <c>ZB.MOM.WW.OtOpcUa.Core</c> (which drags in Polly + driver hosting) — the same boundary
|
||||
/// rationale as <see cref="IDriverFactory"/>. The fused Host binds a real invoker (built by
|
||||
/// <see cref="IDriverCapabilityInvokerFactory"/>) per spawned driver; when none is supplied the
|
||||
/// dispatch layer uses <see cref="NullDriverCapabilityInvoker"/>, a genuine pass-through.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The method shapes mirror <c>CapabilityInvoker</c> exactly so the concrete invoker satisfies
|
||||
/// this interface without adapters. Callers pass the resolved <c>hostName</c> per call (a
|
||||
/// multi-host driver resolves it from the tag reference via <see cref="IPerCallHostResolver"/>;
|
||||
/// a single-host driver passes its <see cref="IDriver.DriverInstanceId"/>) so per-host breaker
|
||||
/// isolation + bulkhead-depth accounting key on the right target.
|
||||
/// </remarks>
|
||||
public interface IDriverCapabilityInvoker
|
||||
{
|
||||
/// <summary>Execute a value-returning capability call through the per-capability pipeline.</summary>
|
||||
/// <typeparam name="TResult">Return type of the underlying driver call.</typeparam>
|
||||
/// <param name="capability">The driver capability being executed.</param>
|
||||
/// <param name="hostName">The resolved host name for pipeline keying + status tracking.</param>
|
||||
/// <param name="callSite">The async function to execute (the raw driver-capability call).</param>
|
||||
/// <param name="cancellationToken">Cancellation token for the operation.</param>
|
||||
/// <returns>The result of the underlying driver call.</returns>
|
||||
ValueTask<TResult> ExecuteAsync<TResult>(
|
||||
DriverCapability capability,
|
||||
string hostName,
|
||||
Func<CancellationToken, ValueTask<TResult>> callSite,
|
||||
CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>Execute a void-returning capability call through the per-capability pipeline.</summary>
|
||||
/// <param name="capability">The driver capability being executed.</param>
|
||||
/// <param name="hostName">The resolved host name for pipeline keying + status tracking.</param>
|
||||
/// <param name="callSite">The async function to execute (the raw driver-capability call).</param>
|
||||
/// <param name="cancellationToken">Cancellation token for the operation.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
ValueTask ExecuteAsync(
|
||||
DriverCapability capability,
|
||||
string hostName,
|
||||
Func<CancellationToken, ValueTask> callSite,
|
||||
CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Execute a <see cref="DriverCapability.Write"/> call honoring idempotence semantics — when
|
||||
/// <paramref name="isIdempotent"/> is <c>false</c>, retries are disabled regardless of the
|
||||
/// configured policy; when <c>true</c>, the call runs through the Write pipeline which may
|
||||
/// retry if the tier configuration permits.
|
||||
/// </summary>
|
||||
/// <typeparam name="TResult">Return type of the underlying driver write call.</typeparam>
|
||||
/// <param name="hostName">The resolved host name for pipeline keying + status tracking.</param>
|
||||
/// <param name="isIdempotent">Whether the write operation is safe to retry.</param>
|
||||
/// <param name="callSite">The async function to execute (the raw driver write call).</param>
|
||||
/// <param name="cancellationToken">Cancellation token for the operation.</param>
|
||||
/// <returns>The result of the underlying driver write call.</returns>
|
||||
ValueTask<TResult> ExecuteWriteAsync<TResult>(
|
||||
string hostName,
|
||||
bool isIdempotent,
|
||||
Func<CancellationToken, ValueTask<TResult>> callSite,
|
||||
CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pass-through <see cref="IDriverCapabilityInvoker"/> that invokes the call site directly with
|
||||
/// NO pipeline — no timeout, retry, breaker, or telemetry. Bound when the dispatch layer is
|
||||
/// constructed without a real invoker (unit tests, the F7 skeleton path, admin-only nodes that
|
||||
/// never run drivers). Keeps behavior byte-for-byte identical to a raw driver call so existing
|
||||
/// dispatch-layer tests are unaffected by the resilience wiring.
|
||||
/// </summary>
|
||||
public sealed class NullDriverCapabilityInvoker : IDriverCapabilityInvoker
|
||||
{
|
||||
/// <summary>The shared singleton instance.</summary>
|
||||
public static readonly NullDriverCapabilityInvoker Instance = new();
|
||||
private NullDriverCapabilityInvoker() { }
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask<TResult> ExecuteAsync<TResult>(
|
||||
DriverCapability capability,
|
||||
string hostName,
|
||||
Func<CancellationToken, ValueTask<TResult>> callSite,
|
||||
CancellationToken cancellationToken) => callSite(cancellationToken);
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask ExecuteAsync(
|
||||
DriverCapability capability,
|
||||
string hostName,
|
||||
Func<CancellationToken, ValueTask> callSite,
|
||||
CancellationToken cancellationToken) => callSite(cancellationToken);
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask<TResult> ExecuteWriteAsync<TResult>(
|
||||
string hostName,
|
||||
bool isIdempotent,
|
||||
Func<CancellationToken, ValueTask<TResult>> callSite,
|
||||
CancellationToken cancellationToken) => callSite(cancellationToken);
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Builds a per-driver-instance <see cref="IDriverCapabilityInvoker"/>. The Runtime
|
||||
/// <c>DriverHostActor</c> consumes this factory (resolved from DI, like <see cref="IDriverFactory"/>)
|
||||
/// to attach a resilience invoker to each spawned <c>DriverInstanceActor</c> WITHOUT the actor
|
||||
/// assembly referencing <c>ZB.MOM.WW.OtOpcUa.Core</c> (which drags in Polly). The fused Host
|
||||
/// binds a concrete factory closing over the process-singleton pipeline builder + status tracker +
|
||||
/// driver-tier resolver; when unbound, <see cref="NullDriverCapabilityInvokerFactory"/> yields the
|
||||
/// pass-through invoker so dispatch behaves exactly as before.
|
||||
/// </summary>
|
||||
public interface IDriverCapabilityInvokerFactory
|
||||
{
|
||||
/// <summary>
|
||||
/// Create the resilience invoker for one driver instance. The returned invoker is keyed on
|
||||
/// <paramref name="driverInstanceId"/> and applies the tier policy resolved from
|
||||
/// <paramref name="driverType"/>.
|
||||
/// </summary>
|
||||
/// <param name="driverInstanceId">The driver instance identifier (pipeline + tracker key).</param>
|
||||
/// <param name="driverType">The driver type name, used to resolve the resilience tier defaults.</param>
|
||||
/// <returns>A per-instance <see cref="IDriverCapabilityInvoker"/>.</returns>
|
||||
IDriverCapabilityInvoker Create(string driverInstanceId, string driverType);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns <see cref="NullDriverCapabilityInvoker.Instance"/> for every driver instance. Bound
|
||||
/// when the fused Host hasn't registered a real resilience factory (Mac dev path, smoke tests,
|
||||
/// admin-only nodes). Dispatch then routes calls straight through with no pipeline.
|
||||
/// </summary>
|
||||
public sealed class NullDriverCapabilityInvokerFactory : IDriverCapabilityInvokerFactory
|
||||
{
|
||||
/// <summary>The shared singleton instance.</summary>
|
||||
public static readonly NullDriverCapabilityInvokerFactory Instance = new();
|
||||
private NullDriverCapabilityInvokerFactory() { }
|
||||
|
||||
/// <inheritdoc />
|
||||
public IDriverCapabilityInvoker Create(string driverInstanceId, string driverType) =>
|
||||
NullDriverCapabilityInvoker.Instance;
|
||||
}
|
||||
@@ -68,7 +68,7 @@ public class GenericDriverNodeManager(IDriver driver) : IDisposable
|
||||
throw new NotSupportedException($"Driver '{Driver.DriverInstanceId}' does not implement ITagDiscovery.");
|
||||
|
||||
var capturing = new CapturingBuilder(builder, _alarmSinks);
|
||||
#pragma warning disable OTOPCUA0001 // RESILIENCE-DISPATCH-GAP (07/C-1): dispatch bypasses CapabilityInvoker — Phase 6.1 pipeline built but never wired into GenericDriverNodeManager; tracked, NOT intentional.
|
||||
#pragma warning disable OTOPCUA0001 // NOT a production dispatch path: this Core scaffolding node-manager is exercised only by GenericDriverNodeManagerTests — the live server materialises the address space via the Server project's own OpcUaAddressSpaceBuilder path. No CapabilityInvoker by design (verified 07/#10: zero production references). If this class is ever wired into live dispatch, route this call through IDriverCapabilityInvoker.
|
||||
await discovery.DiscoverAsync(capturing, ct);
|
||||
#pragma warning restore OTOPCUA0001
|
||||
|
||||
|
||||
@@ -7,16 +7,21 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
/// <summary>
|
||||
/// Executes driver-capability calls through a shared Polly pipeline. One invoker per
|
||||
/// <c>(DriverInstance, IDriver)</c> pair; the underlying <see cref="DriverResiliencePipelineBuilder"/>
|
||||
/// is process-singleton so all invokers share its cache.
|
||||
/// is process-singleton so all invokers share its cache. Implements
|
||||
/// <see cref="IDriverCapabilityInvoker"/> so the Polly-free Runtime dispatch layer can consume it
|
||||
/// through the abstraction (bound per-driver by <see cref="IDriverCapabilityInvokerFactory"/>).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The server's dispatch
|
||||
/// layer routes every capability call (<c>IReadable.ReadAsync</c>, <c>IWritable.WriteAsync</c>,
|
||||
/// <c>ITagDiscovery.DiscoverAsync</c>, <c>ISubscribable.SubscribeAsync/UnsubscribeAsync</c>,
|
||||
/// <c>IHostConnectivityProbe</c> probe loop, <c>IAlarmSource.SubscribeAlarmsAsync/AcknowledgeAsync</c>,
|
||||
/// and all four <c>IHistoryProvider</c> reads) through this invoker.
|
||||
/// The Runtime dispatch layer (<c>DriverInstanceActor</c>) routes its capability calls
|
||||
/// (<c>IWritable.WriteAsync</c>, <c>ITagDiscovery.DiscoverAsync</c>,
|
||||
/// <c>ISubscribable.SubscribeAsync/UnsubscribeAsync</c>,
|
||||
/// <c>IAlarmSource.SubscribeAlarmsAsync/AcknowledgeAsync</c>) through this invoker via the
|
||||
/// <see cref="IDriverCapabilityInvoker"/> seam. The <c>UnwrappedCapabilityCallAnalyzer</c>
|
||||
/// (OTOPCUA0001) enforces that every guarded call is wrapped in an <see cref="ExecuteAsync{TResult}"/>
|
||||
/// / <see cref="ExecuteWriteAsync{TResult}"/> lambda (the interface's methods are registered wrapper
|
||||
/// homes too, so an interface-typed invoker call suppresses the diagnostic).
|
||||
/// </remarks>
|
||||
public sealed class CapabilityInvoker
|
||||
public sealed class CapabilityInvoker : IDriverCapabilityInvoker
|
||||
{
|
||||
private readonly DriverResiliencePipelineBuilder _builder;
|
||||
private readonly string _driverInstanceId;
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
|
||||
/// <summary>
|
||||
/// Concrete <see cref="IDriverCapabilityInvokerFactory"/> that builds a per-driver-instance
|
||||
/// <see cref="CapabilityInvoker"/> over the process-singleton pipeline builder + status tracker.
|
||||
/// The fused Host registers this (it references <c>Core</c> + Polly); the Runtime dispatch layer
|
||||
/// resolves the abstraction from DI and never sees this type.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Resilience policy is currently the <b>tier default</b> for the driver type — resolved
|
||||
/// via the injected tier resolver (backed by <c>DriverFactoryRegistry.GetTier</c>). The
|
||||
/// per-instance <c>DriverInstance.ResilienceConfig</c> JSON override is NOT yet applied because
|
||||
/// the deployment artifact (<c>DriverInstanceSpec</c>) does not carry it; plumbing that column
|
||||
/// through the composer/artifact is a tracked follow-up. Until then every instance of a given
|
||||
/// driver type gets that type's tier defaults.</para>
|
||||
///
|
||||
/// <para>Options are snapshotted once per <see cref="Create"/> (tier is fixed for a driver type),
|
||||
/// so the invoker's per-call options accessor is allocation-free on the hot path.</para>
|
||||
/// </remarks>
|
||||
public sealed class DriverCapabilityInvokerFactory : IDriverCapabilityInvokerFactory
|
||||
{
|
||||
private readonly DriverResiliencePipelineBuilder _builder;
|
||||
private readonly DriverResilienceStatusTracker _statusTracker;
|
||||
private readonly Func<string, DriverTier> _tierResolver;
|
||||
|
||||
/// <summary>Construct the factory over the shared resilience infrastructure.</summary>
|
||||
/// <param name="builder">Process-singleton Polly pipeline builder (shared pipeline cache).</param>
|
||||
/// <param name="statusTracker">Process-singleton resilience status tracker feeding Admin <c>/hosts</c>.</param>
|
||||
/// <param name="tierResolver">Resolves a driver type name to its <see cref="DriverTier"/> (e.g. <c>DriverFactoryRegistry.GetTier</c>).</param>
|
||||
public DriverCapabilityInvokerFactory(
|
||||
DriverResiliencePipelineBuilder builder,
|
||||
DriverResilienceStatusTracker statusTracker,
|
||||
Func<string, DriverTier> tierResolver)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(builder);
|
||||
ArgumentNullException.ThrowIfNull(statusTracker);
|
||||
ArgumentNullException.ThrowIfNull(tierResolver);
|
||||
_builder = builder;
|
||||
_statusTracker = statusTracker;
|
||||
_tierResolver = tierResolver;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IDriverCapabilityInvoker Create(string driverInstanceId, string driverType)
|
||||
{
|
||||
var tier = _tierResolver(driverType);
|
||||
// No per-instance ResilienceConfig in the artifact yet — tier defaults only (see remarks).
|
||||
var options = DriverResilienceOptionsParser.ParseOrDefaults(tier, resilienceConfigJson: null, out _);
|
||||
return new CapabilityInvoker(
|
||||
_builder,
|
||||
driverInstanceId,
|
||||
optionsAccessor: () => options,
|
||||
driverType: driverType,
|
||||
statusTracker: _statusTracker);
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Polly;
|
||||
using Polly.CircuitBreaker;
|
||||
using Polly.Retry;
|
||||
@@ -25,6 +26,7 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
private readonly ConcurrentDictionary<PipelineKey, ResiliencePipeline> _pipelines = new();
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly DriverResilienceStatusTracker? _statusTracker;
|
||||
private readonly ILogger? _logger;
|
||||
|
||||
/// <summary>Construct with the ambient clock (use <see cref="TimeProvider.System"/> in prod).</summary>
|
||||
/// <param name="timeProvider">Clock source for pipeline timeouts + breaker sampling. Defaults to system.</param>
|
||||
@@ -33,12 +35,19 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
/// <c>LastBreakerOpenUtc</c>, breaker-close resets failures. Feeds Admin <c>/hosts</c> +
|
||||
/// the Polly bulkhead-depth column. Absent tracker means no telemetry (unit tests +
|
||||
/// deployments that don't care about resilience observability).</param>
|
||||
/// <param name="logger">When non-null, retry / circuit-breaker-open / breaker-close events are
|
||||
/// logged (instance + host + capability + attempt/exception). This is the operator-facing
|
||||
/// observability surface for the resilience pipeline until the Admin <c>/hosts</c> reader
|
||||
/// (Phase 6.1 Stream E.2/E.3) ships — without it the pipeline runs silently. Absent logger =
|
||||
/// no resilience logging (unit tests).</param>
|
||||
public DriverResiliencePipelineBuilder(
|
||||
TimeProvider? timeProvider = null,
|
||||
DriverResilienceStatusTracker? statusTracker = null)
|
||||
DriverResilienceStatusTracker? statusTracker = null,
|
||||
ILogger? logger = null)
|
||||
{
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
_statusTracker = statusTracker;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -66,8 +75,8 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
|
||||
var key = new PipelineKey(driverInstanceId, hostName, capability);
|
||||
return _pipelines.GetOrAdd(key, static (k, state) => Build(
|
||||
k.DriverInstanceId, k.HostName, state.capability, state.options, state.timeProvider, state.tracker),
|
||||
(capability, options, timeProvider: _timeProvider, tracker: _statusTracker));
|
||||
k.DriverInstanceId, k.HostName, state.capability, state.options, state.timeProvider, state.tracker, state.logger),
|
||||
(capability, options, timeProvider: _timeProvider, tracker: _statusTracker, logger: _logger));
|
||||
}
|
||||
|
||||
/// <summary>Drop cached pipelines for one driver instance (e.g. on ResilienceConfig change). Test + Admin-reload use.</summary>
|
||||
@@ -93,7 +102,8 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
DriverCapability capability,
|
||||
DriverResilienceOptions options,
|
||||
TimeProvider timeProvider,
|
||||
DriverResilienceStatusTracker? tracker)
|
||||
DriverResilienceStatusTracker? tracker,
|
||||
ILogger? logger)
|
||||
{
|
||||
var policy = options.Resolve(capability);
|
||||
var builder = new ResiliencePipelineBuilder { TimeProvider = timeProvider };
|
||||
@@ -114,11 +124,14 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
MaxDelay = TimeSpan.FromSeconds(5),
|
||||
ShouldHandle = new PredicateBuilder().Handle<Exception>(ex => ex is not OperationCanceledException),
|
||||
};
|
||||
if (tracker is not null)
|
||||
if (tracker is not null || logger is not null)
|
||||
{
|
||||
retryOptions.OnRetry = args =>
|
||||
{
|
||||
tracker.RecordFailure(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
tracker?.RecordFailure(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
logger?.LogWarning(
|
||||
"Driver resilience retry: instance={DriverInstanceId} host={HostName} capability={Capability} attempt={Attempt}: {Reason}",
|
||||
driverInstanceId, hostName, capability, args.AttemptNumber, args.Outcome.Exception?.Message ?? "unknown");
|
||||
return default;
|
||||
};
|
||||
}
|
||||
@@ -135,18 +148,24 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
BreakDuration = TimeSpan.FromSeconds(15),
|
||||
ShouldHandle = new PredicateBuilder().Handle<Exception>(ex => ex is not OperationCanceledException),
|
||||
};
|
||||
if (tracker is not null)
|
||||
if (tracker is not null || logger is not null)
|
||||
{
|
||||
breakerOptions.OnOpened = args =>
|
||||
{
|
||||
tracker.RecordBreakerOpen(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
tracker?.RecordBreakerOpen(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
logger?.LogWarning(
|
||||
"Driver resilience circuit-breaker OPENED: instance={DriverInstanceId} host={HostName} capability={Capability} breakDuration={BreakDuration}: {Reason}",
|
||||
driverInstanceId, hostName, capability, args.BreakDuration, args.Outcome.Exception?.Message ?? "unknown");
|
||||
return default;
|
||||
};
|
||||
breakerOptions.OnClosed = args =>
|
||||
{
|
||||
// Closing the breaker means the target recovered — reset the consecutive-
|
||||
// failure counter so Admin UI stops flashing red for this host.
|
||||
tracker.RecordSuccess(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
tracker?.RecordSuccess(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
logger?.LogInformation(
|
||||
"Driver resilience circuit-breaker closed (recovered): instance={DriverInstanceId} host={HostName} capability={Capability}",
|
||||
driverInstanceId, hostName, capability);
|
||||
return default;
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user