dac7e1885d
The connect path was the one capability call the resilience pipeline never saw, so a driver that could not connect contributed zero breaker input and the /hosts resilience panel read 'All targets healthy' against a dead endpoint. InitializeAsync now routes through the invoker as DriverCapability.Initialize, keyed on the driver-instance id. - No pipeline retry: the actor's RetryConnect timer is the retry loop (parser forces retryCount overrides to 0, same funnel as write-shaped). - Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three attempts fit one window — threshold 5 could never open. Pinned by its own test. - An open breaker fast-fails into the same InitializeFailed path as a real connect failure; the 15 s break window costs at most one extra tick. The best-effort teardown ShutdownAsync stays unwrapped. - Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the instance's tracker snapshots with its pipelines — a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag, stranding a false 'Breaker OPEN' chip forever. Live gate PASSED on docker-dev (image symbol-verified per container): rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat 5021→5020 forwarder recovered it (both breakers closed within 1 s, panel back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker events. Full suite 50/50 green. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
228 lines
10 KiB
C#
228 lines
10 KiB
C#
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
|
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.Resilience;
|
|
|
|
[Trait("Category", "Unit")]
|
|
public sealed class DriverResilienceOptionsTests
|
|
{
|
|
/// <summary>
|
|
/// 01/U-6 knob-inertness guard (OVERALL theme #1): the public property set of
|
|
/// <see cref="DriverResilienceOptions"/> must be EXACTLY the pipeline-wired knobs. Every option a
|
|
/// parser can populate must map to a strategy the builder actually composes — a parsed-but-unapplied
|
|
/// knob (the bulkhead genre this pass deleted) is inertness the interface-forwarding and
|
|
/// unwrapped-dispatch guards can't catch. To add a knob: wire it in <c>DriverResiliencePipelineBuilder</c>,
|
|
/// add a behavior test that proves it engages, THEN add it to the expected set below.
|
|
/// <para><c>RecycleIntervalSeconds</c> used to sit in the expected set under an explicit carve-out —
|
|
/// Tier-C-dormant per 01/U-2, "a documented open question, out of this pass's scope", justified by
|
|
/// the recycle scheduler rather than the Polly pipeline being its consumer. Gitea #522 closed that
|
|
/// question by deleting the scheduler (it was constructed only in tests, and the
|
|
/// <c>IDriverSupervisor</c> that would perform the recycle had no implementations), so the knob went
|
|
/// with it and the carve-out is gone. The expected set below is now literally what the name of this
|
|
/// test claims.</para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void Options_properties_are_exactly_the_pipeline_wired_set()
|
|
{
|
|
var expected = new[] { "Tier", "CapabilityPolicies" };
|
|
|
|
var actual = typeof(DriverResilienceOptions)
|
|
.GetProperties()
|
|
.Select(p => p.Name)
|
|
.OrderBy(n => n)
|
|
.ToArray();
|
|
|
|
actual.ShouldBe(expected.OrderBy(n => n).ToArray(),
|
|
"DriverResilienceOptions must expose only pipeline-wired knobs — a new option needs a builder " +
|
|
"strategy + a behavior test before it is added to the expected set (guards against the deleted " +
|
|
"bulkhead 'parsed-but-unapplied' genre; see 01/U-6).");
|
|
}
|
|
|
|
/// <summary>Verifies that tier defaults cover every capability.</summary>
|
|
/// <param name="tier">The driver tier to test.</param>
|
|
[Theory]
|
|
[InlineData(DriverTier.A)]
|
|
[InlineData(DriverTier.B)]
|
|
[InlineData(DriverTier.C)]
|
|
public void TierDefaults_Cover_EveryCapability(DriverTier tier)
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
|
|
|
|
foreach (var capability in Enum.GetValues<DriverCapability>())
|
|
defaults.ShouldContainKey(capability);
|
|
}
|
|
|
|
/// <summary>Verifies that write never retries by default.</summary>
|
|
/// <param name="tier">The driver tier to test.</param>
|
|
[Theory]
|
|
[InlineData(DriverTier.A)]
|
|
[InlineData(DriverTier.B)]
|
|
[InlineData(DriverTier.C)]
|
|
public void Write_NeverRetries_ByDefault(DriverTier tier)
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
|
|
defaults[DriverCapability.Write].RetryCount.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>Verifies that alarm acknowledge never retries by default.</summary>
|
|
/// <param name="tier">The driver tier to test.</param>
|
|
[Theory]
|
|
[InlineData(DriverTier.A)]
|
|
[InlineData(DriverTier.B)]
|
|
[InlineData(DriverTier.C)]
|
|
public void AlarmAcknowledge_NeverRetries_ByDefault(DriverTier tier)
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
|
|
defaults[DriverCapability.AlarmAcknowledge].RetryCount.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Initialize never retries by default — the <c>DriverInstanceActor</c> reconnect timer IS the
|
|
/// retry loop; pipeline retries would stack inside each timer tick.
|
|
/// </summary>
|
|
/// <param name="tier">The driver tier to test.</param>
|
|
[Theory]
|
|
[InlineData(DriverTier.A)]
|
|
[InlineData(DriverTier.B)]
|
|
[InlineData(DriverTier.C)]
|
|
public void Initialize_NeverRetries_ByDefault(DriverTier tier)
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
|
|
defaults[DriverCapability.Initialize].RetryCount.ShouldBe(0);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The Initialize breaker threshold must be reachable at the reconnect cadence (R2-10 T10):
|
|
/// the breaker samples over 30 s and the reconnect timer fires every 10 s, so at most THREE
|
|
/// init attempts fit one sampling window. A threshold above 3 can never open on connect
|
|
/// failures — the exact silent-inertness this capability exists to fix — so the default is
|
|
/// pinned at 2 (Polly's minimum enabled throughput). Tier C stays breaker-less, consistent
|
|
/// with every other Tier C capability (the supervisor's process-level breaker owns it).
|
|
/// </summary>
|
|
[Theory]
|
|
[InlineData(DriverTier.A, 2)]
|
|
[InlineData(DriverTier.B, 2)]
|
|
[InlineData(DriverTier.C, 0)]
|
|
public void Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence(DriverTier tier, int expectedThreshold)
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
|
|
defaults[DriverCapability.Initialize].BreakerFailureThreshold.ShouldBe(expectedThreshold);
|
|
}
|
|
|
|
/// <summary>Verifies that idempotent capabilities retry by default.</summary>
|
|
/// <param name="tier">The driver tier to test.</param>
|
|
/// <param name="capability">The driver capability to test.</param>
|
|
[Theory]
|
|
[InlineData(DriverTier.A, DriverCapability.Read)]
|
|
[InlineData(DriverTier.A, DriverCapability.HistoryRead)]
|
|
[InlineData(DriverTier.B, DriverCapability.Discover)]
|
|
[InlineData(DriverTier.B, DriverCapability.Probe)]
|
|
[InlineData(DriverTier.C, DriverCapability.AlarmSubscribe)]
|
|
public void IdempotentCapabilities_Retry_ByDefault(DriverTier tier, DriverCapability capability)
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
|
|
defaults[capability].RetryCount.ShouldBeGreaterThan(0);
|
|
}
|
|
|
|
/// <summary>Verifies that TierC disables circuit breaker deferring to supervisor.</summary>
|
|
[Fact]
|
|
public void TierC_DisablesCircuitBreaker_DeferringToSupervisor()
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(DriverTier.C);
|
|
|
|
foreach (var (_, policy) in defaults)
|
|
policy.BreakerFailureThreshold.ShouldBe(0, "Tier C breaker is handled by the Proxy supervisor (decision #68)");
|
|
}
|
|
|
|
/// <summary>Verifies that TierA and TierB enable circuit breaker.</summary>
|
|
/// <param name="tier">The driver tier to test.</param>
|
|
[Theory]
|
|
[InlineData(DriverTier.A)]
|
|
[InlineData(DriverTier.B)]
|
|
public void TierAAndB_EnableCircuitBreaker(DriverTier tier)
|
|
{
|
|
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
|
|
|
|
foreach (var (_, policy) in defaults)
|
|
policy.BreakerFailureThreshold.ShouldBeGreaterThan(0);
|
|
}
|
|
|
|
/// <summary>Verifies that resolve uses tier defaults when no override is set.</summary>
|
|
[Fact]
|
|
public void Resolve_Uses_TierDefaults_When_NoOverride()
|
|
{
|
|
var options = new DriverResilienceOptions { Tier = DriverTier.A };
|
|
|
|
var resolved = options.Resolve(DriverCapability.Read);
|
|
|
|
resolved.ShouldBe(DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Read]);
|
|
}
|
|
|
|
/// <summary>Verifies that resolve uses override when configured.</summary>
|
|
[Fact]
|
|
public void Resolve_Uses_Override_When_Configured()
|
|
{
|
|
var custom = new CapabilityPolicy(TimeoutSeconds: 42, RetryCount: 7, BreakerFailureThreshold: 9);
|
|
var options = new DriverResilienceOptions
|
|
{
|
|
Tier = DriverTier.A,
|
|
CapabilityPolicies = new Dictionary<DriverCapability, CapabilityPolicy>
|
|
{
|
|
[DriverCapability.Read] = custom,
|
|
},
|
|
};
|
|
|
|
options.Resolve(DriverCapability.Read).ShouldBe(custom);
|
|
options.Resolve(DriverCapability.Write).ShouldBe(
|
|
DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Write]);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Core-010 regression: every <see cref="DriverCapability"/> value must successfully resolve
|
|
/// under every tier with a default <see cref="DriverResilienceOptions"/>. A future
|
|
/// enum-only addition that forgets to update <c>GetTierDefaults</c> would otherwise blow up
|
|
/// on the hot path with <see cref="KeyNotFoundException"/>.
|
|
/// </summary>
|
|
/// <param name="tier">The driver tier to test.</param>
|
|
[Theory]
|
|
[InlineData(DriverTier.A)]
|
|
[InlineData(DriverTier.B)]
|
|
[InlineData(DriverTier.C)]
|
|
public void Resolve_Returns_NonNull_Policy_For_Every_Capability(DriverTier tier)
|
|
{
|
|
var options = new DriverResilienceOptions { Tier = tier };
|
|
|
|
foreach (var capability in Enum.GetValues<DriverCapability>())
|
|
{
|
|
var policy = options.Resolve(capability);
|
|
policy.ShouldNotBeNull(
|
|
$"every DriverCapability must resolve to a non-null policy for tier {tier} — {capability} did not");
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// Core-010 regression: when a capability is somehow missing from BOTH the override
|
|
/// map and the tier defaults (defensive — should be impossible thanks to the
|
|
/// <c>TierDefaults_Cover_EveryCapability</c> invariant, but is the failure mode the
|
|
/// finding flagged), <c>Resolve</c> must throw a diagnostic <see cref="KeyNotFoundException"/>
|
|
/// that names the missing capability and tier — not a bare lookup failure.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Resolve_Throws_Diagnostic_When_Capability_Missing_From_Tier_Defaults()
|
|
{
|
|
// Use a CapabilityPolicies dict that purposely omits one capability and use reflection
|
|
// to confirm the message names the capability when the tier defaults also omit it.
|
|
// We can't easily mutate GetTierDefaults so we exercise the documented behavior on a
|
|
// synthetic non-tier-known capability (we cast an out-of-range enum value).
|
|
var options = new DriverResilienceOptions { Tier = DriverTier.A };
|
|
var bogus = (DriverCapability)int.MaxValue;
|
|
|
|
var ex = Should.Throw<KeyNotFoundException>(() => options.Resolve(bogus));
|
|
ex.Message.ShouldContain(bogus.ToString());
|
|
ex.Message.ShouldContain(DriverTier.A.ToString());
|
|
ex.Message.ShouldContain(nameof(DriverResilienceOptions.GetTierDefaults));
|
|
}
|
|
}
|