Files
lmxopcua/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/DriverResilienceOptionsTests.cs
T
Joseph Doherty dac7e1885d feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)
The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.

- No pipeline retry: the actor's RetryConnect timer is the retry loop
  (parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
  the reconnect timer fires every 10 s, so at most three attempts fit one
  window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
  connect failure; the 15 s break window costs at most one extra tick.
  The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
  instance's tracker snapshots with its pipelines — a respawn during an
  outage discarded the only breaker whose OnClosed could clear the OPEN
  flag, stranding a false 'Breaker OPEN' chip forever.

Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
2026-08-08 05:45:08 -04:00

228 lines
10 KiB
C#

using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.Resilience;
[Trait("Category", "Unit")]
public sealed class DriverResilienceOptionsTests
{
/// <summary>
/// 01/U-6 knob-inertness guard (OVERALL theme #1): the public property set of
/// <see cref="DriverResilienceOptions"/> must be EXACTLY the pipeline-wired knobs. Every option a
/// parser can populate must map to a strategy the builder actually composes — a parsed-but-unapplied
/// knob (the bulkhead genre this pass deleted) is inertness the interface-forwarding and
/// unwrapped-dispatch guards can't catch. To add a knob: wire it in <c>DriverResiliencePipelineBuilder</c>,
/// add a behavior test that proves it engages, THEN add it to the expected set below.
/// <para><c>RecycleIntervalSeconds</c> used to sit in the expected set under an explicit carve-out —
/// Tier-C-dormant per 01/U-2, "a documented open question, out of this pass's scope", justified by
/// the recycle scheduler rather than the Polly pipeline being its consumer. Gitea #522 closed that
/// question by deleting the scheduler (it was constructed only in tests, and the
/// <c>IDriverSupervisor</c> that would perform the recycle had no implementations), so the knob went
/// with it and the carve-out is gone. The expected set below is now literally what the name of this
/// test claims.</para>
/// </summary>
[Fact]
public void Options_properties_are_exactly_the_pipeline_wired_set()
{
var expected = new[] { "Tier", "CapabilityPolicies" };
var actual = typeof(DriverResilienceOptions)
.GetProperties()
.Select(p => p.Name)
.OrderBy(n => n)
.ToArray();
actual.ShouldBe(expected.OrderBy(n => n).ToArray(),
"DriverResilienceOptions must expose only pipeline-wired knobs — a new option needs a builder " +
"strategy + a behavior test before it is added to the expected set (guards against the deleted " +
"bulkhead 'parsed-but-unapplied' genre; see 01/U-6).");
}
/// <summary>Verifies that tier defaults cover every capability.</summary>
/// <param name="tier">The driver tier to test.</param>
[Theory]
[InlineData(DriverTier.A)]
[InlineData(DriverTier.B)]
[InlineData(DriverTier.C)]
public void TierDefaults_Cover_EveryCapability(DriverTier tier)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
foreach (var capability in Enum.GetValues<DriverCapability>())
defaults.ShouldContainKey(capability);
}
/// <summary>Verifies that write never retries by default.</summary>
/// <param name="tier">The driver tier to test.</param>
[Theory]
[InlineData(DriverTier.A)]
[InlineData(DriverTier.B)]
[InlineData(DriverTier.C)]
public void Write_NeverRetries_ByDefault(DriverTier tier)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
defaults[DriverCapability.Write].RetryCount.ShouldBe(0);
}
/// <summary>Verifies that alarm acknowledge never retries by default.</summary>
/// <param name="tier">The driver tier to test.</param>
[Theory]
[InlineData(DriverTier.A)]
[InlineData(DriverTier.B)]
[InlineData(DriverTier.C)]
public void AlarmAcknowledge_NeverRetries_ByDefault(DriverTier tier)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
defaults[DriverCapability.AlarmAcknowledge].RetryCount.ShouldBe(0);
}
/// <summary>
/// Initialize never retries by default — the <c>DriverInstanceActor</c> reconnect timer IS the
/// retry loop; pipeline retries would stack inside each timer tick.
/// </summary>
/// <param name="tier">The driver tier to test.</param>
[Theory]
[InlineData(DriverTier.A)]
[InlineData(DriverTier.B)]
[InlineData(DriverTier.C)]
public void Initialize_NeverRetries_ByDefault(DriverTier tier)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
defaults[DriverCapability.Initialize].RetryCount.ShouldBe(0);
}
/// <summary>
/// The Initialize breaker threshold must be reachable at the reconnect cadence (R2-10 T10):
/// the breaker samples over 30 s and the reconnect timer fires every 10 s, so at most THREE
/// init attempts fit one sampling window. A threshold above 3 can never open on connect
/// failures — the exact silent-inertness this capability exists to fix — so the default is
/// pinned at 2 (Polly's minimum enabled throughput). Tier C stays breaker-less, consistent
/// with every other Tier C capability (the supervisor's process-level breaker owns it).
/// </summary>
[Theory]
[InlineData(DriverTier.A, 2)]
[InlineData(DriverTier.B, 2)]
[InlineData(DriverTier.C, 0)]
public void Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence(DriverTier tier, int expectedThreshold)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
defaults[DriverCapability.Initialize].BreakerFailureThreshold.ShouldBe(expectedThreshold);
}
/// <summary>Verifies that idempotent capabilities retry by default.</summary>
/// <param name="tier">The driver tier to test.</param>
/// <param name="capability">The driver capability to test.</param>
[Theory]
[InlineData(DriverTier.A, DriverCapability.Read)]
[InlineData(DriverTier.A, DriverCapability.HistoryRead)]
[InlineData(DriverTier.B, DriverCapability.Discover)]
[InlineData(DriverTier.B, DriverCapability.Probe)]
[InlineData(DriverTier.C, DriverCapability.AlarmSubscribe)]
public void IdempotentCapabilities_Retry_ByDefault(DriverTier tier, DriverCapability capability)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
defaults[capability].RetryCount.ShouldBeGreaterThan(0);
}
/// <summary>Verifies that TierC disables circuit breaker deferring to supervisor.</summary>
[Fact]
public void TierC_DisablesCircuitBreaker_DeferringToSupervisor()
{
var defaults = DriverResilienceOptions.GetTierDefaults(DriverTier.C);
foreach (var (_, policy) in defaults)
policy.BreakerFailureThreshold.ShouldBe(0, "Tier C breaker is handled by the Proxy supervisor (decision #68)");
}
/// <summary>Verifies that TierA and TierB enable circuit breaker.</summary>
/// <param name="tier">The driver tier to test.</param>
[Theory]
[InlineData(DriverTier.A)]
[InlineData(DriverTier.B)]
public void TierAAndB_EnableCircuitBreaker(DriverTier tier)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
foreach (var (_, policy) in defaults)
policy.BreakerFailureThreshold.ShouldBeGreaterThan(0);
}
/// <summary>Verifies that resolve uses tier defaults when no override is set.</summary>
[Fact]
public void Resolve_Uses_TierDefaults_When_NoOverride()
{
var options = new DriverResilienceOptions { Tier = DriverTier.A };
var resolved = options.Resolve(DriverCapability.Read);
resolved.ShouldBe(DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Read]);
}
/// <summary>Verifies that resolve uses override when configured.</summary>
[Fact]
public void Resolve_Uses_Override_When_Configured()
{
var custom = new CapabilityPolicy(TimeoutSeconds: 42, RetryCount: 7, BreakerFailureThreshold: 9);
var options = new DriverResilienceOptions
{
Tier = DriverTier.A,
CapabilityPolicies = new Dictionary<DriverCapability, CapabilityPolicy>
{
[DriverCapability.Read] = custom,
},
};
options.Resolve(DriverCapability.Read).ShouldBe(custom);
options.Resolve(DriverCapability.Write).ShouldBe(
DriverResilienceOptions.GetTierDefaults(DriverTier.A)[DriverCapability.Write]);
}
/// <summary>
/// Core-010 regression: every <see cref="DriverCapability"/> value must successfully resolve
/// under every tier with a default <see cref="DriverResilienceOptions"/>. A future
/// enum-only addition that forgets to update <c>GetTierDefaults</c> would otherwise blow up
/// on the hot path with <see cref="KeyNotFoundException"/>.
/// </summary>
/// <param name="tier">The driver tier to test.</param>
[Theory]
[InlineData(DriverTier.A)]
[InlineData(DriverTier.B)]
[InlineData(DriverTier.C)]
public void Resolve_Returns_NonNull_Policy_For_Every_Capability(DriverTier tier)
{
var options = new DriverResilienceOptions { Tier = tier };
foreach (var capability in Enum.GetValues<DriverCapability>())
{
var policy = options.Resolve(capability);
policy.ShouldNotBeNull(
$"every DriverCapability must resolve to a non-null policy for tier {tier} — {capability} did not");
}
}
/// <summary>
/// Core-010 regression: when a capability is somehow missing from BOTH the override
/// map and the tier defaults (defensive — should be impossible thanks to the
/// <c>TierDefaults_Cover_EveryCapability</c> invariant, but is the failure mode the
/// finding flagged), <c>Resolve</c> must throw a diagnostic <see cref="KeyNotFoundException"/>
/// that names the missing capability and tier — not a bare lookup failure.
/// </summary>
[Fact]
public void Resolve_Throws_Diagnostic_When_Capability_Missing_From_Tier_Defaults()
{
// Use a CapabilityPolicies dict that purposely omits one capability and use reflection
// to confirm the message names the capability when the tier defaults also omit it.
// We can't easily mutate GetTierDefaults so we exercise the documented behavior on a
// synthetic non-tier-known capability (we cast an out-of-range enum value).
var options = new DriverResilienceOptions { Tier = DriverTier.A };
var bogus = (DriverCapability)int.MaxValue;
var ex = Should.Throw<KeyNotFoundException>(() => options.Resolve(bogus));
ex.Message.ShouldContain(bogus.ToString());
ex.Message.ShouldContain(DriverTier.A.ToString());
ex.Message.ShouldContain(nameof(DriverResilienceOptions.GetTierDefaults));
}
}