55ae230fa4
- v2-ci.yml: upload-artifact@v4 -> @v3 + continue-on-error (act_runner reports as GHES and v4 refuses GHES, failing every run); runner-capacity diagnostic step; AKKA_TEST_TIMEFACTOR=3 on the test step. - Runtime/ControlPlane harnesses: akka.test.timefactor from AKKA_TEST_TIMEFACTOR (TestTimeFactor), single-expect-default 15s; ControlPlane gains PresenceBudget. - Leaked spinner threads: the infinite-loop timeout tests (Calculation, Host.IntegrationTests RoslynVirtualTagEvaluator, 3x Core.Scripting) pinned a core for the rest of the test process (abandoned, never stopped). Now a bounded 5 s wall-clock spin; outer compile-inclusive bound 30 s. - Real test races fixed: ScriptedAlarmEngine push-during-drain (Inactive is also the start state + persist-before-update => wait for the Cleared save); DriverReconnectE2e (both nodes host the driver; Created[] is last-writer-wins and ForceReconnect is a no-op while Connecting); ControlPlaneStarterOrder (lost-race heal waited for a 60 s refresh tick). - Fixed sleeps -> poll-for-condition: ReconnectSupervisor LastError, Modbus deadband steps (wait for observed reads), Modbus dispose hygiene precondition, FOCAS probe-loop log. - Short presence budgets -> PresenceBudget in the Runtime/ControlPlane tests seen failing on Gitea; Calculation Settle 10 -> 30 s. No assertion removed, nothing skipped. - infra/README.md §10 CI; stale TreatWarningsAsErrors comment in Directory.Build.props. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
176 lines
10 KiB
C#
176 lines
10 KiB
C#
using System.Collections.Concurrent;
|
|
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
|
|
|
/// <summary>
|
|
/// Opt-in fake <see cref="IDriverFactory"/> the <see cref="TwoNodeClusterHarness"/> can register
|
|
/// so a deployed driver actually reaches <c>Connected</c> (Healthy) in-process — no Docker sim and
|
|
/// no real backend. Materialises a <see cref="FakeReconnectDriver"/> for the <c>"Modbus"</c> driver
|
|
/// type (any other type returns <c>null</c>, mirroring <see cref="NullDriverFactory"/>).
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// The DI wiring resolves <see cref="IDriverFactory"/> from the container; the production
|
|
/// <c>AddOtOpcUaRuntime()</c> seeds <see cref="NullDriverFactory"/> via <c>TryAddSingleton</c>, so a
|
|
/// harness that wants live drivers registers this fake first (its registration wins). Every
|
|
/// initialize/connect succeeds, so the wrapping <c>DriverInstanceActor</c> walks
|
|
/// <c>Connecting → Connected → Healthy</c> and (on <c>ReconnectDriver</c>) drives
|
|
/// <c>ForceReconnect → Reconnecting → re-initialize → Connected</c> without any fault injection.
|
|
/// Each created driver is recorded in <see cref="Created"/> so a test can reach in and flip its
|
|
/// health to <see cref="DriverState.Reconnecting"/> just before issuing the reconnect command.
|
|
/// </remarks>
|
|
public sealed class FakeReconnectDriverFactory : IDriverFactory
|
|
{
|
|
/// <summary>The single driver type this fake factory materialises.</summary>
|
|
public const string FakeDriverType = "Modbus";
|
|
|
|
/// <summary>
|
|
/// Drivers this factory has created, keyed by <c>driverInstanceId</c>, so a test can retrieve the
|
|
/// live instance (e.g. to call <see cref="FakeReconnectDriver.ReportReconnecting"/> before
|
|
/// dispatching a reconnect command). Concurrent because the factory is invoked on the spawning
|
|
/// actor thread while the test reads on its own thread.
|
|
/// </summary>
|
|
public ConcurrentDictionary<string, FakeReconnectDriver> Created { get; } = new();
|
|
|
|
/// <summary>
|
|
/// Returns a <see cref="FakeReconnectDriver"/> when <paramref name="driverType"/> is
|
|
/// <see cref="FakeDriverType"/>; otherwise <c>null</c> (the host logs + skips the row). Records the
|
|
/// created driver in <see cref="Created"/> keyed by <paramref name="driverInstanceId"/>.
|
|
/// </summary>
|
|
/// <param name="driverType">The driver type name from the deployed <c>DriverInstance</c> row.</param>
|
|
/// <param name="driverInstanceId">The stable driver-instance identifier.</param>
|
|
/// <param name="driverConfigJson">The driver configuration as a JSON string (ignored by the fake).</param>
|
|
/// <returns>A new <see cref="FakeReconnectDriver"/>, or <c>null</c> for an unsupported type.</returns>
|
|
public IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)
|
|
{
|
|
if (driverType != FakeDriverType) return null;
|
|
var driver = new FakeReconnectDriver(driverInstanceId, driverType);
|
|
Created[driverInstanceId] = driver;
|
|
_all.Enqueue(driver);
|
|
return driver;
|
|
}
|
|
|
|
private readonly ConcurrentQueue<FakeReconnectDriver> _all = new();
|
|
|
|
/// <summary>
|
|
/// EVERY driver this factory has created for <paramref name="driverInstanceId"/>, in creation order.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// The harness registers ONE factory on BOTH nodes, and both nodes host a deployed driver, so
|
|
/// <see cref="Created"/> (last writer wins) holds whichever node spawned last — and that can change
|
|
/// after a test has read it. A test that flips or inspects "the" driver must use this instead
|
|
/// (#606: the reconnect E2E asserted on an instance the other node had just replaced).
|
|
/// </remarks>
|
|
/// <param name="driverInstanceId">The driver-instance identifier.</param>
|
|
/// <returns>A snapshot of the created instances.</returns>
|
|
public IReadOnlyList<FakeReconnectDriver> InstancesOf(string driverInstanceId) =>
|
|
_all.Where(d => d.DriverInstanceId == driverInstanceId).ToList();
|
|
|
|
/// <summary>Gets the driver-type names this factory can materialise.</summary>
|
|
public IReadOnlyCollection<string> SupportedTypes { get; } = new[] { FakeDriverType };
|
|
}
|
|
|
|
/// <summary>
|
|
/// A minimal in-process <see cref="IDriver"/> whose connect/initialize path always succeeds, so the
|
|
/// wrapping <c>DriverInstanceActor</c> reaches <c>Connected</c> and publishes a
|
|
/// <see cref="DriverState.Healthy"/> snapshot. Reads/writes/subscribes are benign no-ops; this exists
|
|
/// only to let a deployed driver walk the Healthy ↔ Reconnecting FSM in-process for E2E tests.
|
|
///
|
|
/// <para>The driver's reported health is controllable: a test calls <see cref="ReportReconnecting"/>
|
|
/// to make <see cref="GetHealth"/> return <see cref="DriverState.Reconnecting"/> (simulating a lost
|
|
/// connection — the realistic trigger for an operator Reconnect). The <c>DriverInstanceActor</c>'s
|
|
/// <c>ForceReconnect</c> handler POLLS <see cref="GetHealth"/> right after entering its Reconnecting
|
|
/// state, so that snapshot surfaces <see cref="DriverState.Reconnecting"/> on the driver-health topic.
|
|
/// The subsequent retry calls <see cref="InitializeAsync"/>, which clears the flag back to
|
|
/// <see cref="DriverState.Healthy"/>, so the next snapshot returns to Healthy.</para>
|
|
/// </summary>
|
|
public sealed class FakeReconnectDriver : IDriver
|
|
{
|
|
/// <summary>
|
|
/// Initializes a new <see cref="FakeReconnectDriver"/>.
|
|
/// </summary>
|
|
/// <param name="driverInstanceId">The stable logical driver-instance identifier.</param>
|
|
/// <param name="driverType">The driver type name.</param>
|
|
public FakeReconnectDriver(string driverInstanceId, string driverType)
|
|
{
|
|
DriverInstanceId = driverInstanceId;
|
|
DriverType = driverType;
|
|
}
|
|
|
|
/// <summary>Gets the stable logical ID of this driver instance.</summary>
|
|
public string DriverInstanceId { get; }
|
|
|
|
/// <summary>Gets the driver type name (e.g. "Modbus").</summary>
|
|
public string DriverType { get; }
|
|
|
|
/// <summary>
|
|
/// When <c>true</c>, <see cref="GetHealth"/> reports <see cref="DriverState.Reconnecting"/>.
|
|
/// <c>volatile</c> because the actor polls <see cref="GetHealth"/> from its own thread while the
|
|
/// test flips this from another via <see cref="ReportReconnecting"/>.
|
|
/// </summary>
|
|
private volatile bool _reconnecting;
|
|
|
|
/// <summary>Timestamp of the most recent successful initialize; surfaced as the last successful read.</summary>
|
|
private DateTime _lastSuccess = DateTime.UtcNow;
|
|
|
|
/// <summary>Backing field for <see cref="InitializeCount"/>; mutated via <see cref="Interlocked"/>.</summary>
|
|
private int _initializeCount;
|
|
|
|
/// <summary>
|
|
/// Number of times <see cref="InitializeAsync"/> has been invoked. Read by the test to prove a
|
|
/// reconnect genuinely re-initialised the driver through the full cluster path (≥ 2 means the
|
|
/// initial connect plus at least one reconnect retry). The actor's retry path runs on a thread-pool
|
|
/// thread, so the increment is <see cref="Interlocked"/> and the read is <see cref="Volatile"/>.
|
|
/// </summary>
|
|
public int InitializeCount => Volatile.Read(ref _initializeCount);
|
|
|
|
/// <summary>
|
|
/// Marks the driver as having lost its connection so the next <see cref="GetHealth"/> poll reports
|
|
/// <see cref="DriverState.Reconnecting"/>. The test calls this immediately before dispatching the
|
|
/// reconnect command, simulating the realistic operator-Reconnect trigger.
|
|
/// </summary>
|
|
public void ReportReconnecting() => _reconnecting = true;
|
|
|
|
/// <summary>
|
|
/// Connect/initialize path — always succeeds (returns a completed task), so the actor self-Tells
|
|
/// <c>InitializeSucceeded</c> and becomes <c>Connected</c>. This is the method that makes connect
|
|
/// succeed; the FSM's reconnect path re-invokes it and it succeeds again. Increments
|
|
/// <see cref="InitializeCount"/> and clears the reconnecting flag (initialize succeeded → healthy
|
|
/// again).
|
|
/// </summary>
|
|
/// <param name="driverConfigJson">The driver configuration JSON (ignored).</param>
|
|
/// <param name="cancellationToken">Cancellation token for the operation.</param>
|
|
/// <returns>A completed task — initialization always succeeds.</returns>
|
|
public Task InitializeAsync(string driverConfigJson, CancellationToken cancellationToken)
|
|
{
|
|
Interlocked.Increment(ref _initializeCount);
|
|
_lastSuccess = DateTime.UtcNow;
|
|
_reconnecting = false;
|
|
return Task.CompletedTask;
|
|
}
|
|
|
|
/// <summary>Applies a config change in place — a no-op that always succeeds.</summary>
|
|
/// <param name="driverConfigJson">The driver configuration JSON (ignored).</param>
|
|
/// <param name="cancellationToken">Cancellation token for the operation.</param>
|
|
/// <returns>A completed task — reinitialization always succeeds.</returns>
|
|
public Task ReinitializeAsync(string driverConfigJson, CancellationToken cancellationToken)
|
|
=> Task.CompletedTask;
|
|
|
|
/// <summary>Stops the driver — a no-op that always succeeds.</summary>
|
|
/// <param name="cancellationToken">Cancellation token for the operation.</param>
|
|
/// <returns>A completed task.</returns>
|
|
public Task ShutdownAsync(CancellationToken cancellationToken)
|
|
=> Task.CompletedTask;
|
|
|
|
/// <summary>
|
|
/// Returns a <see cref="DriverState.Reconnecting"/> snapshot when the test has flagged a lost
|
|
/// connection via <see cref="ReportReconnecting"/>; otherwise a <see cref="DriverState.Healthy"/>
|
|
/// snapshot. The actor polls this on every observable state change, so the published state tracks
|
|
/// this flag.
|
|
/// </summary>
|
|
/// <returns>A <see cref="DriverHealth"/> reflecting the controllable connection state.</returns>
|
|
public DriverHealth GetHealth() => _reconnecting
|
|
? new DriverHealth(DriverState.Reconnecting, _lastSuccess, null)
|
|
: new DriverHealth(DriverState.Healthy, _lastSuccess, null);
|
|
}
|