Joseph Doherty
99da32d635
P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives up on response headers after ConnectedHeaderTimeout (10s) and then treats the first received EVENT as its fallback connected signal, so an alarm-only site that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight= true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight) skip every tick until the site's 4h max stream lifetime ends the call. Fix: bound one open attempt with a generation-fenced single-shot deadline (GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the 10s header timeout, below the 60s reconcile interval). Firing it releases the flag and nothing else: no retry-budget spend, no node flip, no liveness republish, and the abandoned subscription is left running so a merely-slow connect still comes up normally. Self-healing rather than reachability analysis. The deadline is cancelled on connect / error / graceful completion. P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate correlation id and only THEN checked the concurrency cap. Checking Count >= max after the removal is right at exactly the cap but wrong above it, and above it is reachable because the cap check is documented check-then-act: two subscribes that both clear it before either registers overshoot by one. In that window a routine reconnect killed a healthy stream AND was refused with ResourceExhausted, leaving the correlation with ZERO streams until its next retry. Fix: claim the removal, consult the cap only for a genuinely NEW correlation id, then cancel. A replacement is net-neutral on the count so it never consults the cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so the incumbent survives. Newest-wins is unchanged — a fresh authenticated subscribe still displaces a dead/wedged incumbent. Tests (both shown failing against the unfixed code): - OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen — fails (reopen never happens) with the deadline arming disabled. - OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget — inertness lock on every healthy path. - AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream — fails with ResourceExhausted against the old ordering; the overshoot is driven deterministically by parking both openers on the ServerCallContext token read, which sits between the cap check and the _activeStreams insert. - AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives. The P3-4 test's doc comment carries the honesty note: the aggregator-side starvation reproduces exactly, but how often the >10s header delay that gets you there occurs in production is NOT demonstrated — the site flushes headers explicitly as soon as its relay is attached, so the budget has to be consumed by connect + handshake + server scheduling instead.
965 lines
49 KiB
C#
965 lines
49 KiB
C#
using System.Collections.Concurrent;
|
|
using Akka.Actor;
|
|
using Akka.TestKit.Xunit2;
|
|
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
|
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
|
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
|
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
|
|
|
|
/// <summary>
|
|
/// Tests for <see cref="SiteAlarmAggregatorActor"/> (plan #10, Task 4) — the transient,
|
|
/// in-memory, per-site live alarm aggregator. Covers the tricky bits: seed-then-stream
|
|
/// ordering (a delta arriving before the seed completes is neither lost nor double-applied),
|
|
/// dedup by AlarmKey, placeholder seeded-but-not-on-stream coherence, NodeA↔NodeB reconnect
|
|
/// re-seed, and periodic reconcile authoritative-replace (drift correction).
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Serialized with <c>SiteAlarmLiveCacheTelemetryTests</c> (register row 102): every actor
|
|
/// this class spins up moves the process-global
|
|
/// <c>scadabridge.site.alarm_cache.aggregators.active</c> gauge via
|
|
/// <see cref="SiteAlarmAggregatorActor"/>'s PreStart/PostStop, which the telemetry test's
|
|
/// fixed-baseline assertions require to hold still (or only increase) between its capture
|
|
/// and its checks. Same collection ⇒ never concurrent.
|
|
/// </remarks>
|
|
[Collection("SiteAlarmAggregatorGauge")]
|
|
public class SiteAlarmAggregatorActorTests : TestKit
|
|
{
|
|
private const string SiteId = "site-alpha";
|
|
private const string GrpcNodeA = "http://localhost:5100";
|
|
private const string GrpcNodeB = "http://localhost:5200";
|
|
private const string Instance = "Site1.Pump01";
|
|
|
|
public SiteAlarmAggregatorActorTests() : base(@"akka.loglevel = WARNING")
|
|
{
|
|
}
|
|
|
|
// Former process-global static test seams, now ctor-injected per actor (R2 T12).
|
|
private static readonly TimeSpan TestReconnectDelay = TimeSpan.FromMilliseconds(50);
|
|
private static readonly TimeSpan TestStabilityWindow = TimeSpan.FromSeconds(30);
|
|
|
|
// ── Test doubles ────────────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// Controllable seed fan-out: each invocation returns a fresh Task the test completes
|
|
/// on demand (so seed/reconcile timing relative to buffered deltas is deterministic).
|
|
/// </summary>
|
|
private sealed class SeedStub
|
|
{
|
|
private readonly ConcurrentQueue<TaskCompletionSource<IReadOnlyList<AlarmStateChanged>>> _pending = new();
|
|
private int _callCount;
|
|
|
|
public int CallCount => Volatile.Read(ref _callCount);
|
|
|
|
public Task<IReadOnlyList<AlarmStateChanged>> Seed(CancellationToken ct)
|
|
{
|
|
Interlocked.Increment(ref _callCount);
|
|
var tcs = new TaskCompletionSource<IReadOnlyList<AlarmStateChanged>>(
|
|
TaskCreationOptions.RunContinuationsAsynchronously);
|
|
ct.Register(() => tcs.TrySetCanceled());
|
|
_pending.Enqueue(tcs);
|
|
return tcs.Task;
|
|
}
|
|
|
|
/// <summary>Completes the oldest not-yet-completed seed call with the given rows.</summary>
|
|
public void CompleteNext(params AlarmStateChanged[] rows)
|
|
{
|
|
SpinWait.SpinUntil(() => _pending.TryPeek(out _), TimeSpan.FromSeconds(3));
|
|
if (_pending.TryDequeue(out var tcs))
|
|
tcs.TrySetResult(rows);
|
|
else
|
|
throw new InvalidOperationException("No pending seed to complete.");
|
|
}
|
|
|
|
/// <summary>Faults the oldest not-yet-completed seed call (simulates a whole-fan-out failure).</summary>
|
|
public void FaultNext()
|
|
{
|
|
SpinWait.SpinUntil(() => _pending.TryPeek(out _), TimeSpan.FromSeconds(3));
|
|
if (_pending.TryDequeue(out var tcs))
|
|
tcs.TrySetException(new InvalidOperationException("seed boom"));
|
|
else
|
|
throw new InvalidOperationException("No pending seed to fault.");
|
|
}
|
|
}
|
|
|
|
private sealed class PublishSink
|
|
{
|
|
private readonly object _lock = new();
|
|
public List<IReadOnlyList<AlarmStateChanged>> Snapshots { get; } = new();
|
|
|
|
public void Publish(IReadOnlyList<AlarmStateChanged> snapshot, bool streamLive)
|
|
{
|
|
lock (_lock) { Snapshots.Add(snapshot); StreamLive = streamLive; }
|
|
}
|
|
|
|
/// <summary>Liveness reported with the most recent publish.</summary>
|
|
public bool StreamLive { get; private set; }
|
|
|
|
public IReadOnlyList<AlarmStateChanged>? Latest
|
|
{
|
|
get { lock (_lock) { return Snapshots.Count == 0 ? null : Snapshots[^1]; } }
|
|
}
|
|
|
|
public int Count { get { lock (_lock) { return Snapshots.Count; } } }
|
|
}
|
|
|
|
private sealed record SiteSub(
|
|
string CorrelationId, Action<AlarmStateChanged> OnAlarm, Action<Exception> OnError,
|
|
Action OnCompleted, CancellationToken Ct, Action? OnConnected);
|
|
|
|
private sealed class MockSiteAlarmStreamClient : SiteStreamGrpcClient
|
|
{
|
|
private readonly object _lock = new();
|
|
private readonly List<SiteSub> _subs = new();
|
|
private readonly List<string> _unsubscribed = new();
|
|
|
|
public List<SiteSub> Subs { get { lock (_lock) { return _subs.ToList(); } } }
|
|
public List<string> Unsubscribed { get { lock (_lock) { return _unsubscribed.ToList(); } } }
|
|
|
|
public MockSiteAlarmStreamClient() : base() { }
|
|
|
|
/// <summary>When false the stream never reports connected — the site accepted the TCP
|
|
/// call but never answered, so the aggregator's connect-driven re-seed must not run.</summary>
|
|
public bool AutoConnect { get; set; } = true;
|
|
|
|
public override Task SubscribeSiteAsync(
|
|
string correlationId, Action<AlarmStateChanged> onAlarmEvent, Action<Exception> onError,
|
|
Action onCompleted, CancellationToken ct, Action? onConnected = null)
|
|
{
|
|
lock (_lock)
|
|
{
|
|
_subs.Add(new SiteSub(correlationId, onAlarmEvent, onError, onCompleted, ct, onConnected));
|
|
}
|
|
// A healthy site accepts the subscription immediately (it flushes response headers
|
|
// as soon as its relay is attached), which is the aggregator's connect signal.
|
|
if (AutoConnect) onConnected?.Invoke();
|
|
var tcs = new TaskCompletionSource();
|
|
ct.Register(() => tcs.TrySetResult());
|
|
return tcs.Task; // never completes until cancelled (simulates a live stream)
|
|
}
|
|
|
|
public override void Unsubscribe(string correlationId)
|
|
{
|
|
lock (_lock) { _unsubscribed.Add(correlationId); }
|
|
}
|
|
}
|
|
|
|
private sealed class MockSiteAlarmStreamClientFactory : SiteStreamGrpcClientFactory
|
|
{
|
|
private readonly ConcurrentDictionary<string, MockSiteAlarmStreamClient> _byEndpoint = new();
|
|
|
|
public MockSiteAlarmStreamClientFactory()
|
|
: base(Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance) { }
|
|
|
|
/// <summary>Applied to every client this factory hands out (set before the actor starts).</summary>
|
|
public bool AutoConnect { get; set; } = true;
|
|
|
|
public MockSiteAlarmStreamClient ClientFor(string endpoint) =>
|
|
_byEndpoint.GetOrAdd(endpoint, _ => new MockSiteAlarmStreamClient { AutoConnect = AutoConnect });
|
|
|
|
public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint)
|
|
=> ClientFor(grpcEndpoint);
|
|
|
|
public override SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint)
|
|
=> _byEndpoint.TryGetValue(grpcEndpoint, out var c) ? c : null;
|
|
}
|
|
|
|
private (IActorRef Actor, SeedStub Seed, PublishSink Sink, MockSiteAlarmStreamClientFactory Factory) CreateActor(
|
|
TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null, bool autoConnect = true,
|
|
TimeSpan? openAttemptTimeout = null)
|
|
{
|
|
var seed = new SeedStub();
|
|
var sink = new PublishSink();
|
|
var factory = new MockSiteAlarmStreamClientFactory { AutoConnect = autoConnect };
|
|
|
|
var props = Props.Create(() => new SiteAlarmAggregatorActor(
|
|
SiteId, "corr-1", seed.Seed, sink.Publish, factory, GrpcNodeA, GrpcNodeB,
|
|
reconcileInterval ?? TimeSpan.FromMinutes(10),
|
|
publishCoalesce ?? TimeSpan.Zero,
|
|
TestReconnectDelay,
|
|
TestStabilityWindow,
|
|
0.0, // no reconcile jitter in tests — determinism
|
|
// Production default (30 s) unless a test overrides it: every other test in this
|
|
// class finishes far inside it, so the open-attempt deadline stays dormant and
|
|
// their behaviour is unchanged.
|
|
openAttemptTimeout ?? SiteAlarmAggregatorActor.DefaultOpenAttemptTimeout));
|
|
|
|
var actor = Sys.ActorOf(props);
|
|
return (actor, seed, sink, factory);
|
|
}
|
|
|
|
private static AlarmStateChanged Alarm(
|
|
string alarmName, string sourceRef, int priority, DateTimeOffset ts,
|
|
bool placeholder = false, string instance = Instance) =>
|
|
new(instance, alarmName, AlarmState.Active, priority, ts)
|
|
{
|
|
SourceReference = sourceRef,
|
|
IsConfiguredPlaceholder = placeholder,
|
|
Kind = placeholder ? AlarmKind.NativeOpcUa : AlarmKind.Computed
|
|
};
|
|
|
|
private static AlarmStateChanged? Find(IReadOnlyList<AlarmStateChanged>? rows, string alarmName, string sourceRef) =>
|
|
rows?.FirstOrDefault(r => r.AlarmName == alarmName && r.SourceReference == sourceRef);
|
|
|
|
/// <summary>
|
|
/// Row-40 house pattern for an ABSENCE assertion: proves <paramref name="checks"/> keep
|
|
/// holding throughout <paramref name="window"/> by polling, rather than a single blind
|
|
/// Thread.Sleep(window) followed by one end-of-window sample. A bare sleep-then-assert
|
|
/// cannot distinguish "the mechanism correctly stayed quiet" from "the mechanism just
|
|
/// hadn't run yet when the one sample was taken", and it also can't catch a value that
|
|
/// flips wrong and then flips back before that single sample. This fails immediately on
|
|
/// the first tick any check goes false, and only returns once the full window has
|
|
/// elapsed with every check still true.
|
|
/// </summary>
|
|
private static void AssertHoldsFor(TimeSpan window, params (Func<bool> Holds, string Because)[] checks)
|
|
{
|
|
var deadline = DateTime.UtcNow + window;
|
|
while (true)
|
|
{
|
|
foreach (var (holds, because) in checks)
|
|
Assert.True(holds(), because);
|
|
if (DateTime.UtcNow >= deadline) return;
|
|
Thread.Sleep(20);
|
|
}
|
|
}
|
|
|
|
// ── Tests ───────────────────────────────────────────────────────────────────
|
|
|
|
[Fact]
|
|
public void Delta_Arriving_Before_Seed_Completes_Is_Buffered_And_Applied_Exactly_Once()
|
|
{
|
|
var (actor, seed, sink, _) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
// Live delta arrives DURING the seed window (new key, not in the seed).
|
|
var t1 = DateTimeOffset.UtcNow;
|
|
actor.Tell(Alarm("Overheat", "", 900, t1));
|
|
|
|
// Still buffering — nothing published yet.
|
|
Assert.Equal(0, sink.Count);
|
|
|
|
// Seed completes with a DIFFERENT alarm.
|
|
seed.CompleteNext(Alarm("PumpFault", "", 500, t1.AddSeconds(-1)));
|
|
|
|
AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3));
|
|
var latest = sink.Latest!;
|
|
Assert.NotNull(Find(latest, "PumpFault", "")); // seeded
|
|
var overheat = Find(latest, "Overheat", ""); // buffered gap-window delta survived
|
|
Assert.NotNull(overheat);
|
|
Assert.Single(latest, r => r.AlarmName == "Overheat"); // exactly once
|
|
}
|
|
|
|
[Fact]
|
|
public void Buffered_Delta_Older_Than_Seed_Entry_Is_Dropped()
|
|
{
|
|
var (actor, seed, sink, _) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
var tSeed = DateTimeOffset.UtcNow;
|
|
// Buffered delta for the SAME key but OLDER than the seed entry.
|
|
actor.Tell(Alarm("PumpFault", "", 100, tSeed.AddSeconds(-1)));
|
|
seed.CompleteNext(Alarm("PumpFault", "", 500, tSeed));
|
|
|
|
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
|
// Prove no late (dropped) delta sneaks in over the window, not just at one sample.
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(150),
|
|
(() => Find(sink.Latest, "PumpFault", "") is not null, "seed row must still be present"),
|
|
(() => Find(sink.Latest, "PumpFault", "")?.Priority == 500,
|
|
"seed value kept; older buffered delta must stay dropped"));
|
|
var latest = sink.Latest!;
|
|
var pumpFault = Find(latest, "PumpFault", "");
|
|
Assert.NotNull(pumpFault);
|
|
Assert.Equal(500, pumpFault!.Priority); // seed value kept; older buffered delta dropped
|
|
}
|
|
|
|
[Fact]
|
|
public void Buffered_Delta_Newer_Than_Seed_Entry_Is_Applied()
|
|
{
|
|
var (actor, seed, sink, _) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
var tSeed = DateTimeOffset.UtcNow;
|
|
actor.Tell(Alarm("PumpFault", "", 900, tSeed.AddSeconds(1))); // newer than seed
|
|
seed.CompleteNext(Alarm("PumpFault", "", 500, tSeed));
|
|
|
|
AwaitCondition(() => sink.Latest is { } l && Find(l, "PumpFault", "")?.Priority == 900,
|
|
TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void Dedup_By_AlarmKey_Distinguishes_SourceReference()
|
|
{
|
|
var (actor, seed, sink, _) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(); // empty seed
|
|
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
|
|
|
var t = DateTimeOffset.UtcNow;
|
|
// Same instance + alarm name, different source reference → two distinct rows.
|
|
actor.Tell(Alarm("LevelAlarm", "Tank01.Level.HiHi", 700, t));
|
|
actor.Tell(Alarm("LevelAlarm", "Tank01.Level.LoLo", 300, t));
|
|
|
|
AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3));
|
|
var latest = sink.Latest!;
|
|
Assert.NotNull(Find(latest, "LevelAlarm", "Tank01.Level.HiHi"));
|
|
Assert.NotNull(Find(latest, "LevelAlarm", "Tank01.Level.LoLo"));
|
|
}
|
|
|
|
[Fact]
|
|
public void Same_Key_Live_Update_Replaces_In_Place()
|
|
{
|
|
var (actor, seed, sink, _) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
|
|
|
var t = DateTimeOffset.UtcNow;
|
|
actor.Tell(Alarm("PumpFault", "", 500, t));
|
|
AwaitCondition(() => Find(sink.Latest, "PumpFault", "")?.Priority == 500, TimeSpan.FromSeconds(3));
|
|
|
|
// Newer update for the same key replaces (not duplicates).
|
|
actor.Tell(Alarm("PumpFault", "", 800, t.AddSeconds(1)));
|
|
AwaitCondition(() => Find(sink.Latest, "PumpFault", "")?.Priority == 800, TimeSpan.FromSeconds(3));
|
|
Assert.Single(sink.Latest!); // still exactly one row for the key
|
|
}
|
|
|
|
[Fact]
|
|
public void Seeded_Native_Alarm_And_Its_Live_Delta_Collapse_Onto_One_Key()
|
|
{
|
|
// Identity parity across the two paths (snapshot seed vs live stream): a native
|
|
// alarm seeded with a non-empty SourceReference and a later live delta for the
|
|
// SAME (instance, alarmName, sourceRef) must collapse to ONE in-place-updated row,
|
|
// not a ghost duplicate. Both paths key on SourceReference, so they must agree.
|
|
var (actor, seed, sink, _) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
var t = DateTimeOffset.UtcNow;
|
|
seed.CompleteNext(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 300, t));
|
|
AwaitCondition(() => Find(sink.Latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi") is not null,
|
|
TimeSpan.FromSeconds(3));
|
|
|
|
// Live delta for the exact same identity, newer → updates in place.
|
|
actor.Tell(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 800, t.AddSeconds(1)));
|
|
|
|
AwaitCondition(
|
|
() => Find(sink.Latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi")?.Priority == 800,
|
|
TimeSpan.FromSeconds(3));
|
|
Assert.Single(sink.Latest!); // exactly one row — no seed/live ghost duplicate
|
|
}
|
|
|
|
[Fact]
|
|
public void Placeholder_Seeded_Coexists_With_Live_Real_Alarm_And_Is_Not_Wiped()
|
|
{
|
|
var (actor, seed, sink, _) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
var t = DateTimeOffset.UtcNow;
|
|
// Seed carries a configured-placeholder row (binding with no active conditions).
|
|
var placeholder = Alarm("Motor1.MotorAlarms", "", 0, t, placeholder: true);
|
|
seed.CompleteNext(placeholder);
|
|
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
|
|
|
// A live REAL condition for the same instance arrives (distinct AlarmKey).
|
|
actor.Tell(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 800, t.AddSeconds(1)));
|
|
|
|
AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3));
|
|
var latest = sink.Latest!;
|
|
// Placeholder still present (not wiped by the real alarm under a different key).
|
|
var ph = Find(latest, "Motor1.MotorAlarms", "");
|
|
Assert.NotNull(ph);
|
|
Assert.True(ph!.IsConfiguredPlaceholder);
|
|
Assert.NotNull(Find(latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi"));
|
|
}
|
|
|
|
[Fact]
|
|
public void GrpcError_Flips_Node_Reconnects_And_ReSeeds()
|
|
{
|
|
var (_, seed, sink, factory) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(); // initial seed done
|
|
// Wait for the seed to actually be APPLIED (published) so the re-seed on error is
|
|
// not skipped as "fan-out already in flight".
|
|
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
|
|
|
// Fail the NodeA stream.
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("NodeA down"));
|
|
|
|
// Reconnect must reach NodeB, AND a re-seed fan-out must have been triggered.
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5));
|
|
// The failed NodeA stream was unsubscribed (relay released, not left zombie).
|
|
Assert.Contains("corr-1", factory.ClientFor(GrpcNodeA).Unsubscribed);
|
|
}
|
|
|
|
[Fact]
|
|
public void Periodic_Reconcile_Authoritatively_Replaces_Stale_Rows()
|
|
{
|
|
var (_, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(250));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
var t = DateTimeOffset.UtcNow;
|
|
seed.CompleteNext(Alarm("PumpFault", "", 500, t)); // seed has a row
|
|
AwaitCondition(() => Find(sink.Latest, "PumpFault", "") is not null, TimeSpan.FromSeconds(3));
|
|
|
|
// Reconcile tick fires → seed #2 (e.g. instance was disabled) returns EMPTY.
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(); // empty
|
|
|
|
// The stale PumpFault row must disappear (authoritative replace, not merge).
|
|
AwaitCondition(() => sink.Latest is { } l && l.Count == 0, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void Failed_Initial_Seed_Does_Not_Publish_But_A_Later_Reconcile_Recovers()
|
|
{
|
|
var (_, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(250));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
// The initial seed fan-out throws as a whole.
|
|
seed.FaultNext();
|
|
|
|
// A failed INITIAL seed must not publish (page keeps its poll fallback — not live yet).
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(200),
|
|
(() => sink.Count == 0, "a failed initial seed must not publish"));
|
|
|
|
// The periodic reconcile is the backstop: it re-runs the fan-out (flag was cleared
|
|
// on failure), completes, and the actor publishes → now live.
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(Alarm("PumpFault", "", 500, DateTimeOffset.UtcNow));
|
|
AwaitCondition(() => Find(sink.Latest, "PumpFault", "") is not null, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void Stream_GivenUp_After_MaxRetries_Reopens_On_Reconcile_Tick()
|
|
{
|
|
// After the retry budget is exhausted the live stream is left down, but the
|
|
// aggregator is NOT stopped — the next reconcile tick self-heals it (resets the
|
|
// budget + reopens) so a sustained site outage never permanently kills the feed.
|
|
var (_, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(300));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
|
|
// Drive initial + 3 retries (each flips node) to exhaust the budget.
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("1"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeB).Subs[0].OnError(new Exception("2"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
|
|
int TotalSubs() => factory.ClientFor(GrpcNodeA).Subs.Count + factory.ClientFor(GrpcNodeB).Subs.Count;
|
|
var before = TotalSubs(); // 4
|
|
|
|
// Fourth error exceeds MaxRetries → stream given up (no immediate reopen).
|
|
factory.ClientFor(GrpcNodeB).Subs[1].OnError(new Exception("4"));
|
|
|
|
// The reconcile tick reopens the stream (self-heal).
|
|
AwaitCondition(() => TotalSubs() > before, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
// ── R2 T11: queued re-seed after reconnect + stream-generation stamp (N7.1/N7.2) ──
|
|
|
|
[Fact]
|
|
public void ReseedRequestedDuringInFlightFanout_IsQueued_NotDropped()
|
|
{
|
|
var (actor, seed, sink, factory) = CreateActor(publishCoalesce: TimeSpan.Zero);
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(); // initial seed done (CallCount 1)
|
|
AwaitAssert(() => Assert.Equal(1, sink.Count));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
|
|
// The first tick after a fan-out is deliberately skipped (WP2.3: the initial seed
|
|
// already covered this window); the second one actually fans out.
|
|
actor.Tell(new RunReconcile()); // skipped — consumes the "already fanned out" flag
|
|
actor.Tell(new RunReconcile()); // reconcile fan-out now in flight (CallCount 2)
|
|
AwaitAssert(() => Assert.Equal(2, seed.CallCount));
|
|
|
|
// Stream error while the reconcile is in flight: the reconnect that follows
|
|
// reconnects (the mock accepts immediately) and its connect-driven re-seed must not
|
|
// be silently swallowed. Pre-fix: StartFanout no-ops and CallCount stays 2 until the
|
|
// next 60s reconcile tick.
|
|
factory.ClientFor(GrpcNodeA).Subs.Last().OnError(new Exception("stream fault"));
|
|
|
|
seed.CompleteNext(); // finish the in-flight reconcile
|
|
AwaitAssert(() => Assert.Equal(3, seed.CallCount)); // queued re-seed ran immediately after
|
|
}
|
|
|
|
[Fact]
|
|
public void LateErrorFromAPreviousStreamGeneration_IsIgnored()
|
|
{
|
|
var (_, seed, _, factory) = CreateActor(publishCoalesce: TimeSpan.Zero);
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
|
|
int TotalSubs() => factory.ClientFor(GrpcNodeA).Subs.Count + factory.ClientFor(GrpcNodeB).Subs.Count;
|
|
|
|
var firstSub = factory.ClientFor(GrpcNodeA).Subs.Single();
|
|
firstSub.OnError(new Exception("real fault")); // gen 1 error → flip, reconnect
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5)); // gen 2 stream open
|
|
AwaitAssert(() => Assert.Equal(2, TotalSubs()));
|
|
|
|
firstSub.OnError(new Exception("late zombie fault")); // stale gen-1 error races in
|
|
// Pre-fix this burns retry budget and opens a THIRD stream; post-fix it is ignored.
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(400),
|
|
(() => TotalSubs() == 2, "a stale gen-1 error must not open a third stream"));
|
|
}
|
|
|
|
// ── WP1.1: graceful (status OK) stream completion is a reconnect trigger ──
|
|
|
|
[Fact]
|
|
public void GracefulStreamCompletion_ReopensOnReconcileTick_OnTheSameNode()
|
|
{
|
|
// The site ends every stream with OK at its 4h max lifetime. Pre-fix the client's
|
|
// read loop just finished, nothing was told to the actor, and the site's alarm feed
|
|
// stayed silently dead until the central node restarted.
|
|
var (_, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(300));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnCompleted();
|
|
|
|
// Reopened by the reconcile tick, on the SAME node — a clean close is not a fault,
|
|
// so there is nothing to fail over from.
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(3));
|
|
Assert.Empty(factory.ClientFor(GrpcNodeB).Subs);
|
|
// The finished stream was released, so the site keeps no zombie relay actor.
|
|
Assert.Contains("corr-1", factory.ClientFor(GrpcNodeA).Unsubscribed);
|
|
}
|
|
|
|
[Fact]
|
|
public void GracefulStreamCompletion_NeitherSpendsNorRefunds_TheErrorRetryBudget()
|
|
{
|
|
// Reconcile is far away; reopens are driven explicitly so the budget can be observed.
|
|
var (actor, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
|
|
// Spend the whole error budget (MaxRetries = 3), each error flipping the node.
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("1"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeB).Subs[0].OnError(new Exception("2"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
|
|
// A graceful completion reopens without flipping the node (budget not spent) …
|
|
factory.ClientFor(GrpcNodeB).Subs[1].OnCompleted();
|
|
actor.Tell(new RunReconcile());
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 3, TimeSpan.FromSeconds(5));
|
|
Assert.Equal(2, factory.ClientFor(GrpcNodeA).Subs.Count);
|
|
|
|
int TotalSubs() => factory.ClientFor(GrpcNodeA).Subs.Count + factory.ClientFor(GrpcNodeB).Subs.Count;
|
|
var before = TotalSubs();
|
|
|
|
// … and without refunding it either: the next error is the 4th, so it exceeds
|
|
// MaxRetries and the stream is given up rather than reconnected.
|
|
factory.ClientFor(GrpcNodeB).Subs[2].OnError(new Exception("4"));
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(400),
|
|
(() => TotalSubs() == before, "budget exhausted — the 4th error must not reconnect"));
|
|
}
|
|
|
|
[Fact]
|
|
public void LateCompletionFromAPreviousStreamGeneration_IsIgnored()
|
|
{
|
|
var (_, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
|
|
var firstSub = factory.ClientFor(GrpcNodeA).Subs.Single();
|
|
firstSub.OnError(new Exception("real fault")); // gen 1 dies → gen 2 opens on NodeB
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
|
|
|
// A completion racing out of the dead gen-1 stream must not tear down the live
|
|
// gen-2 stream (which would go deltaless until the next reconcile tick).
|
|
firstSub.OnCompleted();
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
|
|
(() => !factory.ClientFor(GrpcNodeB).Unsubscribed.Contains("corr-1"),
|
|
"a late gen-1 completion must not unsubscribe the live gen-2 stream"),
|
|
(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1,
|
|
"the live gen-2 stream must not be torn down"));
|
|
}
|
|
|
|
// ── R2 T10: live-delta publish coalescing (N6) ──
|
|
|
|
[Fact]
|
|
public void DeltaBurst_IsCoalesced_IntoFewPublishes_ContainingEveryRow()
|
|
{
|
|
var (_, seed, sink, factory) = CreateActor(publishCoalesce: TimeSpan.FromMilliseconds(150));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(); // empty initial seed
|
|
AwaitAssert(() => Assert.Equal(1, sink.Count)); // seed publish, immediate
|
|
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
var sub = factory.ClientFor(GrpcNodeA).Subs.Single();
|
|
var now = DateTimeOffset.UtcNow;
|
|
for (var i = 0; i < 50; i++)
|
|
sub.OnAlarm(Alarm($"A{i}", "", 900, now)); // 50 distinct keys, one burst
|
|
|
|
AwaitAssert(() =>
|
|
{
|
|
Assert.Equal(50, sink.Latest!.Count); // nothing lost
|
|
Assert.InRange(sink.Count, 2, 4); // pre-fix: 51 publishes (1 seed + 1 per delta)
|
|
}, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void ZeroCoalesce_PreservesLegacyPerDeltaPublish()
|
|
{
|
|
var (_, seed, sink, factory) = CreateActor(publishCoalesce: TimeSpan.Zero);
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
AwaitAssert(() => Assert.Equal(1, sink.Count));
|
|
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
var sub = factory.ClientFor(GrpcNodeA).Subs.Single();
|
|
var now = DateTimeOffset.UtcNow;
|
|
sub.OnAlarm(Alarm("A1", "", 900, now));
|
|
sub.OnAlarm(Alarm("A2", "", 900, now));
|
|
AwaitAssert(() => Assert.Equal(3, sink.Count)); // 1 seed + 1 per delta
|
|
}
|
|
|
|
[Fact]
|
|
public void Stop_Message_TearsDown_Grpc_And_Stops_Actor()
|
|
{
|
|
var (actor, seed, _, factory) = CreateActor();
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
|
|
Watch(actor);
|
|
actor.Tell(new StopSiteAlarmAggregator());
|
|
|
|
ExpectTerminated(actor, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Unsubscribed.Contains("corr-1"),
|
|
TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
// -- WP2.3: seed once per SUCCESSFUL (re)connect, and stream-truthful liveness --
|
|
|
|
[Fact]
|
|
public void ReconnectAttempts_DoNotEachReSeed_OnlyTheSuccessfulConnectDoes()
|
|
{
|
|
// Finding #10's per-attempt re-fan-out: every reconnect ATTEMPT used to kick a
|
|
// whole-site snapshot fan-out - against a site that is, by definition of the
|
|
// reconnect, unreachable. Now the re-seed is owed to the connect that succeeds.
|
|
var (_, seed, _, factory) = CreateActor(
|
|
reconcileInterval: TimeSpan.FromMinutes(10), autoConnect: false);
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); // initial seed
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
|
|
// Three failed attempts, each flipping the node - none of them may re-seed.
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("1"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeB).Subs[0].OnError(new Exception("2"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
|
|
(() => seed.CallCount == 1, "failed reconnect attempts must not re-seed (pre-fix: 4)"));
|
|
|
|
// The stream finally comes up - exactly ONE re-seed, on the connect.
|
|
factory.ClientFor(GrpcNodeB).Subs[1].OnConnected!();
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5));
|
|
seed.CompleteNext();
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
|
|
(() => seed.CallCount == 2, "the successful connect must re-seed exactly once"));
|
|
}
|
|
|
|
[Fact]
|
|
public void ReconcileTick_IsSkipped_WhenAFanoutAlreadyRanInTheWindow()
|
|
{
|
|
// The reconcile is a BACKSTOP, not an unconditional 60s whole-site snapshot: a
|
|
// window already covered by a seed costs nothing. Staleness stays bounded because
|
|
// the skip consumes the flag, so the next tick always fans out.
|
|
var (actor, seed, _, _) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
Thread.Sleep(200);
|
|
|
|
actor.Tell(new RunReconcile()); // skipped - the initial seed covered this window
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
|
|
(() => seed.CallCount == 1, "a tick inside the just-seeded window must be skipped"));
|
|
|
|
actor.Tell(new RunReconcile()); // and the next one runs
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void UnchangedReconcileSnapshot_DoesNotRepublish()
|
|
{
|
|
// A reconcile that finds nothing changed used to wake every viewer's render path
|
|
// once a minute regardless. It now publishes as a diff.
|
|
var (actor, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
var t = DateTimeOffset.UtcNow;
|
|
seed.CompleteNext(Alarm("PumpFault", "", 500, t));
|
|
AwaitAssert(() => Assert.Equal(1, sink.Count));
|
|
|
|
actor.Tell(new RunReconcile()); // skipped (seed covered the window)
|
|
actor.Tell(new RunReconcile()); // real fan-out
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(Alarm("PumpFault", "", 500, t)); // identical snapshot
|
|
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
|
|
(() => sink.Count == 1, "an unchanged reconcile snapshot must not republish"));
|
|
|
|
// A genuinely changed snapshot still publishes.
|
|
actor.Tell(new RunReconcile());
|
|
actor.Tell(new RunReconcile());
|
|
AwaitCondition(() => seed.CallCount == 3, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(Alarm("PumpFault", "", 900, t.AddSeconds(1)));
|
|
AwaitAssert(() => Assert.Equal(2, sink.Count));
|
|
}
|
|
|
|
[Fact]
|
|
public void StreamCompletion_DropsLiveness_AndTheReopenRestoresIt()
|
|
{
|
|
// WP2.3 carried residual: a completed (or given-up) stream kept reporting live
|
|
// until the next reconcile publish, so the page grafted a freezing snapshot over
|
|
// fresh poll data.
|
|
// Reconcile is held far away so the drop can be observed before the reopen
|
|
// restores liveness; the reopen is then driven explicitly.
|
|
var (actor, seed, sink, factory) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
AwaitCondition(() => sink.Count >= 1 && sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
|
|
// The site ends the stream cleanly (its 4h max lifetime).
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnCompleted();
|
|
|
|
// Liveness drops IMMEDIATELY - not at the next reconcile.
|
|
AwaitCondition(() => !sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
|
|
// The reconcile tick reopens; the connect restores liveness.
|
|
actor.Tell(new RunReconcile());
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(5));
|
|
}
|
|
|
|
[Fact]
|
|
public void StreamFault_DropsLiveness_Immediately()
|
|
{
|
|
var (_, seed, sink, factory) = CreateActor(
|
|
reconcileInterval: TimeSpan.FromMinutes(10), autoConnect: false);
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnConnected!();
|
|
seed.CompleteNext();
|
|
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("site gone"));
|
|
AwaitCondition(() => !sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void ConsecutiveReconcileTicks_WithNoReconnect_EachRunAFanout()
|
|
{
|
|
// The bug: a tick's OWN fan-out completion armed the skip flag, so steady state ran
|
|
// fan-out → skip → fan-out → skip — one reconcile per TWO intervals, halving both the
|
|
// not-reporting refresh and the alarm reconcile backstop. Nothing was being
|
|
// duplicated: only a connect/failover-driven seed can make a tick redundant.
|
|
var (actor, seed, _, _) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext(); // initial seed — this one DOES stand a tick down
|
|
Thread.Sleep(150);
|
|
|
|
actor.Tell(new RunReconcile()); // consumed by the initial seed's flag
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(200),
|
|
(() => seed.CallCount == 1, "the tick right after the initial seed must be skipped"));
|
|
|
|
// From here on, with no reconnect in between, EVERY tick must fan out.
|
|
for (var expected = 2; expected <= 5; expected++)
|
|
{
|
|
actor.Tell(new RunReconcile());
|
|
AwaitCondition(() => seed.CallCount == expected, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
Thread.Sleep(100);
|
|
}
|
|
|
|
Assert.Equal(5, seed.CallCount);
|
|
}
|
|
|
|
[Fact]
|
|
public void ConnectDrivenReseed_StillStandsTheNextTickDown()
|
|
{
|
|
// The other half of the fix: the skip exists for the connect/failover seed, and that
|
|
// suppression must survive. A reconnect's re-seed still makes the very next tick a
|
|
// no-op, so a flapping stream cannot double the whole-site snapshot rate.
|
|
var (actor, seed, _, factory) = CreateActor(
|
|
reconcileInterval: TimeSpan.FromMinutes(10), autoConnect: false);
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnConnected!();
|
|
seed.CompleteNext();
|
|
Thread.Sleep(150);
|
|
|
|
actor.Tell(new RunReconcile()); // consumed by the initial seed
|
|
actor.Tell(new RunReconcile()); // real tick fan-out
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
Thread.Sleep(150);
|
|
|
|
// Stream faults → reconnect on node B → connect-driven re-seed (call 3).
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("site gone"));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeB).Subs[0].OnConnected!();
|
|
AwaitCondition(() => seed.CallCount == 3, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
Thread.Sleep(150);
|
|
|
|
// That re-seed covers this window: the next tick is skipped…
|
|
actor.Tell(new RunReconcile());
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(250),
|
|
(() => seed.CallCount == 3, "the connect-driven re-seed must stand the next tick down"));
|
|
|
|
// …and the one after it fans out again.
|
|
actor.Tell(new RunReconcile());
|
|
AwaitCondition(() => seed.CallCount == 4, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void ForeignCancelledStreamError_DropsLiveness_AndReopensOnTheOtherNode()
|
|
{
|
|
// End-to-end for the SiteStreamGrpcClient fix: a peer-originated
|
|
// RpcException(Cancelled) now reaches onError instead of being swallowed. Before the
|
|
// fix NONE of onError/onCompleted/onConnected fired, so the aggregator kept
|
|
// _streamDown=false — IsLive stuck true and the reconcile tick's reopen guard, which
|
|
// only fires when the stream is known down, never ran.
|
|
var (_, seed, sink, factory) = CreateActor(
|
|
reconcileInterval: TimeSpan.FromMinutes(10), autoConnect: false);
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnConnected!();
|
|
seed.CompleteNext();
|
|
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
|
|
factory.ClientFor(GrpcNodeA).Subs[0].OnError(
|
|
new global::Grpc.Core.RpcException(new global::Grpc.Core.Status(
|
|
global::Grpc.Core.StatusCode.Cancelled, "cancelled by peer")));
|
|
|
|
AwaitCondition(() => !sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
// …and the stream is reopened on the other node, where a connect earns one re-seed.
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
|
factory.ClientFor(GrpcNodeB).Subs[0].OnConnected!();
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
// ── P3-4: the stream open attempt is deadline-bounded ───────────────────────
|
|
|
|
/// <summary>
|
|
/// Proof of the P3-4 starvation and of its fix.
|
|
/// <para>
|
|
/// An open attempt is cleared by exactly three peer-produced outcomes: connected, error,
|
|
/// or graceful completion. A stream that produces NONE of them pins <c>_openInFlight</c>.
|
|
/// That is reachable: <c>SiteStreamGrpcClient</c> gives up on response headers after
|
|
/// <c>ConnectedHeaderTimeout</c> (10 s) and then treats the FIRST RECEIVED EVENT as its
|
|
/// fallback connected signal — which an alarm-only site that happens to be quiet never
|
|
/// sends (proven independently by
|
|
/// <c>SiteStreamGrpcClientTests.ConsumeStream_HeaderTimeoutOnADeadSite_NeverReportsConnected</c>).
|
|
/// The aggregator then sits with <c>_streamDown == true</c> AND <c>_openInFlight == true</c>,
|
|
/// and <see cref="SiteAlarmAggregatorActor.OnReconcileTick"/>'s reopen guard
|
|
/// (<c>_streamDown && !_openInFlight</c>) is skipped every tick until the site's
|
|
/// 4 h max stream lifetime finally ends the call.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>Honesty note.</b> The mock reproduces the aggregator-side state machine exactly and
|
|
/// deterministically (that is what <c>autoConnect: false</c> models). What it does NOT
|
|
/// prove is how often the >10 s header delay that gets you there actually happens in
|
|
/// production: the site flushes response headers explicitly as soon as its relay is
|
|
/// attached (<c>SiteStreamGrpcServer.RunSubscriptionStreamAsync</c>), so the 10 s budget
|
|
/// has to be consumed by channel connect + HTTP/2 handshake + server scheduling instead —
|
|
/// plausible under a WAN stall or a GC/thread-pool-starved site node, but not something
|
|
/// this suite can demonstrate. The deadline is therefore a self-healing guard rather than
|
|
/// a fix for a measured production incident.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen()
|
|
{
|
|
// 250 ms deadline; the reconcile timer is parked 10 minutes away so the ONLY tick is
|
|
// the one this test issues by hand.
|
|
var (actor, seed, sink, factory) = CreateActor(
|
|
reconcileInterval: TimeSpan.FromMinutes(10),
|
|
autoConnect: false,
|
|
openAttemptTimeout: TimeSpan.FromMilliseconds(250));
|
|
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
|
|
// The stream never reports connected, never faults, never completes — the quiet-site
|
|
// header-timeout shape. Liveness is (correctly) false the whole time.
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
|
|
(() => !sink.StreamLive, "a stream that never connected must never read as live"));
|
|
|
|
// Pre-fix this tick found _openInFlight still true and skipped the reopen, leaving
|
|
// node A on one subscription forever. Post-fix the deadline has released the flag.
|
|
actor.Tell(new RunReconcile());
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
|
|
|
// And the reopen is a real one: its connect brings the feed up and earns a re-seed.
|
|
factory.ClientFor(GrpcNodeA).Subs[1].OnConnected!();
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
}
|
|
|
|
[Fact]
|
|
public void OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget()
|
|
{
|
|
// The deadline must be inert on every healthy path: a connect cancels it, so no
|
|
// spurious reopen, no node flip, and no retry-budget spend can be attributed to it.
|
|
// Timing-independent in the sense that matters — the assertions below would fail for
|
|
// ANY firing of the timer, and the window is 8x the deadline.
|
|
var (actor, seed, sink, factory) = CreateActor(
|
|
reconcileInterval: TimeSpan.FromMinutes(10),
|
|
autoConnect: true,
|
|
openAttemptTimeout: TimeSpan.FromMilliseconds(100));
|
|
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
|
seed.CompleteNext();
|
|
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
|
|
|
|
AssertHoldsFor(TimeSpan.FromMilliseconds(800),
|
|
(() => sink.StreamLive, "a connected stream must stay live across the open deadline"),
|
|
(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, "no second open on node A"),
|
|
(() => factory.ClientFor(GrpcNodeB).Subs.Count == 0, "no node flip"),
|
|
(() => seed.CallCount == 1, "no extra fan-out"));
|
|
|
|
// A reconcile tick on a live stream still must not reopen. (First tick is consumed by
|
|
// the initial seed's skip flag; the second is the real fan-out.)
|
|
actor.Tell(new RunReconcile());
|
|
actor.Tell(new RunReconcile());
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3)); // tick fan-out only
|
|
seed.CompleteNext();
|
|
Assert.Single(factory.ClientFor(GrpcNodeA).Subs);
|
|
}
|
|
|
|
[Fact]
|
|
public void FailedSeed_IsRetried_WithBackoff_BeforeTheNextReconcile()
|
|
{
|
|
// The seed leg had no backoff: a failing fan-out simply waited a full reconcile
|
|
// interval. Now it retries on its own timer (reconnectDelay, doubling).
|
|
var (_, seed, _, _) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10));
|
|
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
|
|
|
seed.FaultNext();
|
|
// TestReconnectDelay is 50 ms, so the first retry lands far inside the 10-minute
|
|
// reconcile interval - pre-fix nothing would run until that interval elapsed.
|
|
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5));
|
|
seed.FaultNext();
|
|
AwaitCondition(() => seed.CallCount == 3, TimeSpan.FromSeconds(5));
|
|
}
|
|
}
|