merge(r2): PLAN-R2-02 Communication & S&F (Critical N1)
This commit is contained in:
@@ -75,8 +75,11 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
|
||||
- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded and published at least once.
|
||||
- **Seed-then-stream** (copied from `DebugStreamBridgeActor` ordering): open the `SubscribeSite` stream first (buffer live deltas), run the snapshot fan-out once via the existing `DebugViewSnapshot` path (bounded by `LiveAlarmCacheSeedConcurrency`), flush the buffer with **dedup by `(InstanceUniqueName, AlarmName, SourceReference)`**, then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream.
|
||||
- **Failover & drift**: NodeA↔NodeB reconnect (client via `SiteStreamGrpcClient.SubscribeSiteAsync`, factory endpoint failover) re-seeds rather than serving stale; a periodic **reconcile snapshot** (default 60s) corrects any missed enable/disable so the cache can't drift indefinitely. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
|
||||
- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200).
|
||||
- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200), `LiveAlarmCachePublishCoalesce` (default 250ms; `0` = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6).
|
||||
- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link).
|
||||
- **Accepted limitations (arch review 02 round 2, N8):**
|
||||
- **Standby-node aggregators**: the live cache is per-node and `SetActorSystem` is wired on every central node, so browsing the standby node directly (diagnostic ports, e.g. 9002) starts a second, fully-functional read-only aggregator + `SubscribeSite` stream there. Accepted — not gated: the aggregator is read-only, bounded (one stream/site, viewer-capped), and torn down by the viewer linger; gating `SetActorSystem` behind the active check would break the diagnostic-browse path and buy nothing. The `[PERM]` "lives only on the active central node" claim is corrected to "per-node; in routine operation only the active node hosts viewers".
|
||||
- **Site deleted while an Alarm Summary viewer is open**: the viewer's aggregator reconciles to an empty snapshot (the deleted site's instances vanish from the fan-out) until its viewers leave, then the linger stop reaps it; the site's cached gRPC channels are disposed at deletion via `SiteStreamGrpcClientFactory.RemoveSiteAsync` (see `ManagementActor.HandleDeleteSite`, arch review 02 round 2, N8).
|
||||
|
||||
#### Site-Side gRPC Streaming Components
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
|
||||
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
|
||||
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
|
||||
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
|
||||
- **Peer-join anti-entropy resync.** Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was **down for an extended period** (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node **(re)tracks its peer**, a **standby** requests a full-buffer snapshot (`RequestSfBufferResync`); the **active** node answers with up to `MaxResyncRows` (10 000) of its oldest rows (`SfBufferSnapshot`), and the standby **replaces its entire local buffer** with that snapshot (`ReplaceAllAsync`, one transaction). Only the active node answers; only a standby applies (each side checks the repo-standard leader+Up active-node predicate, safe-by-default to standby). Because replicated applies are **upserts** (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta. If the buffer exceeds the 10 000-row cap the snapshot is flagged `Truncated` and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers.
|
||||
- **Peer-join anti-entropy resync (chunked, ack-confirmed).** Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was **down for an extended period** (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node **(re)tracks its peer**, a **standby** requests a full-buffer snapshot (`RequestSfBufferResync`); the **active** node loads up to `MaxResyncRows` (10 000) of its oldest rows and answers with a **sequence of byte-budgeted chunks** (`SfBufferSnapshotChunk`), each carrying a shared `ResyncId`, a 1-based `Sequence`, and the `TotalChunks` count. Chunking is mandatory because the monolithic snapshot exceeds Akka remoting's **default 128 000-byte frame** for any realistic backlog and `BuildHocon` sets no override — a single oversized message is silently undeliverable (review 02 round 2, **N2 High**). Rows accumulate into a chunk until the estimated payload budget (`MaxResyncChunkBytes` = 64 000, ≈50% frame headroom) or the row cap (`MaxResyncChunkRows` = 200) is hit; a single row whose payload alone exceeds the budget ships solo with a Warning. The standby **assembles all chunks of one `ResyncId`** (a new `ResyncId` discards any stale partial assembly; a partial that never completes is dropped after `resyncAssemblyTimeout`, default 30 s, and counted a replication failure), then **replaces its entire local buffer** with the assembled snapshot (`ReplaceAllAsync`, one transaction) and returns a delivery confirmation (`SfBufferResyncAck`). The active node arms an ack window (`resyncAckTimeout`, default 60 s): an acknowledged resync increments `scadabridge.store_and_forward.resync.completed`; an unacknowledged one (lost chunks / dead peer) logs a Warning and increments `scadabridge.store_and_forward.resync.ack_missing` — closing N2's silent-loss mode (previously nothing counted a lost snapshot and nothing retried until the next peer-track). Only the active node answers; only a standby applies, and both sides re-check at apply time (a mid-flight active-flip aborts the wipe) — each side checks the repo-standard **oldest-Up member** active-node predicate (singleton-host semantics via the shared `ActiveNodeEvaluator`, the **same predicate as the S&F delivery gate**; review 02 round 2, **N1 Critical** — using cluster *leadership* here let a rolling restart of the lower-address node make the delivering node wipe its own live buffer). Because replicated applies are **upserts** (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta; the one accepted exception is the rare **N5** orphan-row race (a replicated `Remove` ordered before the snapshot chunks can re-add the removed row, re-delivered once and self-corrected at the next resync — inherent to no-ack replication). If the buffer exceeds the 10 000-row cap the snapshot is flagged `Truncated` and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers. The legacy monolithic `SfBufferSnapshot` message + standby handler are **retained** for rolling-upgrade compatibility (an old active node's monolithic snapshot is still applied by a new standby).
|
||||
|
||||
### Operation Tracking Table (lives in Site Runtime, not here)
|
||||
|
||||
|
||||
@@ -47,6 +47,14 @@ public static class ScadaBridgeTelemetry
|
||||
Meter.CreateCounter<long>("scadabridge.store_and_forward.replication.failures", unit: "1",
|
||||
description: "S&F buffer replication operations that failed to dispatch/deliver to the peer node");
|
||||
|
||||
private static readonly Counter<long> _sfResyncCompleted =
|
||||
Meter.CreateCounter<long>("scadabridge.store_and_forward.resync.completed", unit: "1",
|
||||
description: "S&F anti-entropy resyncs the standby acknowledged as applied");
|
||||
|
||||
private static readonly Counter<long> _sfResyncAckMissing =
|
||||
Meter.CreateCounter<long>("scadabridge.store_and_forward.resync.ack_missing", unit: "1",
|
||||
description: "S&F resyncs answered by the active node but never acknowledged by the standby within the ack window (lost chunks / dead peer)");
|
||||
|
||||
/// <summary>
|
||||
/// Incremented each time a per-site live-alarm aggregator re-establishes its site-wide
|
||||
/// gRPC stream — a NodeA↔NodeB failover flip or a reconcile-driven reopen after the
|
||||
@@ -107,6 +115,12 @@ public static class ScadaBridgeTelemetry
|
||||
/// <summary>Records one failed S&F replication dispatch.</summary>
|
||||
public static void RecordReplicationFailure() => _replicationFailures.Add(1);
|
||||
|
||||
/// <summary>Records one acknowledged (applied) S&F buffer resync.</summary>
|
||||
public static void RecordSfResyncCompleted() => _sfResyncCompleted.Add(1);
|
||||
|
||||
/// <summary>Records one resync whose ack window expired.</summary>
|
||||
public static void RecordSfResyncAckMissing() => _sfResyncAckMissing.Add(1);
|
||||
|
||||
/// <summary>Records that a site connection opened (increments the up-count gauge).</summary>
|
||||
public static void SiteConnectionOpened() => Interlocked.Increment(ref _siteConnectionsUp);
|
||||
|
||||
|
||||
@@ -52,21 +52,26 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
private readonly string _grpcNodeAAddress;
|
||||
private readonly string _grpcNodeBAddress;
|
||||
private readonly TimeSpan _reconcileInterval;
|
||||
private readonly TimeSpan _publishCoalesce;
|
||||
|
||||
private const int MaxRetries = 3;
|
||||
private const string ReconnectTimerKey = "alarm-grpc-reconnect";
|
||||
private const string StabilityTimerKey = "alarm-grpc-stability";
|
||||
private const string ReconcileTimerKey = "alarm-reconcile";
|
||||
private const string PublishTimerKey = "alarm-publish-coalesce";
|
||||
|
||||
/// <summary>Delay between gRPC reconnection attempts. Settable for tests.</summary>
|
||||
internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5);
|
||||
/// <summary>True while a coalesced publish is armed (dirty deltas awaiting one tick). Actor-thread only.</summary>
|
||||
private bool _publishPending;
|
||||
|
||||
/// <summary>Delay between gRPC reconnection attempts (ctor-injected; production default 5s).</summary>
|
||||
private readonly TimeSpan _reconnectDelay;
|
||||
|
||||
/// <summary>
|
||||
/// How long a freshly-opened gRPC stream must stay up before its retry budget is
|
||||
/// considered recovered (mirrors <see cref="DebugStreamBridgeActor.StabilityWindow"/>).
|
||||
/// Settable for tests.
|
||||
/// considered recovered (mirrors <see cref="DebugStreamBridgeActor.StabilityWindow"/>);
|
||||
/// ctor-injected (production default 60s).
|
||||
/// </summary>
|
||||
internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60);
|
||||
private readonly TimeSpan _stabilityWindow;
|
||||
|
||||
private int _retryCount;
|
||||
private bool _useNodeA = true;
|
||||
@@ -94,6 +99,21 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
/// <summary>Ordered buffer of live deltas that arrived while a fan-out was in flight. Actor-thread only.</summary>
|
||||
private readonly List<AlarmStateChanged> _buffer = new();
|
||||
|
||||
/// <summary>
|
||||
/// A failover re-seed was requested while a fan-out was already in flight; it must run
|
||||
/// right after the in-flight one completes rather than being silently dropped (N7.1) —
|
||||
/// the in-flight snapshot's read-time predates the stream death. Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _reseedQueued;
|
||||
|
||||
/// <summary>
|
||||
/// Monotonic stream generation stamped on each opened gRPC stream and echoed back on its
|
||||
/// error callback: a late error raced out of a previous (cancelled) stream carries a stale
|
||||
/// generation and is ignored so it never burns retry budget or double-flips (N7.2).
|
||||
/// Actor-thread only.
|
||||
/// </summary>
|
||||
private int _streamGeneration;
|
||||
|
||||
private const int BufferWarnThreshold = 10_000;
|
||||
private bool _bufferWarned;
|
||||
|
||||
@@ -118,6 +138,15 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
/// <param name="grpcNodeAAddress">gRPC address of the site's node A.</param>
|
||||
/// <param name="grpcNodeBAddress">gRPC address of the site's node B.</param>
|
||||
/// <param name="reconcileInterval">Periodic reconcile snapshot cadence.</param>
|
||||
/// <param name="publishCoalesce">
|
||||
/// Publish-coalescing window for live deltas: a positive value batches a delta storm
|
||||
/// into one publish per window (review 02 round 2, N6); <see cref="TimeSpan.Zero"/>
|
||||
/// restores per-delta publishing (legacy). Seed/reconcile publishes stay immediate.
|
||||
/// </param>
|
||||
/// <param name="reconnectDelay">Delay between gRPC reconnection attempts (production 5s).</param>
|
||||
/// <param name="stabilityWindow">
|
||||
/// How long a fresh gRPC stream must stay up before its retry budget recovers (production 60s).
|
||||
/// </param>
|
||||
public SiteAlarmAggregatorActor(
|
||||
string siteIdentifier,
|
||||
string correlationId,
|
||||
@@ -126,7 +155,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
SiteStreamGrpcClientFactory grpcFactory,
|
||||
string grpcNodeAAddress,
|
||||
string grpcNodeBAddress,
|
||||
TimeSpan reconcileInterval)
|
||||
TimeSpan reconcileInterval,
|
||||
TimeSpan publishCoalesce,
|
||||
TimeSpan reconnectDelay,
|
||||
TimeSpan stabilityWindow)
|
||||
{
|
||||
_siteIdentifier = siteIdentifier;
|
||||
_correlationId = correlationId;
|
||||
@@ -136,6 +168,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
_grpcNodeAAddress = grpcNodeAAddress;
|
||||
_grpcNodeBAddress = grpcNodeBAddress;
|
||||
_reconcileInterval = reconcileInterval;
|
||||
_publishCoalesce = publishCoalesce;
|
||||
_reconnectDelay = reconnectDelay;
|
||||
_stabilityWindow = stabilityWindow;
|
||||
|
||||
// Live delta from the site-wide alarm stream (marshalled in via Self.Tell).
|
||||
// A received delta must NOT reset the retry budget (a flapping stream that
|
||||
@@ -152,6 +187,13 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// Periodic reconcile tick (and the re-seed kicked after a reconnect).
|
||||
Receive<RunReconcile>(_ => OnReconcileTick());
|
||||
|
||||
// Coalesced-publish tick: one publish for a batch of dirtying deltas (N6).
|
||||
Receive<PublishCoalesced>(_ =>
|
||||
{
|
||||
_publishPending = false;
|
||||
if (!_stopped) Publish();
|
||||
});
|
||||
|
||||
// Stream stayed up for StabilityWindow — recover the retry budget.
|
||||
Receive<GrpcAlarmStreamStable>(_ =>
|
||||
{
|
||||
@@ -163,6 +205,15 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// gRPC stream error — flip node + reconnect + re-seed.
|
||||
Receive<GrpcAlarmStreamError>(msg =>
|
||||
{
|
||||
// Ignore a late error raced out of a previous (cancelled) stream — the
|
||||
// RpcException(Cancelled) filter at SiteStreamGrpcClient.cs covers the normal
|
||||
// path, but a genuine socket fault can beat the cancel (N7.2).
|
||||
if (msg.Generation != _streamGeneration)
|
||||
{
|
||||
_log.Debug("Ignoring stale gRPC error from stream generation {0} (current {1})",
|
||||
msg.Generation, _streamGeneration);
|
||||
return;
|
||||
}
|
||||
_log.Warning("Site-alarm gRPC stream error for {0}: {1}", _siteIdentifier, msg.Exception.Message);
|
||||
HandleGrpcError();
|
||||
});
|
||||
@@ -247,7 +298,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
if (_stopped) return;
|
||||
if (_fanoutInFlight)
|
||||
{
|
||||
// Already seeding/reconciling — don't stack a second fan-out.
|
||||
// A failover re-seed requested mid-fan-out must run right after the in-flight
|
||||
// one — its snapshot read-time predates the stream death, so skipping it would
|
||||
// serve stale up to the next 60s reconcile (N7.1). An initial-seed collision
|
||||
// never queues (there is only ever one).
|
||||
if (!isInitial) _reseedQueued = true;
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -300,7 +355,18 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
_log.Debug("Site-alarm {0} {1} complete: {2} alarm row(s)",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _cache.Count);
|
||||
|
||||
// The fresh snapshot already carries the buffered deltas; drop any armed coalesce
|
||||
// tick so we publish once, immediately.
|
||||
Timers.Cancel(PublishTimerKey);
|
||||
_publishPending = false;
|
||||
Publish();
|
||||
|
||||
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
|
||||
if (_reseedQueued)
|
||||
{
|
||||
_reseedQueued = false;
|
||||
StartFanout(isInitial: false);
|
||||
}
|
||||
}
|
||||
|
||||
private void OnSeedFailed(SeedFailed msg)
|
||||
@@ -319,9 +385,20 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// Only publish if we already had a seed (so IsLive doesn't flip true on a
|
||||
// failed initial seed — the page keeps its poll fallback until we truly seed).
|
||||
if (_seeded)
|
||||
{
|
||||
Timers.Cancel(PublishTimerKey);
|
||||
_publishPending = false;
|
||||
Publish();
|
||||
}
|
||||
|
||||
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
|
||||
if (_reseedQueued)
|
||||
{
|
||||
_reseedQueued = false;
|
||||
StartFanout(isInitial: false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Flushes the pre-fan-out buffer in arrival order. When <paramref name="dedupAgainstSeed"/>
|
||||
/// is true (normal completion) a buffered delta already reflected in the just-installed
|
||||
@@ -362,9 +439,23 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
return;
|
||||
}
|
||||
|
||||
// Pass-through: apply and publish only if the cache actually changed.
|
||||
// Pass-through: apply and (coalesced) publish only if the cache actually changed.
|
||||
if (ApplyDelta(delta, requireStrictlyNewer: false))
|
||||
Publish();
|
||||
SchedulePublish();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Coalesced publish: with a positive window, the first dirtying delta arms a
|
||||
/// single-shot timer and further deltas ride the same tick — one snapshot copy and
|
||||
/// one viewer fan-out per window instead of per transition (N6). Zero = legacy
|
||||
/// immediate publish. Last write wins, so batching never changes final state.
|
||||
/// </summary>
|
||||
private void SchedulePublish()
|
||||
{
|
||||
if (_publishCoalesce <= TimeSpan.Zero) { Publish(); return; }
|
||||
if (_publishPending) return;
|
||||
_publishPending = true;
|
||||
Timers.StartSingleTimer(PublishTimerKey, new PublishCoalesced(), _publishCoalesce);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -416,8 +507,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = new CancellationTokenSource();
|
||||
|
||||
Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), StabilityWindow);
|
||||
Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), _stabilityWindow);
|
||||
|
||||
var generation = ++_streamGeneration;
|
||||
var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint);
|
||||
var self = Self;
|
||||
var ct = _grpcCts.Token;
|
||||
@@ -427,7 +519,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
await client.SubscribeSiteAsync(
|
||||
_correlationId,
|
||||
alarm => self.Tell(alarm),
|
||||
ex => self.Tell(new GrpcAlarmStreamError(ex)),
|
||||
ex => self.Tell(new GrpcAlarmStreamError(ex, generation)),
|
||||
ct);
|
||||
}, ct);
|
||||
}
|
||||
@@ -474,7 +566,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
if (_retryCount == 1)
|
||||
Self.Tell(new ReconnectAlarmStream());
|
||||
else
|
||||
Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), ReconnectDelay);
|
||||
Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), _reconnectDelay);
|
||||
}
|
||||
|
||||
private void CleanupGrpc()
|
||||
@@ -520,8 +612,12 @@ internal sealed record SeedFailed(Exception Exception, bool IsInitial);
|
||||
/// <summary>Internal: periodic reconcile tick (and the re-seed kicked after a reconnect).</summary>
|
||||
internal sealed record RunReconcile;
|
||||
|
||||
/// <summary>Internal: site-alarm gRPC stream error occurred.</summary>
|
||||
internal sealed record GrpcAlarmStreamError(Exception Exception);
|
||||
/// <summary>Internal: coalesced-publish tick — flush the dirty cache to viewers once (N6).</summary>
|
||||
internal sealed record PublishCoalesced;
|
||||
|
||||
/// <summary>Internal: site-alarm gRPC stream error occurred, stamped with the stream
|
||||
/// generation it came from so a late error from a cancelled stream can be ignored (N7.2).</summary>
|
||||
internal sealed record GrpcAlarmStreamError(Exception Exception, int Generation);
|
||||
|
||||
/// <summary>Internal: reconnect the site-alarm gRPC stream (flip node).</summary>
|
||||
internal sealed record ReconnectAlarmStream;
|
||||
|
||||
@@ -67,10 +67,10 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
|
||||
/// <param name="deploymentManagerProxy">Local reference to the Deployment Manager singleton proxy.</param>
|
||||
/// <param name="isActiveCheck">
|
||||
/// Optional override returning <c>true</c> when this node
|
||||
/// is the active member of the site cluster. <c>null</c> uses the real
|
||||
/// Akka <see cref="Cluster"/> leader check (the default for production
|
||||
/// wiring); tests pass a stub so they do not need to load Akka.Cluster
|
||||
/// into the <c>TestKit</c> ActorSystem.
|
||||
/// is the active member of the site cluster. <c>null</c> uses the shared oldest-Up
|
||||
/// evaluator (production wiring passes the Host's singleton-host delegate); tests
|
||||
/// pass a stub so they do not need to load Akka.Cluster into the <c>TestKit</c>
|
||||
/// ActorSystem.
|
||||
/// </param>
|
||||
public SiteCommunicationActor(
|
||||
string siteId,
|
||||
@@ -506,24 +506,16 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Default active-node check used when no override is
|
||||
/// supplied. Mirrors <c>ActiveNodeGate</c> in the Host (and
|
||||
/// <c>ActiveNodeHealthCheck</c>): the node is the active member of the
|
||||
/// site cluster when it is the current cluster leader AND its own
|
||||
/// <see cref="MemberStatus"/> is <see cref="MemberStatus.Up"/>. Any other
|
||||
/// state (still joining, leaving, no leader yet) reports standby —
|
||||
/// safe-by-default, matching the standby case.
|
||||
/// Default active-node check used when no override is supplied: oldest-Up member
|
||||
/// semantics via the shared <see cref="ClusterState.ActiveNodeEvaluator"/> — the
|
||||
/// same predicate as the S&F delivery gate and the replication resync authority
|
||||
/// (review 02 round 2, N1). Unscoped by role: a site cluster's members all carry
|
||||
/// the site role, so role scoping is a no-op here; production wiring passes the
|
||||
/// Host's role-scoped IClusterNodeProvider delegate anyway. Any other state
|
||||
/// (still joining, leaving) reports standby — safe-by-default.
|
||||
/// </summary>
|
||||
private bool DefaultIsActiveCheck()
|
||||
{
|
||||
var cluster = Cluster.Get(Context.System);
|
||||
var self = cluster.SelfMember;
|
||||
if (self.Status != MemberStatus.Up)
|
||||
return false;
|
||||
|
||||
var leader = cluster.State.Leader;
|
||||
return leader != null && leader == self.Address;
|
||||
}
|
||||
private bool DefaultIsActiveCheck() =>
|
||||
ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(Cluster.Get(Context.System));
|
||||
|
||||
// ── Internal messages ──
|
||||
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
using Akka.Cluster;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
|
||||
|
||||
/// <summary>
|
||||
/// THE single definition of "active node" (review 01 [High]; review 02 round 2 [Critical] N1):
|
||||
/// a node is active when it is the OLDEST Up member (optionally within a role scope) — i.e.
|
||||
/// the member the ClusterSingletonManager places singletons on. Cluster LEADERSHIP (lowest
|
||||
/// address) is an Akka-internal concept that diverges from singleton placement permanently
|
||||
/// once the original first node restarts and rejoins; every product-level active/standby
|
||||
/// decision must use this evaluator, never <c>cluster.State.Leader</c>.
|
||||
/// <para>
|
||||
/// Lives in Communication (not Host) so BOTH <c>SiteCommunicationActor</c> and
|
||||
/// <c>SiteReplicationActor</c> can default to it — Host cannot be referenced from either.
|
||||
/// The Host's <c>ClusterActivityEvaluator.SelfIsOldest</c> delegates here, so the S&F
|
||||
/// delivery gate (<c>IClusterNodeProvider.SelfIsPrimary</c>), the resync authority checks,
|
||||
/// and the heartbeat IsActive stamp all share one implementation.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class ActiveNodeEvaluator
|
||||
{
|
||||
/// <summary>True when self is Up and no other Up member (in the role scope) is older.</summary>
|
||||
/// <param name="cluster">The Akka cluster to evaluate.</param>
|
||||
/// <param name="role">Optional role scope; when set, only members with this role are considered.</param>
|
||||
/// <returns><c>true</c> when self is Up and the oldest Up member in the role scope.</returns>
|
||||
public static bool SelfIsOldestUp(Cluster cluster, string? role = null)
|
||||
{
|
||||
var self = cluster.SelfMember;
|
||||
if (self.Status != MemberStatus.Up)
|
||||
return false;
|
||||
if (role != null && !self.HasRole(role))
|
||||
return false;
|
||||
|
||||
return cluster.State.Members
|
||||
.Where(m => m.Status == MemberStatus.Up)
|
||||
.Where(m => role == null || m.HasRole(role))
|
||||
.All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
|
||||
}
|
||||
}
|
||||
@@ -122,4 +122,13 @@ public class CommunicationOptions
|
||||
/// count; this just bounds the subscriber list. Default 200.
|
||||
/// </summary>
|
||||
public int LiveAlarmCacheMaxSubscribersPerSite { get; set; } = 200;
|
||||
|
||||
/// <summary>
|
||||
/// Publish-coalescing window for live alarm deltas: an applied delta marks the cache
|
||||
/// dirty and one publish (fresh snapshot + per-viewer onChanged fan-out) fires after
|
||||
/// this window, batching an alarm storm into ~4 publishes/second instead of one per
|
||||
/// transition (review 02 round 2, N6). Zero = publish per delta (legacy). Seed and
|
||||
/// reconcile publishes are always immediate. Default 250 ms.
|
||||
/// </summary>
|
||||
public TimeSpan LiveAlarmCachePublishCoalesce { get; set; } = TimeSpan.FromMilliseconds(250);
|
||||
}
|
||||
|
||||
@@ -85,5 +85,10 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase<Communi
|
||||
// Per-site viewer cap must admit at least one viewer, else the page could never go live.
|
||||
builder.RequireThat(options.LiveAlarmCacheMaxSubscribersPerSite >= 1,
|
||||
$"Communication:LiveAlarmCacheMaxSubscribersPerSite must be at least 1 (was {options.LiveAlarmCacheMaxSubscribersPerSite}).");
|
||||
|
||||
// Publish-coalescing window drives a single-shot timer; TimeSpan.Zero is valid
|
||||
// (publish per delta — legacy), only a negative value is invalid.
|
||||
builder.RequireThat(options.LiveAlarmCachePublishCoalesce >= TimeSpan.Zero,
|
||||
$"Communication:LiveAlarmCachePublishCoalesce must be zero or a positive duration (was {options.LiveAlarmCachePublishCoalesce}).");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,9 +11,12 @@ namespace ZB.MOM.WW.ScadaBridge.Communication;
|
||||
/// reflects alarm transitions in near-real-time instead of re-polling every 15s.
|
||||
/// <para>
|
||||
/// <b>Hard constraint (locked <c>[PERM]</c>):</b> there is NO persisted central alarm
|
||||
/// store. This cache is purely in-memory and lives only on the active central node —
|
||||
/// no EF entity/table/migration backs it. On a NodeA↔NodeB failover the new active
|
||||
/// node re-seeds from scratch.
|
||||
/// store. This cache is purely in-memory, per-node — no EF entity/table/migration backs
|
||||
/// it. In routine operation only the active node (behind Traefik) hosts viewers, so only
|
||||
/// it runs aggregators; browsing the standby node directly (diagnostic ports) starts an
|
||||
/// independent read-only aggregator there — accepted (arch review 02 round 2, N8):
|
||||
/// bounded (one stream/site, viewer-capped), read-only, and torn down by the viewer
|
||||
/// linger. On a NodeA↔NodeB failover the new active node re-seeds from scratch.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Reference-counted lifecycle:</b> the first <see cref="Subscribe"/> for a site
|
||||
|
||||
@@ -233,7 +233,10 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache
|
||||
_grpcFactory,
|
||||
grpcA,
|
||||
grpcB,
|
||||
_options.LiveAlarmCacheReconcileInterval));
|
||||
_options.LiveAlarmCacheReconcileInterval,
|
||||
_options.LiveAlarmCachePublishCoalesce,
|
||||
TimeSpan.FromSeconds(5), // reconnect delay — former ReconnectDelay static default
|
||||
TimeSpan.FromSeconds(60))); // stability window — former StabilityWindow static default
|
||||
|
||||
entry.Actor = system.ActorOf(props, $"site-alarm-aggregator-{entry.SiteId}-{Guid.NewGuid():N}");
|
||||
entry.StartRetryTimer?.Dispose();
|
||||
@@ -291,7 +294,9 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache
|
||||
return null;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(site.GrpcNodeAAddress) || string.IsNullOrWhiteSpace(site.GrpcNodeBAddress))
|
||||
var nodeA = site.GrpcNodeAAddress;
|
||||
var nodeB = site.GrpcNodeBAddress;
|
||||
if (string.IsNullOrWhiteSpace(nodeA) && string.IsNullOrWhiteSpace(nodeB))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Live alarm cache: site {SiteId} ({SiteIdentifier}) has no gRPC node addresses; " +
|
||||
@@ -299,7 +304,13 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache
|
||||
return null;
|
||||
}
|
||||
|
||||
return (site.SiteIdentifier, site.GrpcNodeAAddress!, site.GrpcNodeBAddress!);
|
||||
// Single-endpoint site (review 02 round 2, N9): reuse the sole configured
|
||||
// endpoint for both slots — the aggregator's NodeA↔NodeB failover flip
|
||||
// degenerates to a reconnect against the same node, which is exactly the
|
||||
// right behavior for a one-node site.
|
||||
var grpcA = !string.IsNullOrWhiteSpace(nodeA) ? nodeA! : nodeB!;
|
||||
var grpcB = !string.IsNullOrWhiteSpace(nodeB) ? nodeB! : nodeA!;
|
||||
return (site.SiteIdentifier, grpcA, grpcB);
|
||||
}
|
||||
|
||||
// ── Snapshot fan-out (the seed / reconcile) ─────────────────────────────────
|
||||
|
||||
@@ -772,10 +772,20 @@ akka {{
|
||||
var replicationLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
|
||||
.CreateLogger<SiteReplicationActor>();
|
||||
|
||||
// ONE active-node predicate instance governs the S&F delivery gate, the resync
|
||||
// authority checks (SiteReplicationActor), and the heartbeat IsActive stamp
|
||||
// (SiteCommunicationActor, wired below) — review 02 round 2, N1. Null in
|
||||
// non-clustered test hosts: the actors fall back to the shared oldest-Up
|
||||
// evaluator, never to a leader check.
|
||||
var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>();
|
||||
Func<bool>? activeNodeCheck = clusterNodeProvider != null
|
||||
? () => clusterNodeProvider.SelfIsPrimary
|
||||
: null;
|
||||
|
||||
var replicationActor = _actorSystem!.ActorOf(
|
||||
Props.Create(() => new SiteReplicationActor(
|
||||
storage, sfStorage, replicationService, siteRole, replicationLogger,
|
||||
deploymentConfigFetcher, null, siteRuntimeOptionsValue, null)),
|
||||
deploymentConfigFetcher, activeNodeCheck, siteRuntimeOptionsValue, null)),
|
||||
"site-replication");
|
||||
|
||||
// Wire S&F replication handler to forward operations via the replication actor
|
||||
@@ -823,7 +833,8 @@ akka {{
|
||||
Props.Create(() => new SiteCommunicationActor(
|
||||
_nodeOptions.SiteId!,
|
||||
_communicationOptions,
|
||||
dmProxy)),
|
||||
dmProxy,
|
||||
activeNodeCheck)),
|
||||
"site-communication");
|
||||
|
||||
// Register local handlers with SiteCommunicationActor
|
||||
@@ -872,10 +883,11 @@ akka {{
|
||||
// tick so failover resumes delivery within one RetryTimerInterval.
|
||||
// IClusterNodeProvider.SelfIsPrimary is the canonical "this node is the
|
||||
// oldest Up member (singleton host)" check from the cluster-infrastructure
|
||||
// fix plan — the shared helper this seam was designed to accept. In a
|
||||
// non-clustered test host the provider is unregistered, so the gate stays
|
||||
// unset and the sweep is ungated (legacy behaviour, preserved).
|
||||
var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>();
|
||||
// fix plan — the shared helper this seam was designed to accept. The provider
|
||||
// is resolved once above (the same instance gating the resync authority and
|
||||
// heartbeat IsActive stamp — N1). In a non-clustered test host the provider is
|
||||
// unregistered, so the gate stays unset and the sweep is ungated (legacy
|
||||
// behaviour, preserved).
|
||||
if (clusterNodeProvider != null)
|
||||
{
|
||||
storeAndForwardService.SetDeliveryGate(() => clusterNodeProvider.SelfIsPrimary);
|
||||
|
||||
@@ -13,23 +13,15 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Health;
|
||||
/// </summary>
|
||||
public static class ClusterActivityEvaluator
|
||||
{
|
||||
/// <summary>True when self is Up and no other Up member (in the role scope) is older.</summary>
|
||||
/// <summary>True when self is Up and no other Up member (in the role scope) is older.
|
||||
/// Delegates to the shared <see cref="Communication.ClusterState.ActiveNodeEvaluator"/> —
|
||||
/// one implementation for the delivery gate, the resync authority checks, and the
|
||||
/// heartbeat IsActive stamp (review 02 round 2, N1).</summary>
|
||||
/// <param name="cluster">The Akka cluster to evaluate.</param>
|
||||
/// <param name="role">Optional role scope; when set, only members with this role are considered.</param>
|
||||
/// <returns><c>true</c> when self is Up and the oldest Up member in the role scope.</returns>
|
||||
public static bool SelfIsOldest(Cluster cluster, string? role = null)
|
||||
{
|
||||
var self = cluster.SelfMember;
|
||||
if (self.Status != MemberStatus.Up)
|
||||
return false;
|
||||
if (role != null && !self.HasRole(role))
|
||||
return false;
|
||||
|
||||
return cluster.State.Members
|
||||
.Where(m => m.Status == MemberStatus.Up)
|
||||
.Where(m => role == null || m.HasRole(role))
|
||||
.All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
|
||||
}
|
||||
public static bool SelfIsOldest(Cluster cluster, string? role = null) =>
|
||||
Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(cluster, role);
|
||||
|
||||
/// <summary>The oldest Up member in the role scope, or null while none is Up. Used for Primary/Standby labelling.</summary>
|
||||
/// <param name="cluster">The Akka cluster to evaluate.</param>
|
||||
|
||||
@@ -1583,6 +1583,18 @@ public class ManagementActor : ReceiveActor
|
||||
await repo.SaveChangesAsync();
|
||||
var commService = sp.GetService<CommunicationService>();
|
||||
commService?.RefreshSiteAddresses();
|
||||
|
||||
// Dispose the deleted site's cached gRPC channels — RemoveSiteAsync is the
|
||||
// factory's designed site-deletion disposal path and previously had no
|
||||
// production caller (arch review 02 round 2, N8): a deleted site's channels
|
||||
// (both node endpoints) otherwise persist until process restart. Null-safe:
|
||||
// test/composition roots without the factory skip it. Any live-alarm
|
||||
// aggregator for the site reconciles to an empty snapshot and is reaped by
|
||||
// the viewer linger (documented acceptance — see Component-Communication.md).
|
||||
var grpcFactory = sp.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcClientFactory>();
|
||||
if (grpcFactory is not null && site is not null)
|
||||
await grpcFactory.RemoveSiteAsync(site.SiteIdentifier);
|
||||
|
||||
await AuditAsync(sp, user, "Delete", "Site", cmd.SiteId.ToString(), site?.Name ?? cmd.SiteId.ToString(), null);
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -18,7 +18,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
/// Inbound: receives replicated operations from peer and applies to local SQLite.
|
||||
/// Uses fire-and-forget (Tell) — no ack wait per design.
|
||||
/// </summary>
|
||||
public class SiteReplicationActor : ReceiveActor
|
||||
public class SiteReplicationActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
private readonly SiteStorageService _storage;
|
||||
private readonly StoreAndForwardStorage _sfStorage;
|
||||
@@ -32,6 +32,30 @@ public class SiteReplicationActor : ReceiveActor
|
||||
private readonly TimeSpan _configFetchRetryDelay;
|
||||
private Address? _peerAddress;
|
||||
|
||||
/// <summary>Akka timer scheduler injected by the framework via <see cref="IWithTimers"/>.</summary>
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
// ── Chunked-resync assembly (standby side; actor-thread only) ──
|
||||
private string? _assemblingResyncId;
|
||||
private int _assemblingTotalChunks;
|
||||
private bool _assemblingTruncated;
|
||||
private readonly Dictionary<int, List<StoreAndForwardMessage>> _assemblingChunks = new();
|
||||
private const string ResyncAssemblyTimerKey = "sf-resync-assembly-timeout";
|
||||
|
||||
/// <summary>How long a partial chunk assembly may wait for its missing chunks before
|
||||
/// being discarded (a lost chunk = lost resync; the next peer-track retries). Ctor
|
||||
/// test seam; production default 30 s.</summary>
|
||||
private readonly TimeSpan _resyncAssemblyTimeout;
|
||||
|
||||
// ── Resync delivery confirmation (active side; actor-thread only) ──
|
||||
private string? _pendingAckResyncId;
|
||||
private const string ResyncAckTimerKey = "sf-resync-ack-timeout";
|
||||
|
||||
/// <summary>How long the active node waits for the standby's <see cref="SfBufferResyncAck"/>
|
||||
/// before warning + counting the resync as unacknowledged (lost chunks / dead peer). Ctor
|
||||
/// test seam; production default 60 s.</summary>
|
||||
private readonly TimeSpan _resyncAckTimeout;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum rows an active node returns in a single anti-entropy resync snapshot.
|
||||
/// A standby whose buffer exceeded this (Truncated snapshot) resyncs the oldest
|
||||
@@ -39,6 +63,48 @@ public class SiteReplicationActor : ReceiveActor
|
||||
/// </summary>
|
||||
private const int MaxResyncRows = 10_000;
|
||||
|
||||
/// <summary>
|
||||
/// Estimated per-chunk payload budget for a resync snapshot. Akka remoting's default
|
||||
/// <c>maximum-frame-size</c> is 128 000 bytes and <c>BuildHocon</c> sets no override,
|
||||
/// so the monolithic <see cref="SfBufferSnapshot"/> is silently undeliverable for any
|
||||
/// realistic backlog (review 02 round 2, N2). 64 000 bytes leaves ≈50% headroom for
|
||||
/// the JSON envelope, CLR type manifests, and the non-payload columns.
|
||||
/// </summary>
|
||||
internal const int MaxResyncChunkBytes = 64_000;
|
||||
|
||||
/// <summary>Row cap per resync chunk (bounds a chunk even when every row is tiny).</summary>
|
||||
internal const int MaxResyncChunkRows = 200;
|
||||
|
||||
/// <summary>
|
||||
/// Splits a resync snapshot into chunks that fit Akka remoting's default
|
||||
/// 128 000-byte frame (review 02 round 2, N2): rows accumulate until the estimated
|
||||
/// payload budget or the row cap is hit. Estimation is payload-dominated
|
||||
/// (payload_json length + 512 bytes fixed overhead per row); a single row whose
|
||||
/// payload exceeds the budget ships alone (Warning at the call site). Order is
|
||||
/// preserved (oldest-first, matching GetAllMessagesAsync).
|
||||
/// </summary>
|
||||
internal static List<List<StoreAndForwardMessage>> ChunkForRemoting(
|
||||
IReadOnlyList<StoreAndForwardMessage> rows, int maxChunkBytes, int maxChunkRows)
|
||||
{
|
||||
var chunks = new List<List<StoreAndForwardMessage>>();
|
||||
var current = new List<StoreAndForwardMessage>();
|
||||
var currentBytes = 0;
|
||||
foreach (var row in rows)
|
||||
{
|
||||
var estimate = (row.PayloadJson?.Length ?? 0) + 512;
|
||||
if (current.Count > 0 && (currentBytes + estimate > maxChunkBytes || current.Count >= maxChunkRows))
|
||||
{
|
||||
chunks.Add(current);
|
||||
current = new List<StoreAndForwardMessage>();
|
||||
currentBytes = 0;
|
||||
}
|
||||
current.Add(row);
|
||||
currentBytes += estimate;
|
||||
}
|
||||
if (current.Count > 0) chunks.Add(current);
|
||||
return chunks;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="SiteReplicationActor"/> and registers Akka message handlers.
|
||||
/// </summary>
|
||||
@@ -54,10 +120,11 @@ public class SiteReplicationActor : ReceiveActor
|
||||
/// config never crosses the intra-site Akka hop. Null on nodes/tests without a fetcher.
|
||||
/// </param>
|
||||
/// <param name="isActiveOverride">
|
||||
/// Test seam for the active-node check that gates the buffer-resync roles (a
|
||||
/// standby requests a resync, the active node answers). Null (production) uses
|
||||
/// the repo-standard leader+Up check via <see cref="Cluster"/> — swap point for
|
||||
/// plan 01's shared active-node helper.
|
||||
/// Active-node check that gates the buffer-resync roles (a standby requests a
|
||||
/// resync, the active node answers). Production wiring passes the Host's
|
||||
/// <c>IClusterNodeProvider.SelfIsPrimary</c> delegate (the same instance gating the
|
||||
/// S&F delivery sweep); null falls back to the shared oldest-Up evaluator
|
||||
/// (<see cref="Communication.ClusterState.ActiveNodeEvaluator"/>).
|
||||
/// </param>
|
||||
/// <param name="options">Site runtime options, including the config-fetch retry count; production defaults apply when null.</param>
|
||||
/// <param name="configFetchRetryDelay">Delay between config-fetch retry attempts; defaults to 2 seconds when null.</param>
|
||||
@@ -70,7 +137,9 @@ public class SiteReplicationActor : ReceiveActor
|
||||
IDeploymentConfigFetcher? configFetcher = null,
|
||||
Func<bool>? isActiveOverride = null,
|
||||
SiteRuntimeOptions? options = null,
|
||||
TimeSpan? configFetchRetryDelay = null)
|
||||
TimeSpan? configFetchRetryDelay = null,
|
||||
TimeSpan? resyncAssemblyTimeout = null,
|
||||
TimeSpan? resyncAckTimeout = null)
|
||||
{
|
||||
_storage = storage;
|
||||
_sfStorage = sfStorage;
|
||||
@@ -80,6 +149,8 @@ public class SiteReplicationActor : ReceiveActor
|
||||
_logger = logger;
|
||||
_cluster = Cluster.Get(Context.System);
|
||||
_isActive = isActiveOverride ?? DefaultIsActive;
|
||||
_resyncAssemblyTimeout = resyncAssemblyTimeout ?? TimeSpan.FromSeconds(30);
|
||||
_resyncAckTimeout = resyncAckTimeout ?? TimeSpan.FromSeconds(60);
|
||||
// UA2: bound the standby's replicated-config fetch retries. At least one
|
||||
// attempt always runs; the fixed inter-attempt delay is a test seam
|
||||
// (production default 2 s).
|
||||
@@ -110,7 +181,30 @@ public class SiteReplicationActor : ReceiveActor
|
||||
|
||||
// Anti-entropy — full S&F buffer resync on peer (re)join
|
||||
Receive<RequestSfBufferResync>(HandleRequestSfBufferResync);
|
||||
Receive<SfBufferSnapshot>(HandleSfBufferSnapshot);
|
||||
Receive<SfResyncSnapshotLoaded>(HandleSfResyncSnapshotLoaded);
|
||||
Receive<SfBufferSnapshotChunk>(HandleSfBufferSnapshotChunk);
|
||||
Receive<ResyncAssemblyTimedOut>(HandleResyncAssemblyTimedOut);
|
||||
Receive<SfBufferResyncAck>(msg =>
|
||||
{
|
||||
if (msg.ResyncId == _pendingAckResyncId)
|
||||
{
|
||||
_pendingAckResyncId = null;
|
||||
Timers.Cancel(ResyncAckTimerKey);
|
||||
}
|
||||
ScadaBridgeTelemetry.RecordSfResyncCompleted();
|
||||
_logger.LogInformation("S&F resync {ResyncId} acknowledged by standby: {Rows} row(s) applied",
|
||||
msg.ResyncId, msg.RowCount);
|
||||
});
|
||||
Receive<ResyncAckTimedOut>(msg =>
|
||||
{
|
||||
if (msg.ResyncId != _pendingAckResyncId) return;
|
||||
_pendingAckResyncId = null;
|
||||
ScadaBridgeTelemetry.RecordSfResyncAckMissing();
|
||||
_logger.LogWarning(
|
||||
"S&F resync {ResyncId} was never acknowledged within {Window} — snapshot chunks may have been lost (frame drop / dead peer); the next peer-track retries",
|
||||
msg.ResyncId, _resyncAckTimeout);
|
||||
});
|
||||
Receive<SfBufferSnapshot>(HandleSfBufferSnapshot); // legacy monolithic handler — retained for rolling compat
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -181,21 +275,17 @@ public class SiteReplicationActor : ReceiveActor
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Repo-standard active-node check: this node is active when it is the current
|
||||
/// cluster leader AND its own <see cref="MemberStatus"/> is
|
||||
/// <see cref="MemberStatus.Up"/>. Mirrors <c>SiteCommunicationActor.DefaultIsActiveCheck</c>
|
||||
/// (swap point for plan 01's shared helper). Any other state reports standby —
|
||||
/// safe-by-default.
|
||||
/// Repo-standard active-node check: this node is active when it is the OLDEST Up
|
||||
/// member carrying the site role — the same oldest-Up semantics as the S&F delivery
|
||||
/// gate (IClusterNodeProvider.SelfIsPrimary → ClusterActivityEvaluator → shared
|
||||
/// ActiveNodeEvaluator). NEVER the cluster leader: leadership is lowest-address and
|
||||
/// diverges from singleton/delivery placement permanently after the lower-address
|
||||
/// node restarts — the divergence that made the delivering node wipe its own live
|
||||
/// buffer via a wrong-direction resync (review 02 round 2, N1 Critical). Any other
|
||||
/// state reports standby — safe-by-default.
|
||||
/// </summary>
|
||||
private bool DefaultIsActive()
|
||||
{
|
||||
var self = _cluster.SelfMember;
|
||||
if (self.Status != MemberStatus.Up)
|
||||
return false;
|
||||
|
||||
var leader = _cluster.State.Leader;
|
||||
return leader != null && leader == self.Address;
|
||||
}
|
||||
private bool DefaultIsActive() =>
|
||||
Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(_cluster, _siteRole);
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates the active-node check, treating a throwing check as standby
|
||||
@@ -391,9 +481,10 @@ public class SiteReplicationActor : ReceiveActor
|
||||
|
||||
/// <summary>
|
||||
/// Active-node side of the anti-entropy resync: answers a standby's
|
||||
/// <see cref="RequestSfBufferResync"/> with a full-buffer <see cref="SfBufferSnapshot"/>
|
||||
/// (up to <see cref="MaxResyncRows"/> oldest rows). A non-active node ignores the
|
||||
/// request — only the authoritative node may answer.
|
||||
/// <see cref="RequestSfBufferResync"/> with a sequence of byte-budgeted
|
||||
/// <see cref="SfBufferSnapshotChunk"/>s (up to <see cref="MaxResyncRows"/> oldest rows).
|
||||
/// A non-active node ignores the request — only the authoritative node may answer.
|
||||
/// The snapshot is piped back to Self so chunking + ack bookkeeping stays actor-safe.
|
||||
/// </summary>
|
||||
private void HandleRequestSfBufferResync(RequestSfBufferResync msg)
|
||||
{
|
||||
@@ -405,9 +496,37 @@ public class SiteReplicationActor : ReceiveActor
|
||||
|
||||
var replyTo = Sender;
|
||||
_sfStorage.GetAllMessagesAsync(MaxResyncRows).PipeTo(
|
||||
replyTo,
|
||||
Self,
|
||||
success: result => new SfBufferSnapshot(result.Messages, result.Truncated));
|
||||
failure: ex => new Status.Failure(ex),
|
||||
success: result => new SfResyncSnapshotLoaded(replyTo, result.Messages, result.Truncated));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Active-node continuation: the resync snapshot finished loading; chunk it to fit the
|
||||
/// remoting frame and send the sequenced chunks to the requester (all sharing one
|
||||
/// resyncId). Task 7 arms the ack-timeout here.
|
||||
/// </summary>
|
||||
private void HandleSfResyncSnapshotLoaded(SfResyncSnapshotLoaded msg)
|
||||
{
|
||||
var chunks = ChunkForRemoting(msg.Messages, MaxResyncChunkBytes, MaxResyncChunkRows);
|
||||
if (chunks.Count == 0) chunks.Add(new List<StoreAndForwardMessage>()); // empty buffer still resyncs (clears the standby)
|
||||
var resyncId = Guid.NewGuid().ToString("N");
|
||||
for (var i = 0; i < chunks.Count; i++)
|
||||
{
|
||||
if (chunks[i].Count == 1 && (chunks[i][0].PayloadJson?.Length ?? 0) + 512 > MaxResyncChunkBytes)
|
||||
_logger.LogWarning(
|
||||
"Resync row {Id} alone exceeds the chunk budget ({Bytes}B payload); sending solo — it may exceed the remoting frame",
|
||||
chunks[i][0].Id, chunks[i][0].PayloadJson?.Length ?? 0);
|
||||
msg.ReplyTo.Tell(new SfBufferSnapshotChunk(resyncId, i + 1, chunks.Count, chunks[i], msg.Truncated), Self);
|
||||
}
|
||||
_logger.LogInformation(
|
||||
"Answered S&F resync request with {Rows} row(s) in {Chunks} chunk(s), resyncId={ResyncId}",
|
||||
msg.Messages.Count, chunks.Count, resyncId);
|
||||
// Arm the ack window: absence of an SfBufferResyncAck within it surfaces as a
|
||||
// Warning + counter (the silent-loss mode N2 flagged). Single-outstanding-resync
|
||||
// bookkeeping: a new request supersedes by overwriting the id and restarting the timer.
|
||||
_pendingAckResyncId = resyncId;
|
||||
Timers.StartSingleTimer(ResyncAckTimerKey, new ResyncAckTimedOut(resyncId), _resyncAckTimeout);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -435,13 +554,120 @@ public class SiteReplicationActor : ReceiveActor
|
||||
_logger.LogInformation(
|
||||
"Applying S&F buffer resync snapshot ({Count} rows), replacing local buffer", msg.Messages.Count);
|
||||
|
||||
_sfStorage.ReplaceAllAsync(msg.Messages)
|
||||
Task.Run(async () =>
|
||||
{
|
||||
// Belt-and-braces (N1): re-check at apply time. ReplaceAllAsync discards
|
||||
// every in-flight row (StoreAndForwardStorage.cs "Never call on an active
|
||||
// node"); a flip between message receipt and this point must abort.
|
||||
if (SafeIsActive())
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Discarding S&F buffer resync snapshot: this node became active before apply");
|
||||
return;
|
||||
}
|
||||
await _sfStorage.ReplaceAllAsync(msg.Messages);
|
||||
})
|
||||
.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
_logger.LogError(t.Exception, "Failed to apply S&F buffer resync snapshot");
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Standby-node side of the chunked anti-entropy resync: accumulates the chunks of one
|
||||
/// <c>ResyncId</c>, and once all have arrived, assembles them in sequence order and
|
||||
/// replaces the local buffer atomically, then acks. A new <c>ResyncId</c> discards any
|
||||
/// stale partial assembly (review 02 round 2, N2). An active node ignores chunks.
|
||||
/// </summary>
|
||||
private void HandleSfBufferSnapshotChunk(SfBufferSnapshotChunk msg)
|
||||
{
|
||||
if (SafeIsActive())
|
||||
{
|
||||
_logger.LogDebug("Ignoring S&F resync chunk — this node is active");
|
||||
return;
|
||||
}
|
||||
|
||||
if (_assemblingResyncId != msg.ResyncId)
|
||||
{
|
||||
if (_assemblingResyncId != null)
|
||||
_logger.LogWarning(
|
||||
"Discarding partial S&F resync assembly {Old} ({Have}/{Want} chunks): a new resync {New} superseded it",
|
||||
_assemblingResyncId, _assemblingChunks.Count, _assemblingTotalChunks, msg.ResyncId);
|
||||
_assemblingResyncId = msg.ResyncId;
|
||||
_assemblingTotalChunks = msg.TotalChunks;
|
||||
_assemblingTruncated = msg.Truncated;
|
||||
_assemblingChunks.Clear();
|
||||
}
|
||||
|
||||
_assemblingChunks[msg.Sequence] = msg.Messages;
|
||||
Timers.StartSingleTimer(ResyncAssemblyTimerKey, new ResyncAssemblyTimedOut(msg.ResyncId), _resyncAssemblyTimeout);
|
||||
|
||||
if (_assemblingChunks.Count < _assemblingTotalChunks)
|
||||
return;
|
||||
|
||||
// Complete: assemble in sequence order and apply atomically.
|
||||
var assembled = Enumerable.Range(1, _assemblingTotalChunks)
|
||||
.SelectMany(seq => _assemblingChunks[seq])
|
||||
.ToList();
|
||||
var resyncId = _assemblingResyncId!;
|
||||
var truncated = _assemblingTruncated;
|
||||
_assemblingResyncId = null;
|
||||
_assemblingChunks.Clear();
|
||||
Timers.Cancel(ResyncAssemblyTimerKey);
|
||||
|
||||
if (truncated)
|
||||
_logger.LogWarning(
|
||||
"S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally",
|
||||
MaxResyncRows);
|
||||
_logger.LogInformation(
|
||||
"Applying chunked S&F resync {ResyncId} ({Count} rows), replacing local buffer", resyncId, assembled.Count);
|
||||
|
||||
// KNOWN, ACCEPTED race (review 02 round 2, N5 — do NOT "fix" this into something
|
||||
// worse): a replicated Remove sent after the active node read its snapshot but
|
||||
// before the snapshot's chunks is ordered BEFORE them on the wire (same
|
||||
// sender/receiver pair), so this apply can re-add the removed row → an orphan
|
||||
// Pending row that a later failover re-delivers ONCE. Bounded, self-correcting at
|
||||
// the next resync, and inherent to no-ack replication; a delivered-side dedup or
|
||||
// op-sequencing scheme would cost far more than one rare duplicate.
|
||||
var replyTo = Sender;
|
||||
Task.Run(async () =>
|
||||
{
|
||||
if (SafeIsActive()) // belt-and-braces, mirrors the monolithic path (T3)
|
||||
{
|
||||
_logger.LogWarning("Discarding chunked S&F resync {ResyncId}: node became active before apply", resyncId);
|
||||
return;
|
||||
}
|
||||
await _sfStorage.ReplaceAllAsync(assembled);
|
||||
replyTo.Tell(new SfBufferResyncAck(resyncId, assembled.Count));
|
||||
})
|
||||
.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
_logger.LogError(t.Exception, "Failed to apply chunked S&F resync {ResyncId}", resyncId);
|
||||
});
|
||||
}
|
||||
|
||||
private void HandleResyncAssemblyTimedOut(ResyncAssemblyTimedOut msg)
|
||||
{
|
||||
if (_assemblingResyncId != msg.ResyncId) return; // superseded already
|
||||
_logger.LogWarning(
|
||||
"S&F resync assembly {ResyncId} timed out with {Have}/{Want} chunks — discarding partial (a lost chunk; the next peer-track retries)",
|
||||
msg.ResyncId, _assemblingChunks.Count, _assemblingTotalChunks);
|
||||
ScadaBridgeTelemetry.RecordReplicationFailure();
|
||||
_assemblingResyncId = null;
|
||||
_assemblingChunks.Clear();
|
||||
}
|
||||
|
||||
/// <summary>Internal: the resync snapshot finished loading; chunk and send to the requester.</summary>
|
||||
internal sealed record SfResyncSnapshotLoaded(
|
||||
IActorRef ReplyTo, List<StoreAndForwardMessage> Messages, bool Truncated);
|
||||
|
||||
/// <summary>Internal: a partial chunk assembly for <paramref name="ResyncId"/> exceeded its window.</summary>
|
||||
internal sealed record ResyncAssemblyTimedOut(string ResyncId);
|
||||
|
||||
/// <summary>Internal: the active node's ack window for <paramref name="ResyncId"/> expired.</summary>
|
||||
internal sealed record ResyncAckTimedOut(string ResyncId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -458,3 +684,24 @@ public sealed record RequestSfBufferResync;
|
||||
/// Akka remoting; the message list rides the default serializer.
|
||||
/// </summary>
|
||||
public sealed record SfBufferSnapshot(List<StoreAndForwardMessage> Messages, bool Truncated);
|
||||
|
||||
/// <summary>
|
||||
/// Active→standby: one sequenced chunk of a full-buffer anti-entropy snapshot
|
||||
/// (review 02 round 2, N2 — the monolithic <see cref="SfBufferSnapshot"/> exceeds Akka
|
||||
/// remoting's default 128 000-byte frame for any realistic backlog). All chunks of one
|
||||
/// resync share <paramref name="ResyncId"/>; <paramref name="Sequence"/> is 1-based up to
|
||||
/// <paramref name="TotalChunks"/>. Additive message — the legacy monolithic snapshot
|
||||
/// handler is retained for rolling upgrades. Crosses intra-site Akka remoting (NOT
|
||||
/// ClusterClient — ClusterClientContractLockTests is intentionally not involved).
|
||||
/// </summary>
|
||||
public sealed record SfBufferSnapshotChunk(
|
||||
string ResyncId, int Sequence, int TotalChunks,
|
||||
List<StoreAndForwardMessage> Messages, bool Truncated);
|
||||
|
||||
/// <summary>
|
||||
/// Standby→active: delivery confirmation — the standby assembled all chunks of
|
||||
/// <paramref name="ResyncId"/> and applied them atomically (<paramref name="RowCount"/>
|
||||
/// rows installed). Absence within the ack window is surfaced by the active node
|
||||
/// (Warning + counter) — the silent-loss mode N2 flagged.
|
||||
/// </summary>
|
||||
public sealed record SfBufferResyncAck(string ResyncId, int RowCount);
|
||||
|
||||
@@ -31,5 +31,13 @@ public sealed class StoreAndForwardOptionsValidator : OptionsValidatorBase<Store
|
||||
builder.RequireThat(options.DefaultMaxRetries >= 0,
|
||||
$"ScadaBridge:StoreAndForward:DefaultMaxRetries must be >= 0 " +
|
||||
$"(was {options.DefaultMaxRetries}).");
|
||||
|
||||
builder.RequireThat(options.SweepBatchLimit >= 0,
|
||||
$"ScadaBridge:StoreAndForward:SweepBatchLimit must be >= 0 " +
|
||||
$"(was {options.SweepBatchLimit}); it bounds due rows per retry sweep — 0 means unlimited (legacy).");
|
||||
|
||||
builder.RequireThat(options.SweepTargetParallelism >= 1,
|
||||
$"ScadaBridge:StoreAndForward:SweepTargetParallelism must be >= 1 " +
|
||||
$"(was {options.SweepTargetParallelism}); it caps concurrent (category,target) sweep lanes — 1 means serial.");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -394,7 +394,7 @@ public class StoreAndForwardService
|
||||
});
|
||||
|
||||
_retryTimer = new Timer(
|
||||
_ => Volatile.Write(ref _sweepTask, RetryPendingMessagesAsync()),
|
||||
_ => KickSweep(),
|
||||
null,
|
||||
_options.RetryTimerInterval,
|
||||
_options.RetryTimerInterval);
|
||||
@@ -658,17 +658,37 @@ public class StoreAndForwardService
|
||||
/// <summary>
|
||||
/// Kicks a background retry sweep now (fire-and-forget). No-op before
|
||||
/// <see cref="StartAsync"/> (storage may be uninitialized and the timer, whose
|
||||
/// presence gates this, is not yet set). Overlap-safe: the sweep's
|
||||
/// <see cref="_retryInProgress"/> CAS makes a concurrent kick a cheap no-op.
|
||||
/// presence gates this, is not yet set). Overlap-safe: publishes into
|
||||
/// <see cref="_sweepTask"/> only when the CAS is won — see <see cref="KickSweep"/>.
|
||||
/// </summary>
|
||||
public void TriggerSweep()
|
||||
{
|
||||
if (_retryTimer == null) return;
|
||||
Volatile.Write(ref _sweepTask, RetryPendingMessagesAsync());
|
||||
KickSweep();
|
||||
}
|
||||
|
||||
/// <summary>The current drain handle — test seam for the N3 clobber regression.</summary>
|
||||
internal Task? CurrentSweepTaskForTest => Volatile.Read(ref _sweepTask);
|
||||
|
||||
/// <summary>
|
||||
/// Starts a sweep IF none is in flight, publishing the new task into
|
||||
/// <see cref="_sweepTask"/> only when this call wins the <see cref="_retryInProgress"/>
|
||||
/// CAS. A kick that loses the CAS returns without touching <see cref="_sweepTask"/> —
|
||||
/// pre-fix it overwrote the drain handle with an instantly-completed no-op, so
|
||||
/// <see cref="StopAsync"/> proceeded with disposal under a still-running sweep
|
||||
/// (review 02 round 2, N3) — defeating the drain exactly when a sweep outlives a tick.
|
||||
/// </summary>
|
||||
private void KickSweep()
|
||||
{
|
||||
if (Interlocked.CompareExchange(ref _retryInProgress, 1, 0) != 0)
|
||||
return;
|
||||
Volatile.Write(ref _sweepTask, RunSweepOwnedAsync());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Background retry sweep. Processes all pending messages that are due for retry.
|
||||
/// Self-CASes for direct callers (existing tests); the entry CAS is skipped when
|
||||
/// ownership is already held by <see cref="KickSweep"/>.
|
||||
/// </summary>
|
||||
/// <returns>A task representing the asynchronous retry sweep.</returns>
|
||||
internal async Task RetryPendingMessagesAsync()
|
||||
@@ -676,7 +696,16 @@ public class StoreAndForwardService
|
||||
// Prevent overlapping retry sweeps
|
||||
if (Interlocked.CompareExchange(ref _retryInProgress, 1, 0) != 0)
|
||||
return;
|
||||
await RunSweepOwnedAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The actual retry sweep body. The caller MUST already own the
|
||||
/// <see cref="_retryInProgress"/> flag (won the CAS); this method releases it in its
|
||||
/// finally. Never call directly without holding ownership.
|
||||
/// </summary>
|
||||
private async Task RunSweepOwnedAsync()
|
||||
{
|
||||
try
|
||||
{
|
||||
var gate = _deliveryGate;
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using Akka.Actor;
|
||||
using Akka.Configuration;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
|
||||
|
||||
public class ActiveNodeEvaluatorTests : IAsyncLifetime
|
||||
{
|
||||
private ActorSystem? _system;
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
var port = FreePort();
|
||||
var config = ConfigurationFactory.ParseString($@"
|
||||
akka {{
|
||||
actor.provider = cluster
|
||||
remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {port} }}
|
||||
cluster {{
|
||||
seed-nodes = [""akka.tcp://ane-test@127.0.0.1:{port}""]
|
||||
roles = [""site-x""]
|
||||
min-nr-of-members = 1
|
||||
}}
|
||||
}}");
|
||||
_system = ActorSystem.Create("ane-test", config);
|
||||
var cluster = Akka.Cluster.Cluster.Get(_system);
|
||||
var deadline = DateTime.UtcNow.AddSeconds(20);
|
||||
while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline)
|
||||
await Task.Delay(100);
|
||||
}
|
||||
|
||||
public async Task DisposeAsync() { if (_system != null) await _system.Terminate(); }
|
||||
|
||||
[Fact]
|
||||
public void SelfIsOldestUp_SoleUpMember_ReturnsTrue()
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(_system!);
|
||||
Assert.True(ActiveNodeEvaluator.SelfIsOldestUp(cluster));
|
||||
Assert.True(ActiveNodeEvaluator.SelfIsOldestUp(cluster, "site-x"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfIsOldestUp_RoleNotHeld_ReturnsFalse()
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(_system!);
|
||||
Assert.False(ActiveNodeEvaluator.SelfIsOldestUp(cluster, "role-nonexistent"));
|
||||
}
|
||||
|
||||
private static int FreePort()
|
||||
{
|
||||
using var l = new TcpListener(IPAddress.Loopback, 0);
|
||||
l.Start();
|
||||
return ((IPEndPoint)l.LocalEndpoint).Port;
|
||||
}
|
||||
}
|
||||
@@ -86,4 +86,22 @@ public class CommunicationOptionsValidatorTests
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LiveAlarmCacheMaxSubscribersPerSite", result.FailureMessage);
|
||||
}
|
||||
|
||||
// ── R2 T10: live-delta publish-coalescing window (N6) ────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void ZeroLiveAlarmCachePublishCoalesce_IsValid()
|
||||
{
|
||||
// Zero = publish per delta (legacy behavior).
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCachePublishCoalesce = TimeSpan.Zero });
|
||||
Assert.True(result.Succeeded, result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeLiveAlarmCachePublishCoalesce_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCachePublishCoalesce = TimeSpan.FromMilliseconds(-1) });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LiveAlarmCachePublishCoalesce", result.FailureMessage);
|
||||
}
|
||||
}
|
||||
|
||||
+92
-4
@@ -24,10 +24,12 @@ public class SiteAlarmAggregatorActorTests : TestKit
|
||||
|
||||
public SiteAlarmAggregatorActorTests() : base(@"akka.loglevel = WARNING")
|
||||
{
|
||||
SiteAlarmAggregatorActor.ReconnectDelay = TimeSpan.FromMilliseconds(50);
|
||||
SiteAlarmAggregatorActor.StabilityWindow = TimeSpan.FromSeconds(30);
|
||||
}
|
||||
|
||||
// Former process-global static test seams, now ctor-injected per actor (R2 T12).
|
||||
private static readonly TimeSpan TestReconnectDelay = TimeSpan.FromMilliseconds(50);
|
||||
private static readonly TimeSpan TestStabilityWindow = TimeSpan.FromSeconds(30);
|
||||
|
||||
// ── Test doubles ────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
@@ -137,7 +139,7 @@ public class SiteAlarmAggregatorActorTests : TestKit
|
||||
}
|
||||
|
||||
private (IActorRef Actor, SeedStub Seed, PublishSink Sink, MockSiteAlarmStreamClientFactory Factory) CreateActor(
|
||||
TimeSpan? reconcileInterval = null)
|
||||
TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null)
|
||||
{
|
||||
var seed = new SeedStub();
|
||||
var sink = new PublishSink();
|
||||
@@ -145,7 +147,10 @@ public class SiteAlarmAggregatorActorTests : TestKit
|
||||
|
||||
var props = Props.Create(() => new SiteAlarmAggregatorActor(
|
||||
SiteId, "corr-1", seed.Seed, sink.Publish, factory, GrpcNodeA, GrpcNodeB,
|
||||
reconcileInterval ?? TimeSpan.FromMinutes(10)));
|
||||
reconcileInterval ?? TimeSpan.FromMinutes(10),
|
||||
publishCoalesce ?? TimeSpan.Zero,
|
||||
TestReconnectDelay,
|
||||
TestStabilityWindow));
|
||||
|
||||
var actor = Sys.ActorOf(props);
|
||||
return (actor, seed, sink, factory);
|
||||
@@ -396,6 +401,89 @@ public class SiteAlarmAggregatorActorTests : TestKit
|
||||
AwaitCondition(() => TotalSubs() > before, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
// ── R2 T11: queued re-seed after reconnect + stream-generation stamp (N7.1/N7.2) ──
|
||||
|
||||
[Fact]
|
||||
public void ReseedRequestedDuringInFlightFanout_IsQueued_NotDropped()
|
||||
{
|
||||
var (actor, seed, sink, factory) = CreateActor(publishCoalesce: TimeSpan.Zero);
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext(); // initial seed done (CallCount 1)
|
||||
AwaitAssert(() => Assert.Equal(1, sink.Count));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
actor.Tell(new RunReconcile()); // reconcile fan-out now in flight (CallCount 2)
|
||||
AwaitAssert(() => Assert.Equal(2, seed.CallCount));
|
||||
|
||||
// Stream error while the reconcile is in flight → the failover re-seed must not be
|
||||
// silently skipped. Pre-fix: StartFanout no-ops and CallCount stays 2 until the next
|
||||
// 60s reconcile tick.
|
||||
factory.ClientFor(GrpcNodeA).Subs.Last().OnError(new Exception("stream fault"));
|
||||
|
||||
seed.CompleteNext(); // finish the in-flight reconcile
|
||||
AwaitAssert(() => Assert.Equal(3, seed.CallCount)); // queued re-seed ran immediately after
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LateErrorFromAPreviousStreamGeneration_IsIgnored()
|
||||
{
|
||||
var (_, seed, _, factory) = CreateActor(publishCoalesce: TimeSpan.Zero);
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext();
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
int TotalSubs() => factory.ClientFor(GrpcNodeA).Subs.Count + factory.ClientFor(GrpcNodeB).Subs.Count;
|
||||
|
||||
var firstSub = factory.ClientFor(GrpcNodeA).Subs.Single();
|
||||
firstSub.OnError(new Exception("real fault")); // gen 1 error → flip, reconnect
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5)); // gen 2 stream open
|
||||
AwaitAssert(() => Assert.Equal(2, TotalSubs()));
|
||||
|
||||
firstSub.OnError(new Exception("late zombie fault")); // stale gen-1 error races in
|
||||
// Pre-fix this burns retry budget and opens a THIRD stream; post-fix it is ignored.
|
||||
Thread.Sleep(400);
|
||||
Assert.Equal(2, TotalSubs());
|
||||
}
|
||||
|
||||
// ── R2 T10: live-delta publish coalescing (N6) ──
|
||||
|
||||
[Fact]
|
||||
public void DeltaBurst_IsCoalesced_IntoFewPublishes_ContainingEveryRow()
|
||||
{
|
||||
var (_, seed, sink, factory) = CreateActor(publishCoalesce: TimeSpan.FromMilliseconds(150));
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext(); // empty initial seed
|
||||
AwaitAssert(() => Assert.Equal(1, sink.Count)); // seed publish, immediate
|
||||
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
var sub = factory.ClientFor(GrpcNodeA).Subs.Single();
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
for (var i = 0; i < 50; i++)
|
||||
sub.OnAlarm(Alarm($"A{i}", "", 900, now)); // 50 distinct keys, one burst
|
||||
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
Assert.Equal(50, sink.Latest!.Count); // nothing lost
|
||||
Assert.InRange(sink.Count, 2, 4); // pre-fix: 51 publishes (1 seed + 1 per delta)
|
||||
}, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ZeroCoalesce_PreservesLegacyPerDeltaPublish()
|
||||
{
|
||||
var (_, seed, sink, factory) = CreateActor(publishCoalesce: TimeSpan.Zero);
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext();
|
||||
AwaitAssert(() => Assert.Equal(1, sink.Count));
|
||||
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
var sub = factory.ClientFor(GrpcNodeA).Subs.Single();
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
sub.OnAlarm(Alarm("A1", "", 900, now));
|
||||
sub.OnAlarm(Alarm("A2", "", 900, now));
|
||||
AwaitAssert(() => Assert.Equal(3, sink.Count)); // 1 seed + 1 per delta
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stop_Message_TearsDown_Grpc_And_Stops_Actor()
|
||||
{
|
||||
|
||||
@@ -258,4 +258,48 @@ public class SiteAlarmLiveCacheServiceTests : TestKit
|
||||
Assert.False(service.IsLive(999));
|
||||
Assert.Empty(service.GetCurrentAlarms(999));
|
||||
}
|
||||
|
||||
// ── R2 T15: single-endpoint sites can go live (N9) ──
|
||||
|
||||
[Fact]
|
||||
public void SiteWithOnlyNodeAEndpoint_StartsAnAggregator_AgainstThatEndpoint()
|
||||
{
|
||||
var site = new Site("Three", "site-3")
|
||||
{
|
||||
Id = 3,
|
||||
GrpcNodeAAddress = "http://only-node:8083",
|
||||
GrpcNodeBAddress = null, // single-endpoint site — pre-fix: never goes live
|
||||
};
|
||||
|
||||
var siteRepo = Substitute.For<ISiteRepository>();
|
||||
siteRepo.GetSiteByIdAsync(3, Arg.Any<CancellationToken>()).Returns(site);
|
||||
|
||||
var instanceRepo = Substitute.For<ITemplateEngineRepository>();
|
||||
instanceRepo.GetInstancesBySiteIdAsync(3, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<Instance>());
|
||||
|
||||
var services = new ServiceCollection();
|
||||
services.AddScoped(_ => siteRepo);
|
||||
services.AddScoped(_ => instanceRepo);
|
||||
var provider = services.BuildServiceProvider();
|
||||
|
||||
var options = Options.Create(new CommunicationOptions
|
||||
{
|
||||
LiveAlarmCacheLinger = TimeSpan.FromSeconds(30),
|
||||
LiveAlarmCacheReconcileInterval = TimeSpan.FromMinutes(10),
|
||||
});
|
||||
var comm = new CommunicationService(Options.Create(new CommunicationOptions()),
|
||||
NullLogger<CommunicationService>.Instance);
|
||||
var factory = new CountingFactory();
|
||||
|
||||
var service = new SiteAlarmLiveCacheService(
|
||||
provider, comm, factory, options, NullLogger<SiteAlarmLiveCacheService>.Instance);
|
||||
service.SetActorSystem(Sys);
|
||||
|
||||
using var sub = service.Subscribe(3, () => { });
|
||||
|
||||
// Pre-fix: ResolveSiteAsync required BOTH endpoints and bailed → never live, no client.
|
||||
AwaitCondition(() => service.IsLive(3), TimeSpan.FromSeconds(5));
|
||||
Assert.True(factory.GetOrCreateCount >= 1); // a client was created for the single endpoint
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// N1 regression (review 02 round 2, Critical): the resync authority must use the same
|
||||
/// oldest-Up predicate as the S&F delivery gate. Divergence scenario = the delivering node
|
||||
/// is OLDEST but not LEADER (leader = lowest address), the exact state a rolling restart of
|
||||
/// the lower-address node produces. Pre-fix the delivering node requests a resync from the
|
||||
/// stale peer and ReplaceAllAsync wipes its live buffer.
|
||||
/// </summary>
|
||||
public class SfBufferResyncPredicateTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task OldestButNotLeaderNode_KeepsItsBuffer_AndSeedsTheJoiner()
|
||||
{
|
||||
// Two explicit ports, deliberately assigned so the FIRST-started (oldest,
|
||||
// delivering) node has the HIGHER address → the second node is cluster leader.
|
||||
var p1 = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var p2 = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1);
|
||||
|
||||
await using var fixture = await TwoNodeClusterFixture.StartAsync(
|
||||
role: "site-int", portA: portHigh, portB: portLow);
|
||||
|
||||
// Real S&F storage + replication actor per node, production default predicate
|
||||
// (no isActiveOverride) — the exact wiring under test.
|
||||
var (storageOldest, _) = await CreateReplicationActorAsync(fixture.NodeA, "oldest");
|
||||
var (storageJoiner, _) = await CreateReplicationActorAsync(fixture.NodeB, "joiner");
|
||||
|
||||
// The delivering (oldest) node has a live buffered row the standby never saw.
|
||||
await storageOldest.EnqueueAsync(NewMessage("live-row"));
|
||||
|
||||
// Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot
|
||||
// in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not
|
||||
// needed; OnPeerTracked already fired on join. The resync exchange is async:
|
||||
// wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner,
|
||||
// the correct direction). Pre-fix this times out (the joiner, as leader, never
|
||||
// requests) AND the oldest node's row is deleted by the stale wipe.
|
||||
await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null,
|
||||
TimeSpan.FromSeconds(20),
|
||||
"joiner never received the resync snapshot (resync ran in the wrong direction)");
|
||||
|
||||
// And the delivering node's buffer is untouched — the N1 wipe assertion.
|
||||
Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row"));
|
||||
}
|
||||
|
||||
private static async Task<(StoreAndForwardStorage Storage, IActorRef Actor)> CreateReplicationActorAsync(
|
||||
ActorSystem node, string tag)
|
||||
{
|
||||
var sfDb = Path.Combine(Path.GetTempPath(), $"sf-resync-{tag}-{Guid.NewGuid():N}.db");
|
||||
var siteDb = Path.Combine(Path.GetTempPath(), $"site-resync-{tag}-{Guid.NewGuid():N}.db");
|
||||
var sfStorage = new StoreAndForwardStorage($"Data Source={sfDb}",
|
||||
NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await sfStorage.InitializeAsync();
|
||||
var siteStorage = new SiteStorageService($"Data Source={siteDb}",
|
||||
NullLogger<SiteStorageService>.Instance);
|
||||
var replicationService = new ReplicationService(
|
||||
new StoreAndForwardOptions(), NullLogger<ReplicationService>.Instance);
|
||||
// Name MUST be "site-replication" — SendToPeer targets /user/site-replication.
|
||||
var actor = node.ActorOf(Props.Create(() => new SiteReplicationActor(
|
||||
siteStorage, sfStorage, replicationService, "site-int",
|
||||
NullLogger<SiteReplicationActor>.Instance, null, null, null, null)),
|
||||
"site-replication");
|
||||
return (sfStorage, actor);
|
||||
}
|
||||
|
||||
private static StoreAndForwardMessage NewMessage(string id) => new()
|
||||
{
|
||||
Id = id,
|
||||
Category = StoreAndForwardCategory.Notification,
|
||||
Target = "central",
|
||||
PayloadJson = "{}",
|
||||
CreatedAt = DateTimeOffset.UtcNow,
|
||||
Status = StoreAndForwardMessageStatus.Pending,
|
||||
MaxRetries = 0,
|
||||
};
|
||||
|
||||
private static async Task AwaitAsync(Func<Task<bool>> condition, TimeSpan timeout, string why)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (await condition()) return;
|
||||
await Task.Delay(250);
|
||||
}
|
||||
throw new TimeoutException(why);
|
||||
}
|
||||
}
|
||||
@@ -22,11 +22,12 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
|
||||
public int PortB { get; private set; }
|
||||
|
||||
public static async Task<TwoNodeClusterFixture> StartAsync(
|
||||
string role = "Central", TimeSpan? stableAfter = null)
|
||||
string role = "Central", TimeSpan? stableAfter = null,
|
||||
int? portA = null, int? portB = null)
|
||||
{
|
||||
var f = new TwoNodeClusterFixture();
|
||||
f.PortA = GetFreeTcpPort();
|
||||
f.PortB = GetFreeTcpPort();
|
||||
f.PortA = portA ?? GetFreeTcpPort();
|
||||
f.PortB = portB ?? GetFreeTcpPort();
|
||||
f.NodeA = f.StartNode(f.PortA, role, stableAfter);
|
||||
await WaitForMembersUp(f.NodeA, 1, TimeSpan.FromSeconds(20));
|
||||
f.NodeB = f.StartNode(f.PortB, role, stableAfter);
|
||||
@@ -98,7 +99,7 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
|
||||
throw new TimeoutException($"Member {removed} was never removed from {cluster.SelfAddress}'s view — SBR did not down it.");
|
||||
}
|
||||
|
||||
private static int GetFreeTcpPort()
|
||||
public static int GetFreeTcpPort()
|
||||
{
|
||||
var listener = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
|
||||
@@ -223,7 +223,8 @@ public class SiteAlarmStreamEndToEndTests : TestKit
|
||||
var sink = new PublishSink();
|
||||
var aggregator = Sys.ActorOf(Props.Create(() => new SiteAlarmAggregatorActor(
|
||||
"site-alpha", "e2e-parity", _ => Task.FromResult<IReadOnlyList<AlarmStateChanged>>(new[] { seedRow }),
|
||||
sink.Publish, factory, "http://a:5100", "http://b:5100", TimeSpan.FromMinutes(10))));
|
||||
sink.Publish, factory, "http://a:5100", "http://b:5100", TimeSpan.FromMinutes(10),
|
||||
TimeSpan.Zero, TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60))));
|
||||
|
||||
await WaitForConditionAsync(() => sink.Latest is { Count: 1 }); // seed applied
|
||||
aggregator.Tell(liveRow);
|
||||
|
||||
@@ -5,8 +5,10 @@ using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
using NSubstitute.ExceptionExtensions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Schemas;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.SecuredWrites;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts;
|
||||
@@ -3599,6 +3601,51 @@ public class ManagementActorTests : TestKit, IDisposable
|
||||
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
// ── R2 T13: site delete disposes the site's cached gRPC channels (N8) ──
|
||||
|
||||
[Fact]
|
||||
public void DeleteSite_DisposesTheSitesCachedGrpcChannels()
|
||||
{
|
||||
var siteRepo = Substitute.For<ISiteRepository>();
|
||||
siteRepo.GetSiteByIdAsync(7, Arg.Any<CancellationToken>())
|
||||
.Returns(new Site("Seven", "site-7") { Id = 7 });
|
||||
siteRepo.GetInstancesBySiteIdAsync(7, Arg.Any<CancellationToken>())
|
||||
.Returns((IReadOnlyList<Instance>)new List<Instance>());
|
||||
_services.AddScoped(_ => siteRepo);
|
||||
|
||||
var factory = new TrackingGrpcFactory();
|
||||
var cached = (TrackingClient)factory.GetOrCreate("site-7", "http://node-a:8083");
|
||||
_services.AddSingleton<SiteStreamGrpcClientFactory>(factory);
|
||||
|
||||
var actor = CreateActor();
|
||||
actor.Tell(Envelope(new DeleteSiteCommand(7), "Administrator"));
|
||||
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
|
||||
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
Assert.True(cached.Disposed);
|
||||
Assert.Null(factory.TryGet("site-7", "http://node-a:8083"));
|
||||
}, TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
private sealed class TrackingClient : SiteStreamGrpcClient
|
||||
{
|
||||
public TrackingClient(string endpoint) : base(endpoint) { }
|
||||
public bool Disposed { get; private set; }
|
||||
public override void Dispose() => Disposed = true;
|
||||
public override ValueTask DisposeAsync()
|
||||
{
|
||||
Disposed = true;
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class TrackingGrpcFactory : SiteStreamGrpcClientFactory
|
||||
{
|
||||
public TrackingGrpcFactory() : base(NullLoggerFactory.Instance) { }
|
||||
protected override SiteStreamGrpcClient CreateClient(string grpcEndpoint) => new TrackingClient(grpcEndpoint);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records remote-query relays for the actor-dispatch tests. The browse /
|
||||
/// search / verify / cert-trust methods on <see cref="CommunicationService"/>
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Diagnostics.Metrics;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.Extensions.Logging;
|
||||
@@ -9,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
|
||||
@@ -248,8 +250,10 @@ akka {
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ActiveNode_AnswersResyncRequest_WithFullBufferSnapshot()
|
||||
public async Task ActiveNode_AnswersResyncRequest_WithChunkedSnapshot()
|
||||
{
|
||||
// Post-R2-T5 the active node answers with byte-budgeted SfBufferSnapshotChunk(s)
|
||||
// (a single small row rides one chunk) rather than the monolithic SfBufferSnapshot.
|
||||
await _sfStorage.EnqueueAsync(NewSfMessage("m1"));
|
||||
var probe = CreateTestProbe();
|
||||
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
|
||||
@@ -258,9 +262,11 @@ akka {
|
||||
|
||||
actor.Tell(new RequestSfBufferResync(), TestActor);
|
||||
|
||||
var snapshot = ExpectMsg<SfBufferSnapshot>(TimeSpan.FromSeconds(3));
|
||||
Assert.Single(snapshot.Messages);
|
||||
Assert.False(snapshot.Truncated);
|
||||
var chunk = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(3));
|
||||
Assert.Equal(1, chunk.TotalChunks);
|
||||
Assert.Equal(1, chunk.Sequence);
|
||||
Assert.Single(chunk.Messages);
|
||||
Assert.False(chunk.Truncated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -281,6 +287,170 @@ akka {
|
||||
}, TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
// ── R2 T5: chunked resync answer ──
|
||||
|
||||
[Fact]
|
||||
public void ChunkForRemoting_SplitsByByteBudget_PreservingOrderAndSequence()
|
||||
{
|
||||
var rows = Enumerable.Range(0, 10)
|
||||
.Select(i => NewMessage($"m{i}", payloadJson: new string('x', 20_000)))
|
||||
.ToList();
|
||||
|
||||
var chunks = SiteReplicationActor.ChunkForRemoting(rows, maxChunkBytes: 64_000, maxChunkRows: 200);
|
||||
|
||||
Assert.True(chunks.Count > 1); // 10 × 20 KB cannot ride one 64 KB chunk
|
||||
Assert.Equal(rows.Select(r => r.Id), chunks.SelectMany(c => c).Select(r => r.Id)); // order preserved
|
||||
Assert.All(chunks, c => Assert.True(
|
||||
c.Sum(r => r.PayloadJson.Length) <= 64_000 || c.Count == 1)); // budget honored (oversized row isolated)
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ChunkForRemoting_RowCapHonored_AndSingleOversizedRowIsolated()
|
||||
{
|
||||
var many = Enumerable.Range(0, 500).Select(i => NewMessage($"s{i}", payloadJson: "{}")).ToList();
|
||||
Assert.All(SiteReplicationActor.ChunkForRemoting(many, 64_000, 200), c => Assert.True(c.Count <= 200));
|
||||
|
||||
var oversized = new List<StoreAndForwardMessage>
|
||||
{ NewMessage("big", payloadJson: new string('y', 100_000)), NewMessage("small", payloadJson: "{}") };
|
||||
var chunks = SiteReplicationActor.ChunkForRemoting(oversized, 64_000, 200);
|
||||
Assert.Equal(2, chunks.Count); // the oversized row rides alone
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ActiveNode_AnswersResyncRequest_WithSequencedChunks_SharingOneResyncId()
|
||||
{
|
||||
for (var i = 0; i < 3; i++)
|
||||
await _sfStorage.EnqueueAsync(NewMessage($"c{i}", payloadJson: new string('z', 30_000)));
|
||||
var actor = CreateResyncActor(isActive: () => true);
|
||||
|
||||
actor.Tell(new RequestSfBufferResync(), TestActor);
|
||||
|
||||
var first = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
|
||||
var rest = Enumerable.Range(1, first.TotalChunks - 1)
|
||||
.Select(_ => ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5)))
|
||||
.Prepend(first)
|
||||
.ToList();
|
||||
|
||||
Assert.True(first.TotalChunks > 1);
|
||||
Assert.All(rest, c => Assert.Equal(first.ResyncId, c.ResyncId));
|
||||
Assert.Equal(Enumerable.Range(1, first.TotalChunks), rest.Select(c => c.Sequence));
|
||||
Assert.Equal(3, rest.Sum(c => c.Messages.Count));
|
||||
}
|
||||
|
||||
// ── R2 T6: standby chunk assembly + atomic apply + ack ──
|
||||
|
||||
[Fact]
|
||||
public async Task StandbyNode_AssemblesChunks_AppliesOnce_AndAcks()
|
||||
{
|
||||
await _sfStorage.EnqueueAsync(NewMessage("stale"));
|
||||
var actor = CreateResyncActor(isActive: () => false);
|
||||
var resyncId = "r1";
|
||||
|
||||
actor.Tell(new SfBufferSnapshotChunk(resyncId, 1, 2,
|
||||
new List<StoreAndForwardMessage> { NewMessage("f1") }, false), TestActor);
|
||||
actor.Tell(new SfBufferSnapshotChunk(resyncId, 2, 2,
|
||||
new List<StoreAndForwardMessage> { NewMessage("f2") }, false), TestActor);
|
||||
|
||||
var ack = ExpectMsg<SfBufferResyncAck>(TimeSpan.FromSeconds(5));
|
||||
Assert.Equal(resyncId, ack.ResyncId);
|
||||
Assert.Equal(2, ack.RowCount);
|
||||
await AwaitAssertAsync(async () =>
|
||||
{
|
||||
Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); // replaced wholesale
|
||||
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f1"));
|
||||
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f2"));
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StandbyNode_NewResyncId_DiscardsStalePartialAssembly()
|
||||
{
|
||||
var actor = CreateResyncActor(isActive: () => false);
|
||||
actor.Tell(new SfBufferSnapshotChunk("old", 1, 2,
|
||||
new List<StoreAndForwardMessage> { NewMessage("orphan") }, false), TestActor);
|
||||
actor.Tell(new SfBufferSnapshotChunk("new", 1, 1,
|
||||
new List<StoreAndForwardMessage> { NewMessage("fresh") }, false), TestActor);
|
||||
|
||||
ExpectMsg<SfBufferResyncAck>(TimeSpan.FromSeconds(5)); // "new" completed
|
||||
await AwaitAssertAsync(async () =>
|
||||
{
|
||||
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh"));
|
||||
Assert.Null(await _sfStorage.GetMessageByIdAsync("orphan")); // stale partial never applied
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ActiveNode_IgnoresChunks_NeverAcks()
|
||||
{
|
||||
var actor = CreateResyncActor(isActive: () => true);
|
||||
actor.Tell(new SfBufferSnapshotChunk("r", 1, 1,
|
||||
new List<StoreAndForwardMessage> { NewMessage("x") }, false), TestActor);
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
||||
}
|
||||
|
||||
// ── R2 T7: active-side resync ack confirmation + telemetry ──
|
||||
//
|
||||
// NOTE (deviation from plan): the actor logs via Microsoft ILogger (NullLogger in
|
||||
// tests), NOT Akka's EventStream, so the plan's EventFilter.Warning assertions can
|
||||
// never observe these warnings. We observe the two OTel counters via a MeterListener
|
||||
// instead — the equivalent, and stronger, observable signal.
|
||||
|
||||
[Fact]
|
||||
public async Task ActiveNode_ReceivingAck_CountsResyncCompleted()
|
||||
{
|
||||
long completed = 0;
|
||||
using var listener = ListenCounter("scadabridge.store_and_forward.resync.completed",
|
||||
m => Interlocked.Add(ref completed, m));
|
||||
|
||||
await _sfStorage.EnqueueAsync(NewMessage("m1"));
|
||||
var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromSeconds(30));
|
||||
actor.Tell(new RequestSfBufferResync(), TestActor);
|
||||
var chunk = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
|
||||
|
||||
actor.Tell(new SfBufferResyncAck(chunk.ResyncId, 1), TestActor);
|
||||
|
||||
await AwaitAssertAsync(() =>
|
||||
{
|
||||
Assert.True(Interlocked.Read(ref completed) >= 1); // ack recorded the completion
|
||||
return Task.CompletedTask;
|
||||
}, TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ActiveNode_MissingAck_WarnsAfterAckTimeout()
|
||||
{
|
||||
long ackMissing = 0;
|
||||
using var listener = ListenCounter("scadabridge.store_and_forward.resync.ack_missing",
|
||||
m => Interlocked.Add(ref ackMissing, m));
|
||||
|
||||
await _sfStorage.EnqueueAsync(NewMessage("m1"));
|
||||
var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromMilliseconds(200));
|
||||
actor.Tell(new RequestSfBufferResync(), TestActor);
|
||||
ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
|
||||
// No ack is sent → the ack window expires and the resync is counted unacknowledged.
|
||||
|
||||
await AwaitAssertAsync(() =>
|
||||
{
|
||||
Assert.True(Interlocked.Read(ref ackMissing) >= 1);
|
||||
return Task.CompletedTask;
|
||||
}, TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
/// <summary>Attaches a <see cref="MeterListener"/> to a single ScadaBridge counter by name,
|
||||
/// forwarding each recorded increment to <paramref name="onMeasurement"/>.</summary>
|
||||
private static MeterListener ListenCounter(string instrumentName, Action<long> onMeasurement)
|
||||
{
|
||||
var listener = new MeterListener();
|
||||
listener.InstrumentPublished = (inst, l) =>
|
||||
{
|
||||
if (inst.Meter.Name == ScadaBridgeTelemetry.MeterName && inst.Name == instrumentName)
|
||||
l.EnableMeasurementEvents(inst);
|
||||
};
|
||||
listener.SetMeasurementEventCallback<long>((_, m, _, _) => onMeasurement(m));
|
||||
listener.Start();
|
||||
return listener;
|
||||
}
|
||||
|
||||
private static StoreAndForwardMessage NewSfMessage(string id) => new()
|
||||
{
|
||||
Id = id,
|
||||
@@ -294,6 +464,31 @@ akka {
|
||||
Status = StoreAndForwardMessageStatus.Pending,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Builds a resync-test message with a settable payload (additive to
|
||||
/// <see cref="NewSfMessage"/> — the chunker sizes on <c>PayloadJson</c> length).
|
||||
/// </summary>
|
||||
private static StoreAndForwardMessage NewMessage(string id, string payloadJson = "{}") => new()
|
||||
{
|
||||
Id = id,
|
||||
Category = StoreAndForwardCategory.ExternalSystem,
|
||||
Target = "t",
|
||||
PayloadJson = payloadJson,
|
||||
RetryCount = 0,
|
||||
MaxRetries = 50,
|
||||
RetryIntervalMs = 30000,
|
||||
CreatedAt = DateTimeOffset.UtcNow,
|
||||
Status = StoreAndForwardMessageStatus.Pending,
|
||||
};
|
||||
|
||||
/// <summary>Constructs a <see cref="ResyncTestActor"/> with the given active-node check
|
||||
/// (the resync chunk/ack tests Tell to and expect from <see cref="TestKit.TestActor"/>).
|
||||
/// <paramref name="ackTimeout"/> is the active-side ack window seam (T7).</summary>
|
||||
private IActorRef CreateResyncActor(Func<bool> isActive, TimeSpan? ackTimeout = null) =>
|
||||
ActorOf(Props.Create(() => new ResyncTestActor(
|
||||
_storage, _sfStorage, _replicationService, SiteRole,
|
||||
NullLogger<SiteReplicationActor>.Instance, CreateTestProbe().Ref, isActive, ackTimeout)));
|
||||
|
||||
/// <summary>Test message: drives <see cref="SiteReplicationActor.OnPeerTracked"/> directly,
|
||||
/// standing in for the MemberUp→TryTrackPeer path (a single-node TestKit cannot form a real peer).</summary>
|
||||
private sealed record TriggerPeerTracked;
|
||||
@@ -309,9 +504,10 @@ akka {
|
||||
public ResyncTestActor(
|
||||
SiteStorageService storage, StoreAndForwardStorage sfStorage,
|
||||
ReplicationService replicationService, string siteRole,
|
||||
ILogger<SiteReplicationActor> logger, IActorRef peerProbe, Func<bool> isActive)
|
||||
ILogger<SiteReplicationActor> logger, IActorRef peerProbe, Func<bool> isActive,
|
||||
TimeSpan? ackTimeout = null)
|
||||
: base(storage, sfStorage, replicationService, siteRole, logger,
|
||||
configFetcher: null, isActiveOverride: isActive)
|
||||
configFetcher: null, isActiveOverride: isActive, resyncAckTimeout: ackTimeout)
|
||||
{
|
||||
_peerProbe = peerProbe;
|
||||
Receive<TriggerPeerTracked>(_ => OnPeerTracked());
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
using Akka.TestKit.Xunit2;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Characterization pin for the chunked anti-entropy resync contract (review 02 round 2,
|
||||
/// N2). <see cref="SfBufferSnapshotChunk"/> (active→standby) and <see cref="SfBufferResyncAck"/>
|
||||
/// (standby→active) ride intra-site Akka remoting on the default reflective-JSON wire
|
||||
/// format. A rename/move, dropped setter, or non-default-constructible message would
|
||||
/// silently break resync across a rolling upgrade and only surface as a divergent buffer
|
||||
/// after a failover. These pin round-trip fidelity and type identity. (These messages are
|
||||
/// NOT ClusterClient traffic, so they are intentionally absent from ClusterClientContractLockTests.)
|
||||
/// </summary>
|
||||
public class ResyncWireSerializationPinTests : TestKit
|
||||
{
|
||||
private T RoundTrip<T>(T message)
|
||||
{
|
||||
var serialization = Sys.Serialization;
|
||||
var serializer = serialization.FindSerializerFor(message);
|
||||
var bytes = serializer.ToBinary(message);
|
||||
return (T)serialization.Deserialize(bytes, serializer.Identifier, message!.GetType());
|
||||
}
|
||||
|
||||
private static StoreAndForwardMessage FullMessage() => new()
|
||||
{
|
||||
Id = Guid.NewGuid().ToString("N"),
|
||||
Category = StoreAndForwardCategory.Notification,
|
||||
Target = "Operators",
|
||||
PayloadJson = "{\"notificationId\":\"abc\"}",
|
||||
RetryCount = 4,
|
||||
MaxRetries = 0,
|
||||
RetryIntervalMs = 30000,
|
||||
CreatedAt = DateTimeOffset.UtcNow,
|
||||
LastAttemptAt = DateTimeOffset.UtcNow,
|
||||
Status = StoreAndForwardMessageStatus.Parked,
|
||||
LastError = "central rejected",
|
||||
OriginInstanceName = "Plant.Pump3",
|
||||
ExecutionId = Guid.NewGuid(),
|
||||
SourceScript = "ScriptActor:MonitorSpeed",
|
||||
ParentExecutionId = Guid.NewGuid(),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void SfBufferSnapshotChunk_WithFullMessage_RoundTripsOnTheWire()
|
||||
{
|
||||
var message = FullMessage();
|
||||
var original = new SfBufferSnapshotChunk(
|
||||
"resync-1", 2, 5, new List<StoreAndForwardMessage> { message }, Truncated: true);
|
||||
|
||||
var back = RoundTrip(original);
|
||||
|
||||
Assert.Equal(original.ResyncId, back.ResyncId);
|
||||
Assert.Equal(original.Sequence, back.Sequence);
|
||||
Assert.Equal(original.TotalChunks, back.TotalChunks);
|
||||
Assert.Equal(original.Truncated, back.Truncated);
|
||||
var m = Assert.Single(back.Messages);
|
||||
Assert.Equal(message.Id, m.Id);
|
||||
Assert.Equal(message.Category, m.Category);
|
||||
Assert.Equal(message.Target, m.Target);
|
||||
Assert.Equal(message.PayloadJson, m.PayloadJson);
|
||||
Assert.Equal(message.RetryCount, m.RetryCount);
|
||||
Assert.Equal(message.MaxRetries, m.MaxRetries);
|
||||
Assert.Equal(message.RetryIntervalMs, m.RetryIntervalMs);
|
||||
Assert.Equal(message.CreatedAt, m.CreatedAt);
|
||||
Assert.Equal(message.LastAttemptAt, m.LastAttemptAt);
|
||||
Assert.Equal(message.Status, m.Status);
|
||||
Assert.Equal(message.LastError, m.LastError);
|
||||
Assert.Equal(message.OriginInstanceName, m.OriginInstanceName);
|
||||
Assert.Equal(message.ExecutionId, m.ExecutionId);
|
||||
Assert.Equal(message.SourceScript, m.SourceScript);
|
||||
Assert.Equal(message.ParentExecutionId, m.ParentExecutionId);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SfBufferResyncAck_RoundTripsOnTheWire()
|
||||
{
|
||||
var original = new SfBufferResyncAck("resync-1", 42);
|
||||
|
||||
var back = RoundTrip(original);
|
||||
|
||||
Assert.Equal(original.ResyncId, back.ResyncId);
|
||||
Assert.Equal(original.RowCount, back.RowCount);
|
||||
}
|
||||
|
||||
// Type-identity pins: the reflective-JSON wire embeds CLR type manifests, so a
|
||||
// rename/move of either type silently breaks resync across a rolling upgrade.
|
||||
[Theory]
|
||||
[InlineData(typeof(SfBufferSnapshotChunk), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferSnapshotChunk")]
|
||||
[InlineData(typeof(SfBufferResyncAck), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferResyncAck")]
|
||||
public void ResyncContract_TypeIdentity_IsPinned(Type type, string expectedFullName) =>
|
||||
Assert.Equal(expectedFullName, type.FullName);
|
||||
}
|
||||
+29
@@ -47,4 +47,33 @@ public class StoreAndForwardOptionsValidatorTests
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("SqliteDbPath", result.FailureMessage);
|
||||
}
|
||||
|
||||
// ── R2 T9: sweep-tuning eager validation (N4) ──
|
||||
|
||||
[Theory]
|
||||
[InlineData(-1)]
|
||||
[InlineData(-500)]
|
||||
public void Validate_NegativeSweepBatchLimit_Fails(int limit)
|
||||
{
|
||||
var result = Validate(new StoreAndForwardOptions { SweepBatchLimit = limit });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("SweepBatchLimit", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_ZeroSweepBatchLimit_IsValidUnlimitedLegacy()
|
||||
{
|
||||
var result = Validate(new StoreAndForwardOptions { SweepBatchLimit = 0 });
|
||||
Assert.True(result.Succeeded, result.FailureMessage);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(0)]
|
||||
[InlineData(-4)]
|
||||
public void Validate_NonPositiveSweepTargetParallelism_Fails(int parallelism)
|
||||
{
|
||||
var result = Validate(new StoreAndForwardOptions { SweepTargetParallelism = parallelism });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("SweepTargetParallelism", result.FailureMessage);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -821,4 +821,67 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable
|
||||
}
|
||||
finally { await service.StopAsync(); }
|
||||
}
|
||||
|
||||
// ── R2 T8: _sweepTask clobber (N3) ──
|
||||
|
||||
[Fact]
|
||||
public async Task TriggerSweep_WhileSweepInFlight_DoesNotClobberTheDrainHandle()
|
||||
{
|
||||
var service = CreateService(retryTimerInterval: TimeSpan.FromHours(1));
|
||||
var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, async _ =>
|
||||
{
|
||||
entered.TrySetResult();
|
||||
await release.Task;
|
||||
return true;
|
||||
});
|
||||
await service.StartAsync();
|
||||
try
|
||||
{
|
||||
await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "t", "{}",
|
||||
attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero);
|
||||
|
||||
service.TriggerSweep(); // real sweep, blocked in the handler
|
||||
await entered.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||
service.TriggerSweep(); // redundant kick — pre-fix clobbers _sweepTask
|
||||
|
||||
var handle = service.CurrentSweepTaskForTest;
|
||||
Assert.NotNull(handle);
|
||||
Assert.False(handle!.IsCompleted); // pre-fix: true (a completed no-op replaced the real sweep)
|
||||
}
|
||||
finally
|
||||
{
|
||||
release.TrySetResult();
|
||||
await service.StopAsync();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StopAsync_WaitsForTheRealInFlightSweep_EvenAfterARedundantTrigger()
|
||||
{
|
||||
var service = CreateService(retryTimerInterval: TimeSpan.FromHours(1));
|
||||
var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, async _ =>
|
||||
{
|
||||
entered.TrySetResult();
|
||||
await release.Task;
|
||||
return true;
|
||||
});
|
||||
await service.StartAsync();
|
||||
await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "t", "{}",
|
||||
attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero);
|
||||
|
||||
service.TriggerSweep();
|
||||
await entered.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||
service.TriggerSweep(); // the clobbering kick
|
||||
|
||||
var stop = service.StopAsync();
|
||||
await Task.Delay(300);
|
||||
Assert.False(stop.IsCompleted); // pre-fix: StopAsync already returned (awaited the no-op)
|
||||
|
||||
release.TrySetResult();
|
||||
await stop.WaitAsync(TimeSpan.FromSeconds(5)); // drains the real sweep promptly once released
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user