9b5cb3dd9d
Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.
Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
InstanceStreamRequest.batching_supported = 3
SiteStreamRequest.batching_supported = 2
SiteStreamEvent.batch = 4 (new oneof case)
SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }
The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.
Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).
Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.
Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.
Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.
Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
270 lines
10 KiB
C#
270 lines
10 KiB
C#
using Microsoft.Extensions.Options;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
|
|
|
|
/// <summary>
|
|
/// Regression: <see cref="CommunicationOptions"/> timeouts feed per-pattern Ask
|
|
/// deadlines and gRPC keepalive/stream-lifetime settings; a zero/negative value
|
|
/// (or a non-positive <c>GrpcMaxConcurrentStreams</c>) must be rejected at startup
|
|
/// by an <see cref="IValidateOptions{TOptions}"/> with a clear, key-naming message
|
|
/// rather than surfacing at first Ask/gRPC use.
|
|
/// </summary>
|
|
public class CommunicationOptionsValidatorTests
|
|
{
|
|
private static ValidateOptionsResult Validate(CommunicationOptions options) =>
|
|
new CommunicationOptionsValidator().Validate(Options.DefaultName, options);
|
|
|
|
[Fact]
|
|
public void DefaultOptions_AreValid()
|
|
{
|
|
var result = Validate(new CommunicationOptions());
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void ZeroDeploymentTimeout_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { DeploymentTimeout = TimeSpan.Zero });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("DeploymentTimeout", result.FailureMessage);
|
|
}
|
|
|
|
// ── Audit-ingest timeout ladder (arch-review phase-2 residual #2) ────────────
|
|
|
|
[Fact]
|
|
public void TheAuditIngestTimeoutLadder_IsStrictlyMonotonic_EndToEnd()
|
|
{
|
|
// 35 (site forward Ask) > 30 (gRPC deadline AND central's Ask of the ingest singleton)
|
|
// > 20 (actor budget) > 15 (SQL command). Ties are the bug this closes: the site Ask
|
|
// used to reuse NotificationForwardTimeout (30 s), so a slow-but-succeeding central
|
|
// write could be acked to a caller that had already given up and re-sent the batch.
|
|
var options = new CommunicationOptions();
|
|
|
|
Assert.Equal(TimeSpan.FromSeconds(35), options.AuditForwardTimeout);
|
|
Assert.True(options.AuditForwardTimeout > ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout);
|
|
Assert.Equal(TimeSpan.FromSeconds(30), ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout);
|
|
}
|
|
|
|
[Fact]
|
|
public void AuditForwardTimeout_EqualToTheAskTimeout_IsRejected()
|
|
{
|
|
// Equality is precisely the pre-fix state, so the validator must refuse it, not just
|
|
// refuse something smaller.
|
|
var result = Validate(new CommunicationOptions
|
|
{
|
|
AuditForwardTimeout = ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout,
|
|
});
|
|
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("AuditForwardTimeout", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void AuditForwardTimeout_ShorterThanTheAskTimeout_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions
|
|
{
|
|
AuditForwardTimeout = TimeSpan.FromSeconds(5),
|
|
});
|
|
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("AuditForwardTimeout", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void AuditForwardTimeout_IsStillConfigurableUpwards()
|
|
{
|
|
var result = Validate(new CommunicationOptions
|
|
{
|
|
AuditForwardTimeout = TimeSpan.FromMinutes(2),
|
|
});
|
|
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void NonPositiveGrpcMaxConcurrentStreams_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { GrpcMaxConcurrentStreams = 0 });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("GrpcMaxConcurrentStreams", result.FailureMessage);
|
|
}
|
|
|
|
// ── R2: site→central stream event batching ──────────────────────────────────
|
|
|
|
[Fact]
|
|
public void DefaultStreamBatchOptions_AreValid()
|
|
{
|
|
var options = new CommunicationOptions();
|
|
Assert.Equal(100, options.GrpcStreamBatchMaxEvents);
|
|
Assert.Equal(TimeSpan.FromMilliseconds(25), options.GrpcStreamBatchWindow);
|
|
Assert.True(Validate(options).Succeeded);
|
|
}
|
|
|
|
[Fact]
|
|
public void StreamBatchMaxEventsOfOne_IsValid_AndMeansBatchingDisabled()
|
|
{
|
|
var result = Validate(new CommunicationOptions { GrpcStreamBatchMaxEvents = 1 });
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void NonPositiveStreamBatchMaxEvents_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { GrpcStreamBatchMaxEvents = 0 });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("GrpcStreamBatchMaxEvents", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void ZeroStreamBatchWindow_IsValid()
|
|
{
|
|
// Zero = "pack only what is already queued, never wait" — a legitimate posture for
|
|
// a latency-critical deployment that still wants the framing saving.
|
|
var result = Validate(new CommunicationOptions { GrpcStreamBatchWindow = TimeSpan.Zero });
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void NegativeStreamBatchWindow_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions
|
|
{
|
|
GrpcStreamBatchWindow = TimeSpan.FromMilliseconds(-1)
|
|
});
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("GrpcStreamBatchWindow", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void StreamBatchWindowAtOrAboveTheLatencyBudget_IsRejected()
|
|
{
|
|
// The coalescing window is the only latency batching adds and the target-scale
|
|
// load test holds end-to-end stream latency to a 250 ms P99 — a window that could
|
|
// spend the whole budget must not boot.
|
|
foreach (var window in new[] { TimeSpan.FromMilliseconds(250), TimeSpan.FromSeconds(1) })
|
|
{
|
|
var result = Validate(new CommunicationOptions { GrpcStreamBatchWindow = window });
|
|
Assert.True(result.Failed, $"{window} was accepted");
|
|
Assert.Contains("GrpcStreamBatchWindow", result.FailureMessage);
|
|
}
|
|
|
|
// Just inside the ceiling is accepted — the bound is exclusive, not a round-down.
|
|
Assert.True(Validate(new CommunicationOptions
|
|
{
|
|
GrpcStreamBatchWindow = CommunicationOptionsValidator.StreamBatchWindowCeiling
|
|
- TimeSpan.FromMilliseconds(1)
|
|
}).Succeeded);
|
|
}
|
|
|
|
// ── Aggregated live alarm cache options (plan #10, Task 6) ───────────────────
|
|
|
|
[Fact]
|
|
public void ZeroLiveAlarmCacheLinger_IsValid()
|
|
{
|
|
// Zero linger = stop the aggregator immediately when the last viewer leaves.
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCacheLinger = TimeSpan.Zero });
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void NegativeLiveAlarmCacheLinger_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCacheLinger = TimeSpan.FromSeconds(-1) });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("LiveAlarmCacheLinger", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void ZeroLiveAlarmCacheReconcileInterval_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCacheReconcileInterval = TimeSpan.Zero });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("LiveAlarmCacheReconcileInterval", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void ZeroLiveAlarmCacheSeedConcurrency_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCacheSeedConcurrency = 0 });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("LiveAlarmCacheSeedConcurrency", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void ExcessiveLiveAlarmCacheSeedConcurrency_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCacheSeedConcurrency = 65 });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("LiveAlarmCacheSeedConcurrency", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void ZeroLiveAlarmCacheMaxSubscribersPerSite_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCacheMaxSubscribersPerSite = 0 });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("LiveAlarmCacheMaxSubscribersPerSite", result.FailureMessage);
|
|
}
|
|
|
|
// ── R2 T10: live-delta publish-coalescing window (N6) ────────────────────────
|
|
|
|
[Fact]
|
|
public void ZeroLiveAlarmCachePublishCoalesce_IsValid()
|
|
{
|
|
// Zero = publish per delta (legacy behavior).
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCachePublishCoalesce = TimeSpan.Zero });
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void NegativeLiveAlarmCachePublishCoalesce_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions { LiveAlarmCachePublishCoalesce = TimeSpan.FromMilliseconds(-1) });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("LiveAlarmCachePublishCoalesce", result.FailureMessage);
|
|
}
|
|
|
|
// ── gRPC central transport endpoints ─────────────────────────────────────────
|
|
// gRPC (CentralControlService) is the only site→central transport after the
|
|
// ClusterClient→gRPC migration's Phase 4. This role-agnostic validator only rejects BLANK
|
|
// entries; an EMPTY list is valid (central nodes legitimately declare none). The role-aware
|
|
// "a Site must list at least one endpoint" rule lives in StartupValidator, tested there.
|
|
|
|
[Fact]
|
|
public void EmptyEndpoints_IsValid()
|
|
{
|
|
// A central node hosts CentralControlService; it does not dial it, so it declares none.
|
|
var result = Validate(new CommunicationOptions
|
|
{
|
|
CentralGrpcEndpoints = new List<string>(),
|
|
});
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void BlankEndpoint_IsRejected()
|
|
{
|
|
var result = Validate(new CommunicationOptions
|
|
{
|
|
CentralGrpcEndpoints = new List<string> { " " },
|
|
});
|
|
Assert.True(result.Failed);
|
|
Assert.Contains("CentralGrpcEndpoints", result.FailureMessage);
|
|
}
|
|
|
|
[Fact]
|
|
public void Endpoints_IsValid()
|
|
{
|
|
var result = Validate(new CommunicationOptions
|
|
{
|
|
CentralGrpcEndpoints = new List<string>
|
|
{
|
|
"http://scadabridge-central-a:8083",
|
|
"http://scadabridge-central-b:8083",
|
|
},
|
|
});
|
|
Assert.True(result.Succeeded, result.FailureMessage);
|
|
}
|
|
}
|