Files
ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs
T
Joseph Doherty 9b5cb3dd9d feat(comm): batch site→central stream events over gRPC (R2)
Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.

Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
  InstanceStreamRequest.batching_supported = 3
  SiteStreamRequest.batching_supported     = 2
  SiteStreamEvent.batch                    = 4  (new oneof case)
  SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }

The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.

Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).

Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.

Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.

Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.

Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
2026-08-15 03:39:59 -04:00

262 lines
14 KiB
C#

namespace ZB.MOM.WW.ScadaBridge.Communication;
/// <summary>
/// Configuration options for central-site communication, including per-pattern
/// timeouts and transport heartbeat settings.
/// </summary>
public class CommunicationOptions
{
/// <summary>
/// Central control-plane gRPC endpoints (preferred first), e.g.
/// <c>["http://scadabridge-central-a:8083", "http://scadabridge-central-b:8083"]</c>. Dialled by
/// <see cref="Grpc.CentralChannelProvider"/> with sticky failover/failback. Required on every site
/// node — gRPC (<c>CentralControlService</c>) is the only site→central transport.
/// </summary>
/// <remarks>
/// Sites reach central by container/host name, NOT via Traefik (which is HTTP/1 only; gRPC is
/// h2c on the central node's dedicated <c>CentralGrpcPort</c>).
/// </remarks>
public List<string> CentralGrpcEndpoints { get; set; } = new();
/// <summary>Timeout for deployment commands (typically longest due to apply logic).</summary>
public TimeSpan DeploymentTimeout { get; set; } = TimeSpan.FromMinutes(2);
/// <summary>Timeout for lifecycle commands (disable, enable, delete).</summary>
public TimeSpan LifecycleTimeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>Timeout for artifact deployment commands.</summary>
public TimeSpan ArtifactDeploymentTimeout { get; set; } = TimeSpan.FromMinutes(1);
/// <summary>Timeout for remote query requests (event logs, parked messages).</summary>
public TimeSpan QueryTimeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>Timeout for integration call routing.</summary>
public TimeSpan IntegrationTimeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>Timeout for debug view subscribe/unsubscribe handshake.</summary>
public TimeSpan DebugViewTimeout { get; set; } = TimeSpan.FromSeconds(10);
/// <summary>Timeout for health report acknowledgement (fire-and-forget, but bounded).</summary>
public TimeSpan HealthReportTimeout { get; set; } = TimeSpan.FromSeconds(10);
/// <summary>
/// Notification Outbox: timeout for forwarding a buffered notification to central
/// and awaiting its <c>NotificationSubmitAck</c>. A timeout is treated as a
/// transient failure — the Store-and-Forward engine keeps the message buffered
/// and retries the forward at the fixed retry interval.
/// </summary>
public TimeSpan NotificationForwardTimeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// Audit Log: timeout for the SITE-side Ask that forwards one audit-telemetry batch
/// (<c>IngestAuditEventsCommand</c> / <c>IngestCachedTelemetryCommand</c>) through the site's
/// <c>SiteCommunicationActor</c> and awaits central's ack. Deliberately the LONGEST rung of the
/// ingest timeout ladder.
/// </summary>
/// <remarks>
/// <para>
/// The ladder is strictly monotonic, outermost first:
/// <c>AuditForwardTimeout</c> (35 s, this option) &gt;
/// <c>SiteStreamGrpcServer.AuditIngestAskTimeout</c> (30 s — both the gRPC call deadline and
/// central's own Ask of the ingest singleton) &gt;
/// <c>AuditLogIngestActor.IngestBudget</c> (20 s) &gt;
/// <c>AuditLogIngestActor.IngestSqlCommandTimeout</c> (15 s).
/// </para>
/// <para>
/// Strictness matters: it used to reuse <see cref="NotificationForwardTimeout"/> (30 s), the
/// same value as the rung below it, so a slow-but-succeeding central write could be acked to a
/// caller whose Ask had already expired — the drain loop would treat the batch as unsent and
/// re-ship it. Central dedups on <c>EventId</c>, so the duplicate was harmless but the retry
/// traffic and the "stalled" telemetry signal were not. A timeout here is still transient: the
/// rows stay <c>Pending</c> and drain on the next tick.
/// </para>
/// <para>
/// <see cref="CommunicationOptionsValidator"/> enforces the outermost inequality — this value
/// must be strictly greater than the 30 s gRPC/central Ask rung.
/// </para>
/// </remarks>
public TimeSpan AuditForwardTimeout { get; set; } = TimeSpan.FromSeconds(35);
/// <summary>
/// Preshared key authenticating this node's gRPC control plane — the site↔central
/// boundary. On a site node this is the key its inbound gate
/// (<c>ControlPlaneAuthInterceptor</c>) expects on every <c>SiteStreamService</c> call, and
/// which central must present; central resolves the matching value per site from its own
/// secret store under the name <c>SB-GRPC-PSK-{siteId}</c>.
/// </summary>
/// <remarks>
/// <para>
/// In production this is supplied as <c>${secret:SB-GRPC-PSK-&lt;siteId&gt;}</c> and expanded
/// out of the secrets store before the host is built, so the plaintext never sits in
/// appsettings. Development rigs set a literal, mirroring the LocalDb replication key.
/// </para>
/// <para>
/// <b>Empty means closed, not open.</b> With no key set the interceptor rejects every gated
/// call. This is not optional configuration: a node that ships without a key serves no
/// streams and no audit pulls.
/// </para>
/// <para>
/// Distinct from <c>LocalDb:Replication:ApiKey</c>, which authenticates the pair partner for
/// database replication over the same listener. The two are never shared.
/// </para>
/// </remarks>
public string GrpcPsk { get; set; } = "";
/// <summary>
/// Central-side per-site gRPC preshared keys, keyed by site identifier — the mirror image
/// of <see cref="GrpcPsk"/>, which is the single key a site node expects on its own inbound
/// gate. An entry here takes precedence over the secret store for that site.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why both a config map and a secret store.</b> The store is the primary source and the
/// only one that works for the real case: sites are added at runtime from the Central UI, so
/// their keys cannot be enumerated in configuration at boot, and <c>SitePskProvider</c>
/// resolves <c>SB-GRPC-PSK-{siteId}</c> on demand. This map covers the cases the store
/// cannot or should not: a development rig that runs with no master key and injects every
/// credential as an environment override, and an operator pinning one site's key without
/// touching the store. Values may themselves be <c>${secret:…}</c> references, since a map
/// declared in configuration IS enumerable at boot.
/// </para>
/// <para>
/// Absence is not a fallback to "unauthenticated" in either source — a site with no key in
/// the map and none in the store cannot be dialed at all.
/// </para>
/// </remarks>
public Dictionary<string, string> SitePsks { get; set; } = new();
/// <summary>gRPC keepalive ping interval for streaming connections.</summary>
public TimeSpan GrpcKeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(15);
/// <summary>gRPC keepalive ping timeout — stream is considered dead if no response within this period.</summary>
public TimeSpan GrpcKeepAlivePingTimeout { get; set; } = TimeSpan.FromSeconds(10);
/// <summary>Maximum lifetime for a single gRPC stream before the server forces re-establishment.</summary>
public TimeSpan GrpcMaxStreamLifetime { get; set; } = TimeSpan.FromHours(4);
/// <summary>Maximum number of concurrent gRPC streaming subscriptions per site node.</summary>
public int GrpcMaxConcurrentStreams { get; set; } = 100;
/// <summary>
/// Send-channel capacity for a per-instance Debug View stream (<c>SubscribeInstance</c>).
/// Lossy by design: the Debug View is a diagnostic surface and drops its oldest events
/// under backpressure rather than stalling the site's event hub.
/// </summary>
public int GrpcInstanceStreamChannelCapacity { get; set; } = 1000;
/// <summary>
/// Send-channel capacity for the site-wide alarm stream (<c>SubscribeSite</c>). Larger
/// than the Debug View's (WP2.3): that feed backs the operator Alarm Summary, where a
/// silently dropped transition is a missed alarm rather than a missed diagnostic frame,
/// and an alarm burst arriving during a WAN stall must survive the stall.
/// </summary>
public int GrpcSiteAlarmStreamChannelCapacity { get; set; } = 20_000;
/// <summary>
/// Maximum events coalesced into one site→central stream frame (R2 — gRPC event
/// batching). At target scale a site emits ~37.5k events/s and every one of them used to
/// cost its own gRPC message; batching amortises that framing overhead. Set to 1 to
/// disable batching on this node without a wire change (every frame then carries exactly
/// one event, which is also what an un-negotiated subscription gets).
/// </summary>
public int GrpcStreamBatchMaxEvents { get; set; } = 100;
/// <summary>
/// Maximum time the site lingers accumulating a stream batch once a backlog has been
/// observed. Deliberately small: the target-scale load test measures end-to-end event
/// latency against a 250 ms P99 threshold (measured P99 4.57 ms), and this window is the
/// only latency batching can add — so it is validated strictly below that threshold.
/// A lone event on a quiet stream is never delayed by it (see
/// <c>SiteStreamEventBatcher</c>); the window applies only after a backlog is proven.
/// <see cref="TimeSpan.Zero"/> means "pack only what is already queued, never wait".
/// </summary>
public TimeSpan GrpcStreamBatchWindow { get; set; } = TimeSpan.FromMilliseconds(25);
/// <summary>Akka.Remote transport heartbeat interval.</summary>
public TimeSpan TransportHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5);
/// <summary>
/// Application-level site→central heartbeat cadence. Distinct from
/// <see cref="TransportHeartbeatInterval"/> (the Akka.Remote failure-detector
/// setting) — tuning the transport FD must not silently retune the health
/// heartbeat. Default equals the old effective value (5s) — no behavior change.
/// </summary>
public TimeSpan ApplicationHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5);
/// <summary>Akka.Remote transport failure detection threshold.</summary>
public TimeSpan TransportFailureThreshold { get; set; } = TimeSpan.FromSeconds(15);
/// <summary>
/// Base URL (Traefik/LB) the SITE uses to fetch deploy configs from central,
/// e.g. "https://central.example:9000". Carried in RefreshDeploymentCommand so
/// sites need no new standing config. Empty disables notify-and-fetch fallback.
/// </summary>
public string CentralFetchBaseUrl { get; set; } = "";
/// <summary>
/// How long a staged PendingDeployment (and its fetch token) stays valid. Must
/// comfortably cover both site nodes' fetches within one deploy window.
/// </summary>
public TimeSpan PendingDeploymentTtl { get; set; } = TimeSpan.FromMinutes(5);
/// <summary>
/// How often the central <c>PendingDeploymentPurgeActor</c> singleton reclaims
/// expired (TTL-elapsed) PendingDeployment staging rows. Best-effort hygiene only:
/// supersession bounds pending rows to ≤1 per instance and the config-fetch endpoint
/// already enforces the TTL, so this purge merely sweeps rows left behind by instances
/// that are deployed once and never re-deployed. Default 1 hour ≫ <see cref="PendingDeploymentTtl"/>.
/// </summary>
public TimeSpan PendingDeploymentPurgeInterval { get; set; } = TimeSpan.FromHours(1);
// ── Aggregated live alarm cache (plan #10, Task 4) ───────────────────────────
// Introduced by Task 4 with sane defaults; Task 6 formalizes eager validation
// (positive interval/linger, positive concurrency/subscriber cap) and telemetry.
/// <summary>
/// How long the shared per-site live alarm aggregator keeps its gRPC stream +
/// in-memory cache warm after its LAST Alarm Summary viewer leaves, before it is
/// torn down. A short linger avoids re-seed thrash when an operator navigates away
/// and straight back. Default 30s.
/// </summary>
public TimeSpan LiveAlarmCacheLinger { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// Cadence of the per-site aggregator's periodic reconcile snapshot fan-out. The
/// backstop that corrects instance-set drift (enable/disable/deploy/delete) and any
/// alarm delta missed by the live stream, since the aggregated stream never carries
/// an explicit "removed" event. Default 60s.
/// </summary>
public TimeSpan LiveAlarmCacheReconcileInterval { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>
/// Max concurrent per-instance snapshot fetches during a live-cache seed/reconcile
/// fan-out — the aggregated analogue of the Alarm Summary poll's
/// <c>MaxConcurrentFetches</c>. Default 8.
/// </summary>
public int LiveAlarmCacheSeedConcurrency { get; set; } = 8;
/// <summary>
/// Upper bound on concurrent Alarm Summary viewers sharing one site's aggregator.
/// A defensive cap only — one gRPC stream per site is shared regardless of viewer
/// count; this just bounds the subscriber list. Default 200.
/// </summary>
public int LiveAlarmCacheMaxSubscribersPerSite { get; set; } = 200;
/// <summary>
/// Publish-coalescing window for live alarm deltas: an applied delta marks the cache
/// dirty and one publish (fresh snapshot + per-viewer onChanged fan-out) fires after
/// this window, batching an alarm storm into ~4 publishes/second instead of one per
/// transition (review 02 round 2, N6). Zero = publish per delta (legacy). Seed and
/// reconcile publishes are always immediate. Default 250 ms.
/// </summary>
public TimeSpan LiveAlarmCachePublishCoalesce { get; set; } = TimeSpan.FromMilliseconds(250);
/// <summary>
/// Random jitter added to each per-site reconcile tick, as a fraction of
/// <see cref="LiveAlarmCacheReconcileInterval"/> (WP2.3). Without it every aggregator
/// started by one central failover fans its whole-site snapshot out on the same 60s
/// boundary forever. Zero disables jitter.
/// </summary>
public double LiveAlarmCacheReconcileJitterFraction { get; set; } = 0.2;
}