Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.
Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
InstanceStreamRequest.batching_supported = 3
SiteStreamRequest.batching_supported = 2
SiteStreamEvent.batch = 4 (new oneof case)
SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }
The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.
Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).
Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.
Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.
Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.
Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
MES alarm-status API §6.4 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 task 1). MES needs a real AckDT for a triggered alarm, and the mirror
carried acked-vs-unacked but never WHEN. AckTime now rides the whole path:
DCL transition -> AlarmStateChanged -> gRPC AlarmStateUpdate -> site SQLite.
Stamping rule, identical on both protocols: non-null ONLY while the condition
is active AND acknowledged. That single predicate yields all three required
behaviours -- null while unacked, cleared on re-raise (a re-raise arrives
unacknowledged), and no phantom ack on a return-to-normal. The last one is
load-bearing for MxGateway, which maps INACTIVE to Acknowledged = true; without
the active check every clear would claim an ack the system never observed.
Provenance is honest, never fabricated:
- OPC UA A&C supplies a TRUE ack instant, so we now select it:
AcknowledgeableConditionType/AckedState/TransitionTime at SelectClause
index 18, APPENDED so the positional reads at 0-17 keep their meaning.
Servers that omit the field fall back to the event's own Time.
- MxAccess Gateway supplies none, so the ack transition's own timestamp is
used -- accurate to when the system SAW the ack. An ACTIVE_ACKED
re-subscribe snapshot restores one from LastTransitionTimestamp rather
than dropping it.
The decision lives in pure mappers (Opc/Mx AlarmMapper.DeriveAckTime), so it is
unit-tested with no live server or gateway.
Additive-only throughout: init-only property on AlarmStateChanged, trailing
optional positional on NativeAlarmTransition (all 14-arg call sites untouched),
proto field 24 (never reusing a number) regenerated via docker/regen-proto.sh
sitestream with the csproj diff verified empty.
Persistence rides native_alarm_state's existing metadata_json blob rather than
a new column -- deliberately. That table is RegisterReplicated in
SiteLocalDbSetup and LocalDb builds its CDC triggers from the column list at
registration time, so an additive JSON property changes no schema, no triggers
and no replication contract; metadata_json is exactly the extension point UA4
introduced for this. Rows written before the field deserialize it as null.
Tests: 4 OPC UA + 6 MxGateway mapper cases, 3 NativeAlarmActor (emit,
failover rehydrate, pre-AckTime row), 1 proto round-trip incl. the null case,
4 Commons additive/back-compat. The OPC UA SelectClause count lock-in moves
18 -> 19 with an index-18 assertion -- intended, the clause is appended, which
is precisely what that guard exists to make visible.
Adds an additive, site-wide, alarm-only gRPC stream backing the aggregated
Alarm Summary. Proto: new SiteStreamRequest { correlation_id } message + rpc
SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent) on
SiteStreamService — purely additive, no field renumbering. Regenerated the
checked-in SiteStreamGrpc/*.cs.
Server: SubscribeInstance and the new SubscribeSite now delegate to a shared
RunSubscriptionStreamAsync helper (readiness/shutdown guards, correlation-id
validation, duplicate replacement, concurrency cap, bounded DropOldest channel,
relay actor, SiteConnectionOpened/Closed telemetry, guaranteed cleanup). The
only variation is the subscribe delegate: SubscribeSite calls
ISiteStreamSubscriber.SubscribeSiteAlarms (no per-instance filter). Added
SubscribeSiteAlarms to the ISiteStreamSubscriber contract (SiteStreamManager
already implements it from T1). StreamRelayActor reused unchanged — it already
drops IsConfiguredPlaceholder rows and maps the enriched AlarmStateUpdate.
Tests: SubscribeSite subscribes site alarms + removes on cancel, rejects unsafe
correlation ids, and relays a domain AlarmStateChanged as a proto
AlarmStateUpdate on the stream.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Site side: additive proto after_id field, ReadChangedSinceAsync gains an optional
afterId cursor and deterministic (UpdatedAtUtc, TrackedOperationId) ordering — a
batch fully inside one UpdatedAtUtc instant no longer re-reads the same page
forever. Absent/empty afterId preserves the exact legacy inclusive >= contract, so
an older central is unaffected. Regenerated the checked-in gRPC C#.
Add two additive init-only fields to AlarmStateChanged so the Debug View can
nest live native conditions under their configured source-binding node:
- NativeSourceCanonicalName (binding canonical name, e.g. "Motor1.MotorAlarms")
- IsConfiguredPlaceholder (quiet-binding placeholder flag; default false)
Flow on BOTH cross-process paths:
- Live: proto AlarmStateUpdate fields 22/23 -> StreamRelayActor packs ->
SiteStreamGrpcClient unpacks (regenerated SiteStreamGrpc/Sitestream.cs).
- Snapshot (Newtonsoft): record defaults carry through; no special handling.
NativeAlarmActor.Emit now stamps NativeSourceCanonicalName = _source.CanonicalName.
Additive-only: no existing positional constructor or wire frame changed.
Tests: StreamRelayActorTests round-trips both fields pack->unpack;
NativeAlarmActorTests asserts the emitted event carries the binding canonical name.
Site Call Audit (#22): build the documented periodic reconciliation PULL
self-heal path for the eventually-consistent central SiteCalls mirror, as a
dedicated PullSiteCalls gRPC RPC kept separate from the audit pull. This is the
pull PLUMBING only; the central reconciliation tick is a separate follow-up.
- IOperationTrackingStore.ReadChangedSinceAsync(sinceUtc, batchSize): inclusive
UpdatedAtUtc cursor, oldest-first, batch-capped; SQLite impl projects tracking
rows onto SiteCallOperational (Kind->Channel, TargetSummary->Target, SourceSite
left empty - the store has no site-id column).
- sitestream.proto: rpc PullSiteCalls + PullSiteCallsRequest/Response, mirroring
PullAuditEvents; regenerated checked-in SiteStreamGrpc/*.cs.
- SiteCallDtoMapper.ToDto(SiteCallOperational): inverse of FromDto for the handler.
- SiteStreamGrpcServer.PullSiteCalls handler + SetOperationTrackingStore seam;
Host wires the seam alongside SetSiteAuditQueue (site roles only).
- Central IPullSiteCallsClient + GrpcPullSiteCallsClient (home: AuditLog/Central to
reuse ISiteEnumerator; SiteCallAudit does not reference AuditLog). Re-stamps
SourceSite from the dialed siteId; no-throw on tolerable transport faults;
SpecifyKind (not ToUniversalTime) cursor handling. Central-only DI registration.
Tests: ReadChangedSinceAsync (4), PullSiteCalls handler (6), GrpcPullSiteCallsClient
(8). Full solution build 0 warnings/0 errors (TreatWarningsAsErrors).
Joseph Doherty