fix(grpc): typed alarm-feed overflow, memoized alarm snapshot, cancellable stream writes, uniform reply correlation id
- StreamAlarms: slow subscribers are completed with AlarmFeedSubscriberOverflowException and surface as ResourceExhausted (like a StreamEvents overflow) instead of warning-with-stack + Unavailable. - GatewayAlarmMonitor.StreamAsync reuses the memoized alarm-set projection instead of cloning the whole set under the broadcast lock per attach; the prefix filter runs outside the lock. - StreamEvents/StreamAlarms pass the call token to response writes; a write failure after client cancel maps to Cancelled without a warning. - Every MxCommandReply.correlation_id now echoes the request's client_correlation_id (previously only constraint-denied replies did; normal replies leaked the internal envelope id).
This commit is contained in:
@@ -82,6 +82,8 @@ return mapper.MapCommandReply(workerReply);
|
||||
|
||||
Carrying the enqueue timestamp into the worker layer is what lets queue-wait time be measured separately from worker-side execution time when troubleshooting timeouts.
|
||||
|
||||
Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies.
|
||||
|
||||
An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.
|
||||
|
||||
Selected control commands have additional public behavior beyond their protobuf shape:
|
||||
@@ -94,6 +96,8 @@ Selected control commands have additional public behavior beyond their protobuf
|
||||
|
||||
`StreamEvents` is a server-streaming RPC. The handler delegates the full pipeline to `IEventStreamService` and just forwards each `MxEvent` onto the response stream. Keeping the channel and producer/consumer machinery out of the handler means cancellation, exception mapping, and metric bookkeeping live in one place.
|
||||
|
||||
Each response write receives the call's cancellation token, so a write parked on a slow or vanished client is released when the call is cancelled rather than waiting on the transport. A write that then fails with an `IOException` or `OperationCanceledException` while the call token is cancelled is a routine client disconnect: it maps to `Cancelled` and logs nothing. The same `IOException` on a live call keeps the generic `Unavailable` + warning mapping, because that is a real transport failure. `StreamAlarms` applies the same rule.
|
||||
|
||||
### `AcknowledgeAlarm`
|
||||
|
||||
`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` and `operator_user` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
|
||||
@@ -125,6 +129,8 @@ The worker calls `AuthenticateUser` as the operator, then writes `operator_comme
|
||||
|
||||
`StreamAlarms` is a server-streaming, **session-less** RPC that attaches to the gateway's central alarm feed. The handler delegates to `IGatewayAlarmService.StreamAsync`. The stream opens with a `provider_status` and a `snapshot_status` `AlarmFeedMessage` (the current provider mode and snapshot-completeness verdict), then one `AlarmFeedMessage` carrying an `active_alarm` per currently-active alarm (the ConditionRefresh snapshot), then a single `snapshot_complete`, then a `transition` for every subsequent raise / acknowledge / clear — interleaved with a further `provider_status` on each failover/failback and a further `snapshot_status` on each change of the truncation verdict. It is served by the always-on `GatewayAlarmMonitor`, which owns a single gateway-managed worker session and fans out to every attached client — clients no longer open a session of their own. `alarm_filter_prefix`, when set, scopes the stream to a sub-tree.
|
||||
|
||||
A subscriber's opening snapshot is taken from the monitor's memoized alarm-set projection (the same one `QueryActiveAlarms` and the dashboard read), captured under the broadcast lock in the same critical section that registers the subscriber, so no transition can fall between the snapshot and the live stream; the prefix filter runs afterwards, outside the lock. Each subscriber has a bounded queue (2048 messages). A subscriber that falls that far behind is dropped and its stream ends with `ResourceExhausted` ("Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."), the same status a `StreamEvents` overflow carries; the remedy is to reconnect, which re-sends the snapshot. The drop is not logged as a gateway failure.
|
||||
|
||||
### `QueryActiveAlarms`
|
||||
|
||||
`QueryActiveAlarms` is a server-streaming, **session-less** RPC that returns a point-in-time snapshot of the currently-active alarm set. Unlike `StreamAlarms`, it does not stay attached for live transitions — it writes the snapshot and completes, which clients use after a reconnect to reseed alarm state or to reconcile alarms that may have been missed during a transport blip. It is served from the always-on `GatewayAlarmMonitor` cache (`IGatewayAlarmService.CurrentAlarms`), so no worker session is opened. The handler does not run through `MxAccessGrpcRequestValidator`; it only null-checks the request inline, then streams one `ActiveAlarmSnapshot` per cached alarm, honouring cancellation between writes. When `alarm_filter_prefix` is non-empty, only alarms whose `alarm_full_reference` starts with that prefix (ordinal `StartsWith`) are emitted; an empty prefix returns the full set. Exceptions are translated by the same `MapException` path as the other handlers. The RPC requires the `events:read` scope (`GatewayScopes.EventsRead`), the same scope as `StreamEvents` and `StreamAlarms`.
|
||||
@@ -294,6 +300,8 @@ Every handler wraps its body in `try { ... } catch (Exception exception) when (e
|
||||
- `SessionManagerException` is mapped by its `ErrorCode`.
|
||||
- `WorkerClientException` is mapped by its `ErrorCode`.
|
||||
|
||||
Two stream-specific cases are handled before `MapException`: on `StreamEvents` and `StreamAlarms` a write failure after the client cancelled becomes `Cancelled` without a log entry, and on `StreamAlarms` an `AlarmFeedSubscriberOverflowException` (slow-subscriber drop) becomes `ResourceExhausted` without a log entry.
|
||||
|
||||
Anything else is logged at warning and surfaced as `Unavailable` with a generic message — clients see a retryable status, and the unexpected exception is captured in gateway logs rather than leaked over the wire.
|
||||
|
||||
```csharp
|
||||
|
||||
Reference in New Issue
Block a user