feat(events): distinguish ResourceExhausted causes with an mxgw-reason trailer (mxaccessgw#216)

Event-queue overflow, alarm-feed overflow, subscriber cap, session limit and
oversized commands all surfaced as ResourceExhausted. The status code is
unchanged; the gateway now attaches a machine-readable `mxgw-reason` trailer
(constants in Contracts ResourceExhaustedReasons). The .NET client maps
ResourceExhausted to MxGatewayResourceExhaustedException (an
MxGatewayException subclass; additive) exposing Reason/IsOverflow/
IsSubscriberLimit. Documented in docs/ClientBehaviorFixtures.md; the other
clients get a doc note only.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:02:24 -05:00
parent 716c845fc7
commit fccc21ba8e
9 changed files with 253 additions and 5 deletions
+28
View File
@@ -137,6 +137,34 @@ covers only what each client does on its own side of the wire.
| Python | none; `Session.stream_events` is an async iterator over the gRPC call | n/a | gRPC flow control applies back-pressure to the server | the same stream |
| Rust | none; `EventStream` is a boxed `Stream` over the tonic response (the design requires any helper channel to be bounded) | n/a | gRPC flow control applies back-pressure to the server | the same stream |
### ResourceExhausted Reason Trailer
The gateway uses gRPC `ResourceExhausted` for several different conditions. The status
code is unchanged, but every such failure now also carries a machine-readable reason in
the `mxgw-reason` response trailer (ASCII value), so a client can tell "you fell behind and
events were lost" (re-subscribe and resync) from "a limit was reached" (back off, do not
retry immediately):
| `mxgw-reason` | Meaning | Client action |
|---|---|---|
| `event-queue-overflow` | The `StreamEvents` subscriber fell behind its bounded queue and was disconnected; events were lost. | Reopen with the last seen `afterWorkerSequence` and resync. |
| `alarm-feed-overflow` | The `StreamAlarms` subscriber fell behind and was dropped. | Reconnect and re-snapshot. |
| `subscriber-limit` | The per-session event-subscriber cap was reached. | Back off. |
| `subscriber-already-active` | Only one subscriber is allowed and one is attached. | Back off. |
| `session-limit` | The gateway session limit was reached. | Back off. |
| `command-too-large` | A command exceeded the maximum frame size. | Do not retry unchanged. |
A missing trailer (an older gateway, or the API-key failure throttle's `ResourceExhausted`)
means "reason unknown"; treat it as the generic status. Constants live in
`ZB.MOM.WW.MxGateway.Contracts.ResourceExhaustedReasons`.
- **.NET** (additive): `ResourceExhausted` now surfaces as `MxGatewayResourceExhaustedException`
(a subclass of `MxGatewayException`, so existing catches keep working) with `Reason`,
`IsOverflow` and `IsSubscriberLimit`.
- **Go, Java, Python, Rust**: no typed surface yet (doc note only). Read the `mxgw-reason`
trailer from the failing call's trailing metadata (`status.Trailer()` / `Metadata` /
`trailing_metadata()` / tonic `Status::metadata()`).
After a slow-consumer cancel or an `EventQueueOverflow` disconnect, reopen with
the last seen `afterWorkerSequence`; if it predates the replay ring the stream
opens with the replay-gap sentinel described in `Event Streams` above.