fix(gateway): split the heartbeat check interval from the send interval; expose the two fixed worker-client timeouts (mxaccessgw#192)
R21-O14: MxGateway:Worker:HeartbeatIntervalSeconds was documented as the worker heartbeat send interval, but it never reached the worker (which sends on its own fixed 5 s) and only set the gateway's HeartbeatCheckInterval. It is renamed HeartbeatCheckIntervalSeconds (default 5, unchanged). The old key stays as a deprecated alias with the same meaning so existing appsettings keep working; setting both is a validation error. The grace-vs-interval check now compares against the resolved check interval. R21-O15: WorkerClientOptions.EventChannelFullModeTimeout and HeartbeatStuckCeiling were always their compile-time defaults. They are now MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds (default 5000, > 0) and HeartbeatStuckCeilingSeconds (default 75, >= HeartbeatGraceSeconds). The factory's mapping moved into SessionWorkerClientFactory.CreateWorkerClientOptions so every field is set from configuration and is testable. The effective-config record shows the resolved check interval and both new values. Docs: GatewayConfiguration.md, GatewayProcessDesign.md sample, MxAccessWorkerInstanceDesign.md heartbeat section; appsettings.json uses the new keys. Tests: CreateWorkerClientOptions mapping + defaults (fail under a mutated mapping), alias honoured, both-keys conflict, and bounds for the new options. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -24,8 +24,10 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
|
||||
"StartupTimeoutSeconds": 30,
|
||||
"PipeConnectAttemptTimeoutMilliseconds": 2000,
|
||||
"ShutdownTimeoutSeconds": 10,
|
||||
"HeartbeatIntervalSeconds": 5,
|
||||
"HeartbeatCheckIntervalSeconds": 5,
|
||||
"HeartbeatGraceSeconds": 15,
|
||||
"HeartbeatStuckCeilingSeconds": 75,
|
||||
"EventChannelFullModeTimeoutMilliseconds": 5000,
|
||||
"MaxMessageBytes": 16842752
|
||||
},
|
||||
"Sessions": {
|
||||
@@ -145,8 +147,11 @@ launch CWD (SEC-01, SEC-33).
|
||||
| `MxGateway:Worker:WriteCompletionWaitMilliseconds` | `1500` | Bounded wait the worker holds a unary write reply (`Write`/`Write2`/`WriteSecured`/`WriteSecured2`; bulk writes excluded) for the matching MXAccess `OnWriteComplete` callback, so the reply's `statuses` carry the real commit outcome. `0` disables the wait (pure fire-and-forget replies). Must be `>= 0`. The gateway conveys the value to the worker via the `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` environment variable. Consumers that time their own writes must budget above this wait: OtOpcUa's GalaxyDriver wraps gateway writes in a 2 s Tier A resilience timeout, so a deployment raising this option past ~2000 must raise that driver `ResilienceConfig` write timeout in step or slow-but-successful commits surface as consumer-side failures. |
|
||||
| `MxGateway:Worker:EventQueueCapacity` | `10000` | Capacity, in events, of the worker's outbound MXAccess event queue. Must be between `1000` and `1000000`. This is burst headroom, not a throttle: the queue has no drop policy, so filling it records a `QueueOverflow` worker fault and faults the session. Raise it for sessions whose subscription set can outrun the drain loop (large advise sets, slow event consumers); the backing queue pre-allocates its slots, so the ceiling keeps a mistyped value from committing the 32-bit worker to an outsized allocation. The gateway conveys the value to the worker via the `MXGATEWAY_EVENT_QUEUE_CAPACITY` environment variable; a missing or unusable value leaves the worker on the 10000 default rather than failing the session. |
|
||||
| `MxGateway:Worker:ShutdownTimeoutSeconds` | `10` | Grace period for worker shutdown before the gateway treats shutdown as failed and may kill the worker process tree. |
|
||||
| `MxGateway:Worker:HeartbeatIntervalSeconds` | `5` | Worker heartbeat send interval and gateway heartbeat check cadence input. |
|
||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatIntervalSeconds`. |
|
||||
| `MxGateway:Worker:HeartbeatCheckIntervalSeconds` | `5` | How often the gateway checks the age of the worker's last heartbeat against `HeartbeatGraceSeconds`. This is the gateway-side check cadence only; the worker sends heartbeats on its own fixed 5-second interval, which no gateway option reaches. Must be greater than zero. |
|
||||
| `MxGateway:Worker:HeartbeatIntervalSeconds` | *(unset)* | **Deprecated** alias of `HeartbeatCheckIntervalSeconds` (mxaccessgw#192). The old name implied the worker's send interval, but the value only ever set the gateway's check cadence, which is what it still sets. Setting both keys fails validation. |
|
||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`. |
|
||||
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog keeps its separate, fixed 75-second ceiling. |
|
||||
| `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. |
|
||||
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. |
|
||||
|
||||
`PipeConnectAttemptTimeoutMilliseconds`, timeout values, heartbeat values, and
|
||||
|
||||
Reference in New Issue
Block a user