feat(gateway,worker): configurable worker watchdog grace; heartbeat settings in effective config and Settings page (mxaccessgw#224)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m24s
ci / java (push) Successful in 2m15s
ci / portable (push) Failing after 4m17s

- New MxGateway:Worker:WorkerHeartbeatGraceSeconds (default 15), exported as
  MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS and read by WorkerPipeSessionOptions.FromEnvironment
  (a grace the ceiling does not exceed falls back to the default so the worker
  still starts). Validation now checks WorkerHeartbeatStuckCeilingSeconds against
  the configured grace; the copied WorkerWatchdogGraceSeconds constant is gone.
- EffectiveWorkerConfiguration and the dashboard Settings page now show the worker
  heartbeat send interval, watchdog grace and stuck ceiling (render test added).
- Docs updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:29:10 -05:00
parent 381e772766
commit 22964e192b
16 changed files with 141 additions and 19 deletions
+2 -1
View File
@@ -155,7 +155,8 @@ launch CWD (SEC-01, SEC-33).
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`, and strictly greater than `WorkerHeartbeatSendIntervalSeconds` — a grace at or below the worker's send interval declares a healthy worker dead between two heartbeats (mxaccessgw#218). |
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog has its separate ceiling, `WorkerHeartbeatStuckCeilingSeconds`. |
| `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` | `5` | Interval at which the worker sends heartbeats to the gateway. Must be between `1` and `86400`, and below `HeartbeatGraceSeconds`. Conveyed to the worker via `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS`; a missing or invalid value leaves the worker on 5 s. |
| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than the worker's fixed 15-second watchdog grace and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. |
| `MxGateway:Worker:WorkerHeartbeatGraceSeconds` | `15` | The worker's own STA-watchdog grace: how long STA activity may go stale before the worker faults `StaHung` (no command in flight). Must be between `1` and `86400` and below `WorkerHeartbeatStuckCeilingSeconds`; validation uses this configured value, not a copied constant (mxaccessgw#224). Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS`; a missing or invalid value, or a grace the ceiling does not exceed, leaves the worker on 15 s. |
| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than `WorkerHeartbeatGraceSeconds` and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. |
| `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. |
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. |
+1
View File
@@ -102,6 +102,7 @@ MXGATEWAY_ALARM_POLL_INTERVAL_MS=<milliseconds>
MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH=<alarms>
MXGATEWAY_ALARM_ACK_CLIENT_NAME=<client name>
MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS=<milliseconds>
MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS=<milliseconds>
MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS=<milliseconds>
```
+2 -1
View File
@@ -40,7 +40,8 @@ worker. The launcher always exports every value below:
| `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms |
| `MXGATEWAY_ALARM_ACK_CLIENT_NAME` | `MxGateway:Alarms:AckClientName` | `MxGateway` |
| `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS` | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` (× 1000) | `5000` ms |
| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's 15 s watchdog grace) |
| `MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS` | `MxGateway:Worker:WorkerHeartbeatGraceSeconds` (× 1000) | `15000` ms (also kept when the ceiling is not above it) |
| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's watchdog grace) |
The write-completion wait applies to the four single-item write variants:
`Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM