feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck ceiling (75 s) were compile-time defaults the gateway never reached, and nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send interval, so a misconfigured grace could kill healthy workers. - New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5) and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other worker settings use. - The worker's production WorkerPipeSession reads them through WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a ceiling not above the worker's 15 s watchdog grace, keeps the default rather than failing the worker. - Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds; send interval > 0; worker ceiling > 15 s; both at most 86400 s. - Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -27,6 +27,8 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
|
||||
"HeartbeatCheckIntervalSeconds": 5,
|
||||
"HeartbeatGraceSeconds": 15,
|
||||
"HeartbeatStuckCeilingSeconds": 75,
|
||||
"WorkerHeartbeatSendIntervalSeconds": 5,
|
||||
"WorkerHeartbeatStuckCeilingSeconds": 75,
|
||||
"EventChannelFullModeTimeoutMilliseconds": 5000,
|
||||
"MaxMessageBytes": 16842752
|
||||
},
|
||||
@@ -148,10 +150,12 @@ launch CWD (SEC-01, SEC-33).
|
||||
| `MxGateway:Worker:WriteCompletionWaitMilliseconds` | `1500` | Bounded wait the worker holds a unary write reply (`Write`/`Write2`/`WriteSecured`/`WriteSecured2`; bulk writes excluded) for the matching MXAccess `OnWriteComplete` callback, so the reply's `statuses` carry the real commit outcome. `0` disables the wait (pure fire-and-forget replies). Must be `>= 0`. The gateway conveys the value to the worker via the `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` environment variable. Consumers that time their own writes must budget above this wait: OtOpcUa's GalaxyDriver wraps gateway writes in a 2 s Tier A resilience timeout, so a deployment raising this option past ~2000 must raise that driver `ResilienceConfig` write timeout in step or slow-but-successful commits surface as consumer-side failures. |
|
||||
| `MxGateway:Worker:EventQueueCapacity` | `10000` | Capacity, in events, of the worker's outbound MXAccess event queue. Must be between `1000` and `1000000`. This is burst headroom, not a throttle: the queue has no drop policy, so filling it records a `QueueOverflow` worker fault and faults the session. Raise it for sessions whose subscription set can outrun the drain loop (large advise sets, slow event consumers); the backing queue pre-allocates its slots, so the ceiling keeps a mistyped value from committing the 32-bit worker to an outsized allocation. The gateway conveys the value to the worker via the `MXGATEWAY_EVENT_QUEUE_CAPACITY` environment variable; a missing or unusable value leaves the worker on the 10000 default rather than failing the session. |
|
||||
| `MxGateway:Worker:ShutdownTimeoutSeconds` | `10` | Grace period for worker shutdown before the gateway treats shutdown as failed and may kill the worker process tree. |
|
||||
| `MxGateway:Worker:HeartbeatCheckIntervalSeconds` | `5` | How often the gateway checks the age of the worker's last heartbeat against `HeartbeatGraceSeconds`. This is the gateway-side check cadence only; the worker sends heartbeats on its own fixed 5-second interval, which no gateway option reaches. Must be greater than zero. |
|
||||
| `MxGateway:Worker:HeartbeatCheckIntervalSeconds` | `5` | How often the gateway checks the age of the worker's last heartbeat against `HeartbeatGraceSeconds`. This is the gateway-side check cadence only; the worker's send interval is `WorkerHeartbeatSendIntervalSeconds`. Must be greater than zero. |
|
||||
| `MxGateway:Worker:HeartbeatIntervalSeconds` | *(unset)* | **Deprecated** alias of `HeartbeatCheckIntervalSeconds` (mxaccessgw#192). The old name implied the worker's send interval, but the value only ever set the gateway's check cadence, which is what it still sets. Setting both keys fails validation. |
|
||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`. |
|
||||
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog keeps its separate, fixed 75-second ceiling. |
|
||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`, and strictly greater than `WorkerHeartbeatSendIntervalSeconds` — a grace at or below the worker's send interval declares a healthy worker dead between two heartbeats (mxaccessgw#218). |
|
||||
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog has its separate ceiling, `WorkerHeartbeatStuckCeilingSeconds`. |
|
||||
| `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` | `5` | Interval at which the worker sends heartbeats to the gateway. Must be between `1` and `86400`, and below `HeartbeatGraceSeconds`. Conveyed to the worker via `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS`; a missing or invalid value leaves the worker on 5 s. |
|
||||
| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than the worker's fixed 15-second watchdog grace and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. |
|
||||
| `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. |
|
||||
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user