feat(gateway,worker): configurable worker watchdog grace; heartbeat settings in effective config and Settings page (mxaccessgw#224)
- New MxGateway:Worker:WorkerHeartbeatGraceSeconds (default 15), exported as MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS and read by WorkerPipeSessionOptions.FromEnvironment (a grace the ceiling does not exceed falls back to the default so the worker still starts). Validation now checks WorkerHeartbeatStuckCeilingSeconds against the configured grace; the copied WorkerWatchdogGraceSeconds constant is gone. - EffectiveWorkerConfiguration and the dashboard Settings page now show the worker heartbeat send interval, watchdog grace and stuck ceiling (render test added). - Docs updated. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -155,7 +155,8 @@ launch CWD (SEC-01, SEC-33).
|
|||||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`, and strictly greater than `WorkerHeartbeatSendIntervalSeconds` — a grace at or below the worker's send interval declares a healthy worker dead between two heartbeats (mxaccessgw#218). |
|
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`, and strictly greater than `WorkerHeartbeatSendIntervalSeconds` — a grace at or below the worker's send interval declares a healthy worker dead between two heartbeats (mxaccessgw#218). |
|
||||||
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog has its separate ceiling, `WorkerHeartbeatStuckCeilingSeconds`. |
|
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog has its separate ceiling, `WorkerHeartbeatStuckCeilingSeconds`. |
|
||||||
| `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` | `5` | Interval at which the worker sends heartbeats to the gateway. Must be between `1` and `86400`, and below `HeartbeatGraceSeconds`. Conveyed to the worker via `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS`; a missing or invalid value leaves the worker on 5 s. |
|
| `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` | `5` | Interval at which the worker sends heartbeats to the gateway. Must be between `1` and `86400`, and below `HeartbeatGraceSeconds`. Conveyed to the worker via `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS`; a missing or invalid value leaves the worker on 5 s. |
|
||||||
| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than the worker's fixed 15-second watchdog grace and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. |
|
| `MxGateway:Worker:WorkerHeartbeatGraceSeconds` | `15` | The worker's own STA-watchdog grace: how long STA activity may go stale before the worker faults `StaHung` (no command in flight). Must be between `1` and `86400` and below `WorkerHeartbeatStuckCeilingSeconds`; validation uses this configured value, not a copied constant (mxaccessgw#224). Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS`; a missing or invalid value, or a grace the ceiling does not exceed, leaves the worker on 15 s. |
|
||||||
|
| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than `WorkerHeartbeatGraceSeconds` and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. |
|
||||||
| `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. |
|
| `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. |
|
||||||
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. |
|
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. |
|
||||||
|
|
||||||
|
|||||||
@@ -102,6 +102,7 @@ MXGATEWAY_ALARM_POLL_INTERVAL_MS=<milliseconds>
|
|||||||
MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH=<alarms>
|
MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH=<alarms>
|
||||||
MXGATEWAY_ALARM_ACK_CLIENT_NAME=<client name>
|
MXGATEWAY_ALARM_ACK_CLIENT_NAME=<client name>
|
||||||
MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS=<milliseconds>
|
MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS=<milliseconds>
|
||||||
|
MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS=<milliseconds>
|
||||||
MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS=<milliseconds>
|
MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS=<milliseconds>
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -40,7 +40,8 @@ worker. The launcher always exports every value below:
|
|||||||
| `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms |
|
| `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms |
|
||||||
| `MXGATEWAY_ALARM_ACK_CLIENT_NAME` | `MxGateway:Alarms:AckClientName` | `MxGateway` |
|
| `MXGATEWAY_ALARM_ACK_CLIENT_NAME` | `MxGateway:Alarms:AckClientName` | `MxGateway` |
|
||||||
| `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS` | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` (× 1000) | `5000` ms |
|
| `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS` | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` (× 1000) | `5000` ms |
|
||||||
| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's 15 s watchdog grace) |
|
| `MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS` | `MxGateway:Worker:WorkerHeartbeatGraceSeconds` (× 1000) | `15000` ms (also kept when the ceiling is not above it) |
|
||||||
|
| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's watchdog grace) |
|
||||||
|
|
||||||
The write-completion wait applies to the four single-item write variants:
|
The write-completion wait applies to the four single-item write variants:
|
||||||
`Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM
|
`Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM
|
||||||
|
|||||||
@@ -42,4 +42,12 @@ public static class WorkerLaunchEnvironment
|
|||||||
/// <c>MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds</c>.
|
/// <c>MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds</c>.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public const string HeartbeatStuckCeilingMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS";
|
public const string HeartbeatStuckCeilingMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Grace, in milliseconds, the worker's own STA watchdog allows STA activity to go stale
|
||||||
|
/// before faulting <c>StaHung</c> (when no command is in flight). Set from
|
||||||
|
/// <c>MxGateway:Worker:WorkerHeartbeatGraceSeconds</c>; the worker's stuck ceiling must
|
||||||
|
/// exceed it.
|
||||||
|
/// </summary>
|
||||||
|
public const string HeartbeatGraceMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS";
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,5 +9,8 @@ public sealed record EffectiveWorkerConfiguration(
|
|||||||
int HeartbeatCheckIntervalSeconds,
|
int HeartbeatCheckIntervalSeconds,
|
||||||
int HeartbeatGraceSeconds,
|
int HeartbeatGraceSeconds,
|
||||||
int HeartbeatStuckCeilingSeconds,
|
int HeartbeatStuckCeilingSeconds,
|
||||||
|
int WorkerHeartbeatSendIntervalSeconds,
|
||||||
|
int WorkerHeartbeatGraceSeconds,
|
||||||
|
int WorkerHeartbeatStuckCeilingSeconds,
|
||||||
int EventChannelFullModeTimeoutMilliseconds,
|
int EventChannelFullModeTimeoutMilliseconds,
|
||||||
int MaxMessageBytes);
|
int MaxMessageBytes);
|
||||||
|
|||||||
@@ -41,6 +41,9 @@ public sealed class GatewayConfigurationProvider(IOptions<GatewayOptions> option
|
|||||||
HeartbeatCheckIntervalSeconds: value.Worker.ResolveHeartbeatCheckIntervalSeconds(),
|
HeartbeatCheckIntervalSeconds: value.Worker.ResolveHeartbeatCheckIntervalSeconds(),
|
||||||
HeartbeatGraceSeconds: value.Worker.HeartbeatGraceSeconds,
|
HeartbeatGraceSeconds: value.Worker.HeartbeatGraceSeconds,
|
||||||
HeartbeatStuckCeilingSeconds: value.Worker.HeartbeatStuckCeilingSeconds,
|
HeartbeatStuckCeilingSeconds: value.Worker.HeartbeatStuckCeilingSeconds,
|
||||||
|
WorkerHeartbeatSendIntervalSeconds: value.Worker.WorkerHeartbeatSendIntervalSeconds,
|
||||||
|
WorkerHeartbeatGraceSeconds: value.Worker.WorkerHeartbeatGraceSeconds,
|
||||||
|
WorkerHeartbeatStuckCeilingSeconds: value.Worker.WorkerHeartbeatStuckCeilingSeconds,
|
||||||
EventChannelFullModeTimeoutMilliseconds: value.Worker.EventChannelFullModeTimeoutMilliseconds,
|
EventChannelFullModeTimeoutMilliseconds: value.Worker.EventChannelFullModeTimeoutMilliseconds,
|
||||||
MaxMessageBytes: value.Worker.MaxMessageBytes),
|
MaxMessageBytes: value.Worker.MaxMessageBytes),
|
||||||
Sessions: new EffectiveSessionConfiguration(
|
Sessions: new EffectiveSessionConfiguration(
|
||||||
|
|||||||
@@ -310,10 +310,23 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
|
|||||||
$"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most {MaximumWorkerHeartbeatSeconds}.");
|
$"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most {MaximumWorkerHeartbeatSeconds}.");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (options.WorkerHeartbeatStuckCeilingSeconds <= WorkerOptions.WorkerWatchdogGraceSeconds)
|
AddIfNotPositive(
|
||||||
|
options.WorkerHeartbeatGraceSeconds,
|
||||||
|
"MxGateway:Worker:WorkerHeartbeatGraceSeconds must be greater than zero.",
|
||||||
|
builder);
|
||||||
|
|
||||||
|
if (options.WorkerHeartbeatGraceSeconds > MaximumWorkerHeartbeatSeconds)
|
||||||
{
|
{
|
||||||
builder.Add(
|
builder.Add(
|
||||||
$"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's {WorkerOptions.WorkerWatchdogGraceSeconds}-second watchdog grace.");
|
$"MxGateway:Worker:WorkerHeartbeatGraceSeconds must be at most {MaximumWorkerHeartbeatSeconds}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
// The worker refuses a ceiling at or below its own grace, so hold it above the configured
|
||||||
|
// grace (not a copied constant) -- mxaccessgw#224.
|
||||||
|
if (options.WorkerHeartbeatStuckCeilingSeconds <= options.WorkerHeartbeatGraceSeconds)
|
||||||
|
{
|
||||||
|
builder.Add(
|
||||||
|
"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than WorkerHeartbeatGraceSeconds.");
|
||||||
}
|
}
|
||||||
|
|
||||||
AddIfNotPositive(
|
AddIfNotPositive(
|
||||||
|
|||||||
@@ -93,14 +93,17 @@ public sealed class WorkerOptions
|
|||||||
/// <summary>Default for <see cref="WorkerHeartbeatStuckCeilingSeconds"/>.</summary>
|
/// <summary>Default for <see cref="WorkerHeartbeatStuckCeilingSeconds"/>.</summary>
|
||||||
public const int DefaultWorkerHeartbeatStuckCeilingSeconds = 75;
|
public const int DefaultWorkerHeartbeatStuckCeilingSeconds = 75;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>Default for <see cref="WorkerHeartbeatGraceSeconds"/>.</summary>
|
||||||
/// The worker's own STA-watchdog grace, in seconds. Not configurable: the worker keeps its
|
public const int DefaultWorkerHeartbeatGraceSeconds = 15;
|
||||||
/// compiled default (<c>WorkerPipeSessionOptions.DefaultHeartbeatGrace</c>). Mirrored here
|
|
||||||
/// only so validation can hold <see cref="WorkerHeartbeatStuckCeilingSeconds"/> above it,
|
|
||||||
/// which the worker requires.
|
|
||||||
/// </summary>
|
|
||||||
public const int WorkerWatchdogGraceSeconds = 15;
|
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The worker's own STA-watchdog grace, in seconds: how long STA activity may go stale
|
||||||
|
/// before the worker faults <c>StaHung</c> (no command in flight). Default 15.
|
||||||
|
/// <see cref="WorkerHeartbeatStuckCeilingSeconds"/> must be greater than this, which
|
||||||
|
/// validation checks against the configured value rather than a copied constant.
|
||||||
|
/// Conveyed through <c>MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS</c>.
|
||||||
|
/// </summary>
|
||||||
|
public int WorkerHeartbeatGraceSeconds { get; init; } = DefaultWorkerHeartbeatGraceSeconds;
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Interval, in seconds, at which the worker sends heartbeats to the gateway. Default 5.
|
/// Interval, in seconds, at which the worker sends heartbeats to the gateway. Default 5.
|
||||||
/// <see cref="HeartbeatGraceSeconds"/> must be greater than this, or the gateway declares
|
/// <see cref="HeartbeatGraceSeconds"/> must be greater than this, or the gateway declares
|
||||||
@@ -112,8 +115,8 @@ public sealed class WorkerOptions
|
|||||||
/// <summary>
|
/// <summary>
|
||||||
/// Ceiling, in seconds, on the worker's own STA watchdog: once STA activity has been stale
|
/// Ceiling, in seconds, on the worker's own STA watchdog: once STA activity has been stale
|
||||||
/// this long the worker faults <c>StaHung</c> even while a command is in flight. Default 75;
|
/// this long the worker faults <c>StaHung</c> even while a command is in flight. Default 75;
|
||||||
/// must be greater than the worker's fixed 15-second watchdog grace
|
/// must be greater than the worker's watchdog grace
|
||||||
/// (<see cref="WorkerWatchdogGraceSeconds"/>). Raise it, together with
|
/// (<see cref="WorkerHeartbeatGraceSeconds"/>). Raise it, together with
|
||||||
/// <see cref="HeartbeatStuckCeilingSeconds"/>, for deployments that run very long bulk
|
/// <see cref="HeartbeatStuckCeilingSeconds"/>, for deployments that run very long bulk
|
||||||
/// operations. Conveyed through <c>MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS</c>.
|
/// operations. Conveyed through <c>MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS</c>.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
|||||||
@@ -93,6 +93,9 @@ else
|
|||||||
<tr><th scope="row">Heartbeat grace</th><td>@Snapshot.Configuration.Worker.HeartbeatGraceSeconds seconds</td></tr>
|
<tr><th scope="row">Heartbeat grace</th><td>@Snapshot.Configuration.Worker.HeartbeatGraceSeconds seconds</td></tr>
|
||||||
<tr><th scope="row">Heartbeat check interval</th><td>@Snapshot.Configuration.Worker.HeartbeatCheckIntervalSeconds seconds</td></tr>
|
<tr><th scope="row">Heartbeat check interval</th><td>@Snapshot.Configuration.Worker.HeartbeatCheckIntervalSeconds seconds</td></tr>
|
||||||
<tr><th scope="row">Heartbeat stuck ceiling</th><td>@Snapshot.Configuration.Worker.HeartbeatStuckCeilingSeconds seconds</td></tr>
|
<tr><th scope="row">Heartbeat stuck ceiling</th><td>@Snapshot.Configuration.Worker.HeartbeatStuckCeilingSeconds seconds</td></tr>
|
||||||
|
<tr><th scope="row">Worker heartbeat send interval</th><td>@Snapshot.Configuration.Worker.WorkerHeartbeatSendIntervalSeconds seconds</td></tr>
|
||||||
|
<tr><th scope="row">Worker watchdog grace</th><td>@Snapshot.Configuration.Worker.WorkerHeartbeatGraceSeconds seconds</td></tr>
|
||||||
|
<tr><th scope="row">Worker watchdog stuck ceiling</th><td>@Snapshot.Configuration.Worker.WorkerHeartbeatStuckCeilingSeconds seconds</td></tr>
|
||||||
<tr><th scope="row">Event channel full-mode timeout</th><td>@Snapshot.Configuration.Worker.EventChannelFullModeTimeoutMilliseconds ms</td></tr>
|
<tr><th scope="row">Event channel full-mode timeout</th><td>@Snapshot.Configuration.Worker.EventChannelFullModeTimeoutMilliseconds ms</td></tr>
|
||||||
<tr><th scope="row">Default command timeout</th><td>@Snapshot.Configuration.Sessions.DefaultCommandTimeoutSeconds seconds</td></tr>
|
<tr><th scope="row">Default command timeout</th><td>@Snapshot.Configuration.Sessions.DefaultCommandTimeoutSeconds seconds</td></tr>
|
||||||
<tr><th scope="row">Max sessions</th><td>@Snapshot.Configuration.Sessions.MaxSessions</td></tr>
|
<tr><th scope="row">Max sessions</th><td>@Snapshot.Configuration.Sessions.MaxSessions</td></tr>
|
||||||
|
|||||||
@@ -219,6 +219,8 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher
|
|||||||
startInfo.Environment[WorkerLaunchEnvironment.AlarmAckClientName] = _alarmsOptions.AckClientName;
|
startInfo.Environment[WorkerLaunchEnvironment.AlarmAckClientName] = _alarmsOptions.AckClientName;
|
||||||
startInfo.Environment[WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds] =
|
startInfo.Environment[WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds] =
|
||||||
(_workerOptions.WorkerHeartbeatSendIntervalSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture);
|
(_workerOptions.WorkerHeartbeatSendIntervalSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||||
|
startInfo.Environment[WorkerLaunchEnvironment.HeartbeatGraceMilliseconds] =
|
||||||
|
(_workerOptions.WorkerHeartbeatGraceSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||||
startInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds] =
|
startInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds] =
|
||||||
(_workerOptions.WorkerHeartbeatStuckCeilingSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture);
|
(_workerOptions.WorkerHeartbeatStuckCeilingSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||||
|
|
||||||
|
|||||||
@@ -53,6 +53,7 @@
|
|||||||
"HeartbeatGraceSeconds": 15,
|
"HeartbeatGraceSeconds": 15,
|
||||||
"HeartbeatStuckCeilingSeconds": 75,
|
"HeartbeatStuckCeilingSeconds": 75,
|
||||||
"WorkerHeartbeatSendIntervalSeconds": 5,
|
"WorkerHeartbeatSendIntervalSeconds": 5,
|
||||||
|
"WorkerHeartbeatGraceSeconds": 15,
|
||||||
"WorkerHeartbeatStuckCeilingSeconds": 75,
|
"WorkerHeartbeatStuckCeilingSeconds": 75,
|
||||||
"EventChannelFullModeTimeoutMilliseconds": 5000,
|
"EventChannelFullModeTimeoutMilliseconds": 5000,
|
||||||
"MaxMessageBytes": 16842752
|
"MaxMessageBytes": 16842752
|
||||||
|
|||||||
@@ -45,6 +45,7 @@ public sealed class GatewayOptionsTests
|
|||||||
Assert.Equal(75, options.Worker.HeartbeatStuckCeilingSeconds);
|
Assert.Equal(75, options.Worker.HeartbeatStuckCeilingSeconds);
|
||||||
Assert.Equal(5, options.Worker.WorkerHeartbeatSendIntervalSeconds);
|
Assert.Equal(5, options.Worker.WorkerHeartbeatSendIntervalSeconds);
|
||||||
Assert.Equal(75, options.Worker.WorkerHeartbeatStuckCeilingSeconds);
|
Assert.Equal(75, options.Worker.WorkerHeartbeatStuckCeilingSeconds);
|
||||||
|
Assert.Equal(15, options.Worker.WorkerHeartbeatGraceSeconds);
|
||||||
Assert.Equal(5000, options.Worker.EventChannelFullModeTimeoutMilliseconds);
|
Assert.Equal(5000, options.Worker.EventChannelFullModeTimeoutMilliseconds);
|
||||||
Assert.Equal(15, options.Worker.HeartbeatGraceSeconds);
|
Assert.Equal(15, options.Worker.HeartbeatGraceSeconds);
|
||||||
// 16 MiB public gRPC cap plus the 64 KiB worker-frame envelope reserve for headroom.
|
// 16 MiB public gRPC cap plus the 64 KiB worker-frame envelope reserve for headroom.
|
||||||
@@ -123,7 +124,10 @@ public sealed class GatewayOptionsTests
|
|||||||
[InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "15", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")]
|
[InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "15", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")]
|
||||||
[InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be at most 86400.")]
|
[InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be at most 86400.")]
|
||||||
[InlineData("MxGateway:Worker:HeartbeatGraceSeconds", "5", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")]
|
[InlineData("MxGateway:Worker:HeartbeatGraceSeconds", "5", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")]
|
||||||
[InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "15", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's 15-second watchdog grace.")]
|
[InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "15", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than WorkerHeartbeatGraceSeconds.")]
|
||||||
|
[InlineData("MxGateway:Worker:WorkerHeartbeatGraceSeconds", "75", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than WorkerHeartbeatGraceSeconds.")]
|
||||||
|
[InlineData("MxGateway:Worker:WorkerHeartbeatGraceSeconds", "0", "MxGateway:Worker:WorkerHeartbeatGraceSeconds must be greater than zero.")]
|
||||||
|
[InlineData("MxGateway:Worker:WorkerHeartbeatGraceSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatGraceSeconds must be at most 86400.")]
|
||||||
[InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most 86400.")]
|
[InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most 86400.")]
|
||||||
[InlineData("MxGateway:Sessions:DefaultLeaseSeconds", "0", "MxGateway:Sessions:DefaultLeaseSeconds must be greater than zero.")]
|
[InlineData("MxGateway:Sessions:DefaultLeaseSeconds", "0", "MxGateway:Sessions:DefaultLeaseSeconds must be greater than zero.")]
|
||||||
[InlineData("MxGateway:Sessions:LeaseSweepIntervalSeconds", "0", "MxGateway:Sessions:LeaseSweepIntervalSeconds must be greater than zero.")]
|
[InlineData("MxGateway:Sessions:LeaseSweepIntervalSeconds", "0", "MxGateway:Sessions:LeaseSweepIntervalSeconds must be greater than zero.")]
|
||||||
|
|||||||
@@ -120,6 +120,31 @@ public sealed class SettingsPageTagVisibilityRenderTests
|
|||||||
Assert.Contains("""<th scope="row">Replay buffer max bytes</th><td>987654321</td>""", html, StringComparison.Ordinal);
|
Assert.Contains("""<th scope="row">Replay buffer max bytes</th><td>987654321</td>""", html, StringComparison.Ordinal);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The worker-side heartbeat send interval, watchdog grace and stuck ceiling (mxaccessgw#218,
|
||||||
|
/// #224) reach the page, each in its own labelled row, distinct from the gateway-side ones.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task SettingsPage_RendersWorkerHeartbeatOptions()
|
||||||
|
{
|
||||||
|
string html = await RenderAsync(new GatewayOptions
|
||||||
|
{
|
||||||
|
Worker = new WorkerOptions
|
||||||
|
{
|
||||||
|
HeartbeatStuckCeilingSeconds = 100,
|
||||||
|
WorkerHeartbeatSendIntervalSeconds = 3,
|
||||||
|
WorkerHeartbeatGraceSeconds = 31,
|
||||||
|
WorkerHeartbeatStuckCeilingSeconds = 241,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.Contains("""<th scope="row">Worker heartbeat send interval</th><td>3 seconds</td>""", html, StringComparison.Ordinal);
|
||||||
|
Assert.Contains("""<th scope="row">Worker watchdog grace</th><td>31 seconds</td>""", html, StringComparison.Ordinal);
|
||||||
|
Assert.Contains("""<th scope="row">Worker watchdog stuck ceiling</th><td>241 seconds</td>""", html, StringComparison.Ordinal);
|
||||||
|
Assert.Contains("""<th scope="row">Heartbeat stuck ceiling</th><td>100 seconds</td>""", html, StringComparison.Ordinal);
|
||||||
|
}
|
||||||
|
|
||||||
private static Task<string> RenderAsync(DashboardOptions dashboard) =>
|
private static Task<string> RenderAsync(DashboardOptions dashboard) =>
|
||||||
RenderAsync(new GatewayOptions { Dashboard = dashboard });
|
RenderAsync(new GatewayOptions { Dashboard = dashboard });
|
||||||
|
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ public sealed class WorkerProcessLauncherTests
|
|||||||
WorkerLaunchEnvironment.MaxAlarmsPerFetch,
|
WorkerLaunchEnvironment.MaxAlarmsPerFetch,
|
||||||
WorkerLaunchEnvironment.AlarmAckClientName,
|
WorkerLaunchEnvironment.AlarmAckClientName,
|
||||||
WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds,
|
WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds,
|
||||||
|
WorkerLaunchEnvironment.HeartbeatGraceMilliseconds,
|
||||||
WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds,
|
WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds,
|
||||||
];
|
];
|
||||||
|
|
||||||
@@ -91,6 +92,10 @@ public sealed class WorkerProcessLauncherTests
|
|||||||
"75000",
|
"75000",
|
||||||
processFactory.LastStartInfo.Environment[
|
processFactory.LastStartInfo.Environment[
|
||||||
WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]);
|
WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]);
|
||||||
|
Assert.Equal(
|
||||||
|
"15000",
|
||||||
|
processFactory.LastStartInfo.Environment[
|
||||||
|
WorkerLaunchEnvironment.HeartbeatGraceMilliseconds]);
|
||||||
Assert.All(
|
Assert.All(
|
||||||
RequiredLaunchEnvironmentVariables,
|
RequiredLaunchEnvironmentVariables,
|
||||||
variable => Assert.True(
|
variable => Assert.True(
|
||||||
@@ -166,7 +171,8 @@ public sealed class WorkerProcessLauncherTests
|
|||||||
processFactory,
|
processFactory,
|
||||||
new SucceedingStartupProbe(),
|
new SucceedingStartupProbe(),
|
||||||
workerHeartbeatSendIntervalSeconds: 3,
|
workerHeartbeatSendIntervalSeconds: 3,
|
||||||
workerHeartbeatStuckCeilingSeconds: 240);
|
workerHeartbeatStuckCeilingSeconds: 240,
|
||||||
|
workerHeartbeatGraceSeconds: 30);
|
||||||
|
|
||||||
using WorkerProcessHandle handle = await launcher.LaunchAsync(CreateRequest());
|
using WorkerProcessHandle handle = await launcher.LaunchAsync(CreateRequest());
|
||||||
|
|
||||||
@@ -177,6 +183,9 @@ public sealed class WorkerProcessLauncherTests
|
|||||||
Assert.Equal(
|
Assert.Equal(
|
||||||
"240000",
|
"240000",
|
||||||
processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]);
|
processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]);
|
||||||
|
Assert.Equal(
|
||||||
|
"30000",
|
||||||
|
processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatGraceMilliseconds]);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Verifies that a failed startup probe kills and disposes the worker process.</summary>
|
/// <summary>Verifies that a failed startup probe kills and disposes the worker process.</summary>
|
||||||
@@ -305,7 +314,8 @@ public sealed class WorkerProcessLauncherTests
|
|||||||
int eventQueueCapacity = 10000,
|
int eventQueueCapacity = 10000,
|
||||||
string ackClientName = "MxGateway",
|
string ackClientName = "MxGateway",
|
||||||
int workerHeartbeatSendIntervalSeconds = WorkerOptions.DefaultWorkerHeartbeatSendIntervalSeconds,
|
int workerHeartbeatSendIntervalSeconds = WorkerOptions.DefaultWorkerHeartbeatSendIntervalSeconds,
|
||||||
int workerHeartbeatStuckCeilingSeconds = WorkerOptions.DefaultWorkerHeartbeatStuckCeilingSeconds)
|
int workerHeartbeatStuckCeilingSeconds = WorkerOptions.DefaultWorkerHeartbeatStuckCeilingSeconds,
|
||||||
|
int workerHeartbeatGraceSeconds = WorkerOptions.DefaultWorkerHeartbeatGraceSeconds)
|
||||||
{
|
{
|
||||||
GatewayOptions options = new()
|
GatewayOptions options = new()
|
||||||
{
|
{
|
||||||
@@ -318,6 +328,7 @@ public sealed class WorkerProcessLauncherTests
|
|||||||
EventQueueCapacity = eventQueueCapacity,
|
EventQueueCapacity = eventQueueCapacity,
|
||||||
WorkerHeartbeatSendIntervalSeconds = workerHeartbeatSendIntervalSeconds,
|
WorkerHeartbeatSendIntervalSeconds = workerHeartbeatSendIntervalSeconds,
|
||||||
WorkerHeartbeatStuckCeilingSeconds = workerHeartbeatStuckCeilingSeconds,
|
WorkerHeartbeatStuckCeilingSeconds = workerHeartbeatStuckCeilingSeconds,
|
||||||
|
WorkerHeartbeatGraceSeconds = workerHeartbeatGraceSeconds,
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -26,6 +26,31 @@ public sealed class WorkerPipeSessionOptionsTests
|
|||||||
options.Validate();
|
options.Validate();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that a launcher-provided grace replaces the compiled default (mxaccessgw#224).</summary>
|
||||||
|
[Fact]
|
||||||
|
public void FromEnvironment_WithGrace_UsesItAndStaysValid()
|
||||||
|
{
|
||||||
|
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup(
|
||||||
|
(WorkerLaunchEnvironment.HeartbeatGraceMilliseconds, "30000"),
|
||||||
|
(WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, "120000")));
|
||||||
|
|
||||||
|
Assert.Equal(TimeSpan.FromSeconds(30), options.HeartbeatGrace);
|
||||||
|
Assert.Equal(TimeSpan.FromSeconds(120), options.HeartbeatStuckCeiling);
|
||||||
|
options.Validate();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that a grace the ceiling does not exceed is dropped rather than failing the worker.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void FromEnvironment_WithGraceAtOrAboveCeiling_FallsBackToDefaultGrace()
|
||||||
|
{
|
||||||
|
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup(
|
||||||
|
(WorkerLaunchEnvironment.HeartbeatGraceMilliseconds, "80000")));
|
||||||
|
|
||||||
|
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatGrace, options.HeartbeatGrace);
|
||||||
|
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling);
|
||||||
|
options.Validate();
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Verifies that an absent environment yields the compiled defaults.</summary>
|
/// <summary>Verifies that an absent environment yields the compiled defaults.</summary>
|
||||||
[Fact]
|
[Fact]
|
||||||
public void FromEnvironment_WithNothingSet_KeepsDefaults()
|
public void FromEnvironment_WithNothingSet_KeepsDefaults()
|
||||||
|
|||||||
@@ -37,7 +37,8 @@ public sealed class WorkerPipeSessionOptions
|
|||||||
/// keeps its default, and so does a ceiling that would fail <see cref="Validate"/> (not
|
/// keeps its default, and so does a ceiling that would fail <see cref="Validate"/> (not
|
||||||
/// above <see cref="HeartbeatGrace"/>): the gateway validates both at startup, and a
|
/// above <see cref="HeartbeatGrace"/>): the gateway validates both at startup, and a
|
||||||
/// worker that refused to start over a stray environment value would turn a tuning
|
/// worker that refused to start over a stray environment value would turn a tuning
|
||||||
/// mistake into a session outage. The grace is not conveyed and keeps its default.
|
/// mistake into a session outage. The watchdog grace is conveyed too
|
||||||
|
/// (<see cref="WorkerLaunchEnvironment.HeartbeatGraceMilliseconds"/>) and must stay below the ceiling.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <param name="getEnvironmentVariable">
|
/// <param name="getEnvironmentVariable">
|
||||||
/// Environment lookup; <see langword="null"/> reads the process environment.
|
/// Environment lookup; <see langword="null"/> reads the process environment.
|
||||||
@@ -53,8 +54,25 @@ public sealed class WorkerPipeSessionOptions
|
|||||||
options.HeartbeatInterval = interval;
|
options.HeartbeatInterval = interval;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, out TimeSpan ceiling)
|
TimeSpan grace = options.HeartbeatGrace;
|
||||||
&& ceiling > options.HeartbeatGrace)
|
bool haveGrace = TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatGraceMilliseconds, out TimeSpan configuredGrace);
|
||||||
|
if (haveGrace)
|
||||||
|
{
|
||||||
|
grace = configuredGrace;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool haveCeiling = TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, out TimeSpan ceiling);
|
||||||
|
TimeSpan effectiveCeiling = haveCeiling ? ceiling : options.HeartbeatStuckCeiling;
|
||||||
|
|
||||||
|
// Keep ceiling > grace so Validate() always passes. A grace the ceiling does not exceed
|
||||||
|
// is dropped (back to the default) rather than failing the worker; an unusable ceiling
|
||||||
|
// is dropped first when the default grace still leaves it valid.
|
||||||
|
if (effectiveCeiling > grace)
|
||||||
|
{
|
||||||
|
options.HeartbeatGrace = grace;
|
||||||
|
options.HeartbeatStuckCeiling = effectiveCeiling;
|
||||||
|
}
|
||||||
|
else if (haveCeiling && ceiling > options.HeartbeatGrace)
|
||||||
{
|
{
|
||||||
options.HeartbeatStuckCeiling = ceiling;
|
options.HeartbeatStuckCeiling = ceiling;
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user