feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck ceiling (75 s) were compile-time defaults the gateway never reached, and nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send interval, so a misconfigured grace could kill healthy workers. - New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5) and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other worker settings use. - The worker's production WorkerPipeSession reads them through WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a ceiling not above the worker's 15 s watchdog grace, keeps the default rather than failing the worker. - Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds; send interval > 0; worker ceiling > 15 s; both at most 86400 s. - Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -27,6 +27,8 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
|
||||
"HeartbeatCheckIntervalSeconds": 5,
|
||||
"HeartbeatGraceSeconds": 15,
|
||||
"HeartbeatStuckCeilingSeconds": 75,
|
||||
"WorkerHeartbeatSendIntervalSeconds": 5,
|
||||
"WorkerHeartbeatStuckCeilingSeconds": 75,
|
||||
"EventChannelFullModeTimeoutMilliseconds": 5000,
|
||||
"MaxMessageBytes": 16842752
|
||||
},
|
||||
@@ -148,10 +150,12 @@ launch CWD (SEC-01, SEC-33).
|
||||
| `MxGateway:Worker:WriteCompletionWaitMilliseconds` | `1500` | Bounded wait the worker holds a unary write reply (`Write`/`Write2`/`WriteSecured`/`WriteSecured2`; bulk writes excluded) for the matching MXAccess `OnWriteComplete` callback, so the reply's `statuses` carry the real commit outcome. `0` disables the wait (pure fire-and-forget replies). Must be `>= 0`. The gateway conveys the value to the worker via the `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` environment variable. Consumers that time their own writes must budget above this wait: OtOpcUa's GalaxyDriver wraps gateway writes in a 2 s Tier A resilience timeout, so a deployment raising this option past ~2000 must raise that driver `ResilienceConfig` write timeout in step or slow-but-successful commits surface as consumer-side failures. |
|
||||
| `MxGateway:Worker:EventQueueCapacity` | `10000` | Capacity, in events, of the worker's outbound MXAccess event queue. Must be between `1000` and `1000000`. This is burst headroom, not a throttle: the queue has no drop policy, so filling it records a `QueueOverflow` worker fault and faults the session. Raise it for sessions whose subscription set can outrun the drain loop (large advise sets, slow event consumers); the backing queue pre-allocates its slots, so the ceiling keeps a mistyped value from committing the 32-bit worker to an outsized allocation. The gateway conveys the value to the worker via the `MXGATEWAY_EVENT_QUEUE_CAPACITY` environment variable; a missing or unusable value leaves the worker on the 10000 default rather than failing the session. |
|
||||
| `MxGateway:Worker:ShutdownTimeoutSeconds` | `10` | Grace period for worker shutdown before the gateway treats shutdown as failed and may kill the worker process tree. |
|
||||
| `MxGateway:Worker:HeartbeatCheckIntervalSeconds` | `5` | How often the gateway checks the age of the worker's last heartbeat against `HeartbeatGraceSeconds`. This is the gateway-side check cadence only; the worker sends heartbeats on its own fixed 5-second interval, which no gateway option reaches. Must be greater than zero. |
|
||||
| `MxGateway:Worker:HeartbeatCheckIntervalSeconds` | `5` | How often the gateway checks the age of the worker's last heartbeat against `HeartbeatGraceSeconds`. This is the gateway-side check cadence only; the worker's send interval is `WorkerHeartbeatSendIntervalSeconds`. Must be greater than zero. |
|
||||
| `MxGateway:Worker:HeartbeatIntervalSeconds` | *(unset)* | **Deprecated** alias of `HeartbeatCheckIntervalSeconds` (mxaccessgw#192). The old name implied the worker's send interval, but the value only ever set the gateway's check cadence, which is what it still sets. Setting both keys fails validation. |
|
||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`. |
|
||||
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog keeps its separate, fixed 75-second ceiling. |
|
||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`, and strictly greater than `WorkerHeartbeatSendIntervalSeconds` — a grace at or below the worker's send interval declares a healthy worker dead between two heartbeats (mxaccessgw#218). |
|
||||
| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog has its separate ceiling, `WorkerHeartbeatStuckCeilingSeconds`. |
|
||||
| `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` | `5` | Interval at which the worker sends heartbeats to the gateway. Must be between `1` and `86400`, and below `HeartbeatGraceSeconds`. Conveyed to the worker via `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS`; a missing or invalid value leaves the worker on 5 s. |
|
||||
| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than the worker's fixed 15-second watchdog grace and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. |
|
||||
| `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. |
|
||||
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. |
|
||||
|
||||
|
||||
@@ -100,6 +100,9 @@ MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS=<milliseconds>
|
||||
MXGATEWAY_EVENT_QUEUE_CAPACITY=<events>
|
||||
MXGATEWAY_ALARM_POLL_INTERVAL_MS=<milliseconds>
|
||||
MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH=<alarms>
|
||||
MXGATEWAY_ALARM_ACK_CLIENT_NAME=<client name>
|
||||
MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS=<milliseconds>
|
||||
MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS=<milliseconds>
|
||||
```
|
||||
|
||||
The nonce is protected bootstrap material. The remaining values are derived
|
||||
|
||||
@@ -28,7 +28,7 @@ diagnostics can report the launch command without exposing the secret.
|
||||
|
||||
All launch-variable names live in the shared Contracts
|
||||
`WorkerLaunchEnvironment` class and are consumed by both the launcher and the
|
||||
worker. The launcher always exports all six values:
|
||||
worker. The launcher always exports every value below:
|
||||
|
||||
| Environment variable | Gateway source | Worker default when missing/invalid |
|
||||
|---|---|---|
|
||||
@@ -38,6 +38,9 @@ worker. The launcher always exports all six values:
|
||||
| `MXGATEWAY_EVENT_QUEUE_CAPACITY` | `MxGateway:Worker:EventQueueCapacity` | `10000` events |
|
||||
| `MXGATEWAY_ALARM_POLL_INTERVAL_MS` | `MxGateway:Alarms:PollIntervalMilliseconds` | `500` ms |
|
||||
| `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms |
|
||||
| `MXGATEWAY_ALARM_ACK_CLIENT_NAME` | `MxGateway:Alarms:AckClientName` | `MxGateway` |
|
||||
| `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS` | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` (× 1000) | `5000` ms |
|
||||
| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's 15 s watchdog grace) |
|
||||
|
||||
The write-completion wait applies to the four single-item write variants:
|
||||
`Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM
|
||||
|
||||
@@ -28,4 +28,18 @@ public static class WorkerLaunchEnvironment
|
||||
/// Historian records it as the acknowledgement's OperatorNode.
|
||||
/// </summary>
|
||||
public const string AlarmAckClientName = "MXGATEWAY_ALARM_ACK_CLIENT_NAME";
|
||||
|
||||
/// <summary>
|
||||
/// Interval, in milliseconds, at which the worker sends heartbeats to the gateway. Set
|
||||
/// from <c>MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds</c>; the gateway's
|
||||
/// <c>HeartbeatGraceSeconds</c> must exceed it or healthy workers are declared dead.
|
||||
/// </summary>
|
||||
public const string HeartbeatSendIntervalMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS";
|
||||
|
||||
/// <summary>
|
||||
/// Ceiling, in milliseconds, after which the worker's own STA watchdog faults
|
||||
/// <c>StaHung</c> even while a command is in flight. Set from
|
||||
/// <c>MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds</c>.
|
||||
/// </summary>
|
||||
public const string HeartbeatStuckCeilingMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS";
|
||||
}
|
||||
|
||||
@@ -16,6 +16,9 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
|
||||
private const int MinimumWorkerEventQueueCapacity = 1000;
|
||||
private const int MaximumWorkerEventQueueCapacity = 1_000_000;
|
||||
|
||||
// The worker heartbeat settings travel as int milliseconds; a day keeps them far inside that.
|
||||
private const int MaximumWorkerHeartbeatSeconds = 86_400;
|
||||
|
||||
// Whether the host is running in the Production environment. Drives the production-only
|
||||
// hard-stops (dashboard login disabled, plaintext LDAP transport) that must abort startup
|
||||
// rather than merely warn. Non-production hosts keep the permissive dev posture.
|
||||
@@ -280,6 +283,39 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
|
||||
"MxGateway:Worker:HeartbeatGraceSeconds must be greater than or equal to HeartbeatCheckIntervalSeconds.");
|
||||
}
|
||||
|
||||
AddIfNotPositive(
|
||||
options.WorkerHeartbeatSendIntervalSeconds,
|
||||
"MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be greater than zero.",
|
||||
builder);
|
||||
|
||||
// The gateway's grace is measured against heartbeats the worker sends on its own interval:
|
||||
// a grace at or below that interval declares a healthy worker dead in the gap between two
|
||||
// heartbeats (mxaccessgw#218).
|
||||
if (options.WorkerHeartbeatSendIntervalSeconds > 0
|
||||
&& options.HeartbeatGraceSeconds <= options.WorkerHeartbeatSendIntervalSeconds)
|
||||
{
|
||||
builder.Add(
|
||||
"MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.");
|
||||
}
|
||||
|
||||
if (options.WorkerHeartbeatSendIntervalSeconds > MaximumWorkerHeartbeatSeconds)
|
||||
{
|
||||
builder.Add(
|
||||
$"MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be at most {MaximumWorkerHeartbeatSeconds}.");
|
||||
}
|
||||
|
||||
if (options.WorkerHeartbeatStuckCeilingSeconds > MaximumWorkerHeartbeatSeconds)
|
||||
{
|
||||
builder.Add(
|
||||
$"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most {MaximumWorkerHeartbeatSeconds}.");
|
||||
}
|
||||
|
||||
if (options.WorkerHeartbeatStuckCeilingSeconds <= WorkerOptions.WorkerWatchdogGraceSeconds)
|
||||
{
|
||||
builder.Add(
|
||||
$"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's {WorkerOptions.WorkerWatchdogGraceSeconds}-second watchdog grace.");
|
||||
}
|
||||
|
||||
AddIfNotPositive(
|
||||
options.EventChannelFullModeTimeoutMilliseconds,
|
||||
"MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds must be greater than zero.",
|
||||
|
||||
@@ -48,8 +48,8 @@ public sealed class WorkerOptions
|
||||
/// <summary>
|
||||
/// How often, in seconds, the gateway checks the age of the worker's last heartbeat
|
||||
/// against <see cref="HeartbeatGraceSeconds"/>. This is the gateway-side <em>check</em>
|
||||
/// cadence only: the worker sends heartbeats on its own fixed 5-second interval, which
|
||||
/// this option does not reach. <see langword="null"/> falls back to the deprecated
|
||||
/// cadence only: the worker's own send interval is
|
||||
/// <see cref="WorkerHeartbeatSendIntervalSeconds"/>. <see langword="null"/> falls back to the deprecated
|
||||
/// <see cref="HeartbeatIntervalSeconds"/> key, then to
|
||||
/// <see cref="DefaultHeartbeatCheckIntervalSeconds"/>; read the resolved value through
|
||||
/// <see cref="ResolveHeartbeatCheckIntervalSeconds"/>.
|
||||
@@ -81,11 +81,44 @@ public sealed class WorkerOptions
|
||||
/// is pending the gateway tolerates a stale heartbeat (the worker STA is busy in a COM
|
||||
/// call), but once the oldest pending command is older than this the heartbeat watchdog
|
||||
/// faults the worker anyway. Default 75; must be at least <see cref="HeartbeatGraceSeconds"/>.
|
||||
/// The worker's own watchdog keeps its separate, fixed 75-second ceiling.
|
||||
/// The worker's own watchdog has its separate ceiling,
|
||||
/// <see cref="WorkerHeartbeatStuckCeilingSeconds"/>.
|
||||
/// </summary>
|
||||
public int HeartbeatStuckCeilingSeconds { get; init; } =
|
||||
(int)Workers.WorkerClientOptions.DefaultHeartbeatStuckCeiling.TotalSeconds;
|
||||
|
||||
/// <summary>Default for <see cref="WorkerHeartbeatSendIntervalSeconds"/>.</summary>
|
||||
public const int DefaultWorkerHeartbeatSendIntervalSeconds = 5;
|
||||
|
||||
/// <summary>Default for <see cref="WorkerHeartbeatStuckCeilingSeconds"/>.</summary>
|
||||
public const int DefaultWorkerHeartbeatStuckCeilingSeconds = 75;
|
||||
|
||||
/// <summary>
|
||||
/// The worker's own STA-watchdog grace, in seconds. Not configurable: the worker keeps its
|
||||
/// compiled default (<c>WorkerPipeSessionOptions.DefaultHeartbeatGrace</c>). Mirrored here
|
||||
/// only so validation can hold <see cref="WorkerHeartbeatStuckCeilingSeconds"/> above it,
|
||||
/// which the worker requires.
|
||||
/// </summary>
|
||||
public const int WorkerWatchdogGraceSeconds = 15;
|
||||
|
||||
/// <summary>
|
||||
/// Interval, in seconds, at which the worker sends heartbeats to the gateway. Default 5.
|
||||
/// <see cref="HeartbeatGraceSeconds"/> must be greater than this, or the gateway declares
|
||||
/// a healthy worker dead between two heartbeats. Conveyed to the worker through the
|
||||
/// <c>MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS</c> environment variable.
|
||||
/// </summary>
|
||||
public int WorkerHeartbeatSendIntervalSeconds { get; init; } = DefaultWorkerHeartbeatSendIntervalSeconds;
|
||||
|
||||
/// <summary>
|
||||
/// Ceiling, in seconds, on the worker's own STA watchdog: once STA activity has been stale
|
||||
/// this long the worker faults <c>StaHung</c> even while a command is in flight. Default 75;
|
||||
/// must be greater than the worker's fixed 15-second watchdog grace
|
||||
/// (<see cref="WorkerWatchdogGraceSeconds"/>). Raise it, together with
|
||||
/// <see cref="HeartbeatStuckCeilingSeconds"/>, for deployments that run very long bulk
|
||||
/// operations. Conveyed through <c>MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS</c>.
|
||||
/// </summary>
|
||||
public int WorkerHeartbeatStuckCeilingSeconds { get; init; } = DefaultWorkerHeartbeatStuckCeilingSeconds;
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the effective heartbeat check interval: <see cref="HeartbeatCheckIntervalSeconds"/>,
|
||||
/// else the deprecated <see cref="HeartbeatIntervalSeconds"/>, else
|
||||
|
||||
@@ -217,6 +217,10 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher
|
||||
_alarmsOptions.MaxAlarmsPerFetch.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
|
||||
startInfo.Environment[WorkerLaunchEnvironment.AlarmAckClientName] = _alarmsOptions.AckClientName;
|
||||
startInfo.Environment[WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds] =
|
||||
(_workerOptions.WorkerHeartbeatSendIntervalSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
startInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds] =
|
||||
(_workerOptions.WorkerHeartbeatStuckCeilingSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
|
||||
commandLine = new WorkerProcessCommandLine(executablePath, arguments);
|
||||
|
||||
|
||||
@@ -52,6 +52,8 @@
|
||||
"HeartbeatCheckIntervalSeconds": 5,
|
||||
"HeartbeatGraceSeconds": 15,
|
||||
"HeartbeatStuckCeilingSeconds": 75,
|
||||
"WorkerHeartbeatSendIntervalSeconds": 5,
|
||||
"WorkerHeartbeatStuckCeilingSeconds": 75,
|
||||
"EventChannelFullModeTimeoutMilliseconds": 5000,
|
||||
"MaxMessageBytes": 16842752
|
||||
},
|
||||
|
||||
@@ -43,6 +43,8 @@ public sealed class GatewayOptionsTests
|
||||
Assert.Equal(10, options.Worker.ShutdownTimeoutSeconds);
|
||||
Assert.Equal(5, options.Worker.ResolveHeartbeatCheckIntervalSeconds());
|
||||
Assert.Equal(75, options.Worker.HeartbeatStuckCeilingSeconds);
|
||||
Assert.Equal(5, options.Worker.WorkerHeartbeatSendIntervalSeconds);
|
||||
Assert.Equal(75, options.Worker.WorkerHeartbeatStuckCeilingSeconds);
|
||||
Assert.Equal(5000, options.Worker.EventChannelFullModeTimeoutMilliseconds);
|
||||
Assert.Equal(15, options.Worker.HeartbeatGraceSeconds);
|
||||
// 16 MiB public gRPC cap plus the 64 KiB worker-frame envelope reserve for headroom.
|
||||
@@ -117,6 +119,12 @@ public sealed class GatewayOptionsTests
|
||||
[InlineData("MxGateway:Worker:HeartbeatCheckIntervalSeconds", "16", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than or equal to HeartbeatCheckIntervalSeconds.")]
|
||||
[InlineData("MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds", "0", "MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds must be greater than zero.")]
|
||||
[InlineData("MxGateway:Worker:HeartbeatStuckCeilingSeconds", "14", "MxGateway:Worker:HeartbeatStuckCeilingSeconds must be greater than or equal to HeartbeatGraceSeconds.")]
|
||||
[InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "0", "MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be greater than zero.")]
|
||||
[InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "15", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")]
|
||||
[InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be at most 86400.")]
|
||||
[InlineData("MxGateway:Worker:HeartbeatGraceSeconds", "5", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")]
|
||||
[InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "15", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's 15-second watchdog grace.")]
|
||||
[InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most 86400.")]
|
||||
[InlineData("MxGateway:Sessions:DefaultLeaseSeconds", "0", "MxGateway:Sessions:DefaultLeaseSeconds must be greater than zero.")]
|
||||
[InlineData("MxGateway:Sessions:LeaseSweepIntervalSeconds", "0", "MxGateway:Sessions:LeaseSweepIntervalSeconds must be greater than zero.")]
|
||||
[InlineData("MxGateway:Sessions:DetachGraceSeconds", "-1", "MxGateway:Sessions:DetachGraceSeconds must be zero or greater (0 disables detach-grace retention).")]
|
||||
|
||||
@@ -23,6 +23,8 @@ public sealed class WorkerProcessLauncherTests
|
||||
WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds,
|
||||
WorkerLaunchEnvironment.MaxAlarmsPerFetch,
|
||||
WorkerLaunchEnvironment.AlarmAckClientName,
|
||||
WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds,
|
||||
WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds,
|
||||
];
|
||||
|
||||
/// <summary>Verifies that a valid worker executable starts with correct bootstrap arguments and nonce environment variable.</summary>
|
||||
@@ -80,6 +82,15 @@ public sealed class WorkerProcessLauncherTests
|
||||
"MxGateway",
|
||||
processFactory.LastStartInfo.Environment[
|
||||
WorkerLaunchEnvironment.AlarmAckClientName]);
|
||||
// The worker's heartbeat send interval and its own watchdog ceiling (mxaccessgw#218).
|
||||
Assert.Equal(
|
||||
"5000",
|
||||
processFactory.LastStartInfo.Environment[
|
||||
WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds]);
|
||||
Assert.Equal(
|
||||
"75000",
|
||||
processFactory.LastStartInfo.Environment[
|
||||
WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]);
|
||||
Assert.All(
|
||||
RequiredLaunchEnvironmentVariables,
|
||||
variable => Assert.True(
|
||||
@@ -139,6 +150,35 @@ public sealed class WorkerProcessLauncherTests
|
||||
WorkerLaunchEnvironment.EventQueueCapacity]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that configured worker heartbeat settings — not the shipped defaults — reach the
|
||||
/// worker in milliseconds (mxaccessgw#218), so they are tunable without a worker rebuild.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task LaunchAsync_WithConfiguredWorkerHeartbeat_ExportsItToTheWorkerEnvironment()
|
||||
{
|
||||
using TestDirectory directory = TestDirectory.Create();
|
||||
string executablePath = directory.CreateWorkerExecutable(machine: 0x014c);
|
||||
FakeWorkerProcessFactory processFactory = new(new FakeWorkerProcess(processId: 1234));
|
||||
WorkerProcessLauncher launcher = CreateLauncher(
|
||||
executablePath,
|
||||
processFactory,
|
||||
new SucceedingStartupProbe(),
|
||||
workerHeartbeatSendIntervalSeconds: 3,
|
||||
workerHeartbeatStuckCeilingSeconds: 240);
|
||||
|
||||
using WorkerProcessHandle handle = await launcher.LaunchAsync(CreateRequest());
|
||||
|
||||
Assert.NotNull(processFactory.LastStartInfo);
|
||||
Assert.Equal(
|
||||
"3000",
|
||||
processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds]);
|
||||
Assert.Equal(
|
||||
"240000",
|
||||
processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that a failed startup probe kills and disposes the worker process.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
@@ -263,7 +303,9 @@ public sealed class WorkerProcessLauncherTests
|
||||
GatewayMetrics? metrics = null,
|
||||
int startupTimeoutSeconds = 30,
|
||||
int eventQueueCapacity = 10000,
|
||||
string ackClientName = "MxGateway")
|
||||
string ackClientName = "MxGateway",
|
||||
int workerHeartbeatSendIntervalSeconds = WorkerOptions.DefaultWorkerHeartbeatSendIntervalSeconds,
|
||||
int workerHeartbeatStuckCeilingSeconds = WorkerOptions.DefaultWorkerHeartbeatStuckCeilingSeconds)
|
||||
{
|
||||
GatewayOptions options = new()
|
||||
{
|
||||
@@ -274,6 +316,8 @@ public sealed class WorkerProcessLauncherTests
|
||||
RequiredArchitecture = WorkerArchitecture.X86,
|
||||
StartupTimeoutSeconds = startupTimeoutSeconds,
|
||||
EventQueueCapacity = eventQueueCapacity,
|
||||
WorkerHeartbeatSendIntervalSeconds = workerHeartbeatSendIntervalSeconds,
|
||||
WorkerHeartbeatStuckCeilingSeconds = workerHeartbeatStuckCeilingSeconds,
|
||||
},
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using ZB.MOM.WW.MxGateway.Contracts;
|
||||
using ZB.MOM.WW.MxGateway.Worker.Ipc;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Worker.Tests.Ipc;
|
||||
|
||||
/// <summary>
|
||||
/// Covers <see cref="WorkerPipeSessionOptions.FromEnvironment"/>: the gateway conveys the
|
||||
/// worker's heartbeat send interval and watchdog stuck ceiling through the launch environment
|
||||
/// (mxaccessgw#218), and a bad value must fall back rather than keep the worker from starting.
|
||||
/// </summary>
|
||||
public sealed class WorkerPipeSessionOptionsTests
|
||||
{
|
||||
/// <summary>Verifies that launcher-provided values replace the compiled defaults.</summary>
|
||||
[Fact]
|
||||
public void FromEnvironment_WithLauncherValues_UsesThem()
|
||||
{
|
||||
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup(
|
||||
(WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, "3000"),
|
||||
(WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, "240000")));
|
||||
|
||||
Assert.Equal(TimeSpan.FromSeconds(3), options.HeartbeatInterval);
|
||||
Assert.Equal(TimeSpan.FromSeconds(240), options.HeartbeatStuckCeiling);
|
||||
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatGrace, options.HeartbeatGrace);
|
||||
options.Validate();
|
||||
}
|
||||
|
||||
/// <summary>Verifies that an absent environment yields the compiled defaults.</summary>
|
||||
[Fact]
|
||||
public void FromEnvironment_WithNothingSet_KeepsDefaults()
|
||||
{
|
||||
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup());
|
||||
|
||||
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatInterval, options.HeartbeatInterval);
|
||||
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that unusable values fall back to the defaults, including a ceiling that would
|
||||
/// fail <see cref="WorkerPipeSessionOptions.Validate"/> by not exceeding the grace.
|
||||
/// </summary>
|
||||
/// <param name="interval">Send-interval environment value.</param>
|
||||
/// <param name="ceiling">Stuck-ceiling environment value.</param>
|
||||
[Theory]
|
||||
[InlineData("0", "0")]
|
||||
[InlineData("-5", "-5")]
|
||||
[InlineData("abc", "1.5")]
|
||||
[InlineData("", "15000")]
|
||||
[InlineData("99999999999", "99999999999")]
|
||||
public void FromEnvironment_WithUnusableValues_FallsBackToDefaults(string interval, string ceiling)
|
||||
{
|
||||
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup(
|
||||
(WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, interval),
|
||||
(WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, ceiling)));
|
||||
|
||||
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatInterval, options.HeartbeatInterval);
|
||||
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling);
|
||||
options.Validate();
|
||||
}
|
||||
|
||||
private static Func<string, string?> Lookup(params (string Name, string Value)[] values)
|
||||
{
|
||||
Dictionary<string, string> map = new(StringComparer.Ordinal);
|
||||
foreach ((string name, string value) in values)
|
||||
{
|
||||
map[name] = value;
|
||||
}
|
||||
|
||||
return name => map.TryGetValue(name, out string? value) ? value : null;
|
||||
}
|
||||
}
|
||||
@@ -75,7 +75,7 @@ public sealed class WorkerPipeSession
|
||||
new WorkerFrameWriter(stream, options),
|
||||
options,
|
||||
() => Process.GetCurrentProcess().Id,
|
||||
new WorkerPipeSessionOptions(),
|
||||
WorkerPipeSessionOptions.FromEnvironment(),
|
||||
() => CreateDefaultRuntimeSession(WorkerCommandLogging.IsEnabled() ? logger : null),
|
||||
logger,
|
||||
stream)
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using System;
|
||||
using System.Globalization;
|
||||
using ZB.MOM.WW.MxGateway.Contracts;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Worker.Ipc;
|
||||
|
||||
@@ -26,6 +28,56 @@ public sealed class WorkerPipeSessionOptions
|
||||
HeartbeatStuckCeiling = DefaultHeartbeatStuckCeiling;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the production options from the launcher-provided environment
|
||||
/// (<see cref="WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds"/> and
|
||||
/// <see cref="WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds"/>), which the
|
||||
/// gateway sets from <c>MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds</c> and
|
||||
/// <c>WorkerHeartbeatStuckCeilingSeconds</c>. A missing, unparseable, or non-positive value
|
||||
/// keeps its default, and so does a ceiling that would fail <see cref="Validate"/> (not
|
||||
/// above <see cref="HeartbeatGrace"/>): the gateway validates both at startup, and a
|
||||
/// worker that refused to start over a stray environment value would turn a tuning
|
||||
/// mistake into a session outage. The grace is not conveyed and keeps its default.
|
||||
/// </summary>
|
||||
/// <param name="getEnvironmentVariable">
|
||||
/// Environment lookup; <see langword="null"/> reads the process environment.
|
||||
/// </param>
|
||||
/// <returns>Options that always pass <see cref="Validate"/>.</returns>
|
||||
public static WorkerPipeSessionOptions FromEnvironment(Func<string, string?>? getEnvironmentVariable = null)
|
||||
{
|
||||
Func<string, string?> lookup = getEnvironmentVariable ?? Environment.GetEnvironmentVariable;
|
||||
WorkerPipeSessionOptions options = new WorkerPipeSessionOptions();
|
||||
|
||||
if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, out TimeSpan interval))
|
||||
{
|
||||
options.HeartbeatInterval = interval;
|
||||
}
|
||||
|
||||
if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, out TimeSpan ceiling)
|
||||
&& ceiling > options.HeartbeatGrace)
|
||||
{
|
||||
options.HeartbeatStuckCeiling = ceiling;
|
||||
}
|
||||
|
||||
return options;
|
||||
}
|
||||
|
||||
private static bool TryReadPositiveMilliseconds(
|
||||
Func<string, string?> lookup,
|
||||
string name,
|
||||
out TimeSpan value)
|
||||
{
|
||||
value = TimeSpan.Zero;
|
||||
if (!int.TryParse(lookup(name), NumberStyles.Integer, CultureInfo.InvariantCulture, out int milliseconds)
|
||||
|| milliseconds <= 0)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
value = TimeSpan.FromMilliseconds(milliseconds);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>Gets or sets the heartbeat interval.</summary>
|
||||
public TimeSpan HeartbeatInterval { get; set; }
|
||||
|
||||
|
||||
Reference in New Issue
Block a user