feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m32s
ci / java (push) Successful in 2m57s
ci / portable (push) Failing after 5m1s

The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.

- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
  and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
  WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
  MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
  worker settings use.
- The worker's production WorkerPipeSession reads them through
  WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
  ceiling not above the worker's 15 s watchdog grace, keeps the default rather
  than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
  send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
  the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:19:32 -05:00
parent 29d9510b4e
commit 2e52f9f5e8
13 changed files with 284 additions and 9 deletions
@@ -75,7 +75,7 @@ public sealed class WorkerPipeSession
new WorkerFrameWriter(stream, options),
options,
() => Process.GetCurrentProcess().Id,
new WorkerPipeSessionOptions(),
WorkerPipeSessionOptions.FromEnvironment(),
() => CreateDefaultRuntimeSession(WorkerCommandLogging.IsEnabled() ? logger : null),
logger,
stream)
@@ -1,4 +1,6 @@
using System;
using System.Globalization;
using ZB.MOM.WW.MxGateway.Contracts;
namespace ZB.MOM.WW.MxGateway.Worker.Ipc;
@@ -26,6 +28,56 @@ public sealed class WorkerPipeSessionOptions
HeartbeatStuckCeiling = DefaultHeartbeatStuckCeiling;
}
/// <summary>
/// Builds the production options from the launcher-provided environment
/// (<see cref="WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds"/> and
/// <see cref="WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds"/>), which the
/// gateway sets from <c>MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds</c> and
/// <c>WorkerHeartbeatStuckCeilingSeconds</c>. A missing, unparseable, or non-positive value
/// keeps its default, and so does a ceiling that would fail <see cref="Validate"/> (not
/// above <see cref="HeartbeatGrace"/>): the gateway validates both at startup, and a
/// worker that refused to start over a stray environment value would turn a tuning
/// mistake into a session outage. The grace is not conveyed and keeps its default.
/// </summary>
/// <param name="getEnvironmentVariable">
/// Environment lookup; <see langword="null"/> reads the process environment.
/// </param>
/// <returns>Options that always pass <see cref="Validate"/>.</returns>
public static WorkerPipeSessionOptions FromEnvironment(Func<string, string?>? getEnvironmentVariable = null)
{
Func<string, string?> lookup = getEnvironmentVariable ?? Environment.GetEnvironmentVariable;
WorkerPipeSessionOptions options = new WorkerPipeSessionOptions();
if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, out TimeSpan interval))
{
options.HeartbeatInterval = interval;
}
if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, out TimeSpan ceiling)
&& ceiling > options.HeartbeatGrace)
{
options.HeartbeatStuckCeiling = ceiling;
}
return options;
}
private static bool TryReadPositiveMilliseconds(
Func<string, string?> lookup,
string name,
out TimeSpan value)
{
value = TimeSpan.Zero;
if (!int.TryParse(lookup(name), NumberStyles.Integer, CultureInfo.InvariantCulture, out int milliseconds)
|| milliseconds <= 0)
{
return false;
}
value = TimeSpan.FromMilliseconds(milliseconds);
return true;
}
/// <summary>Gets or sets the heartbeat interval.</summary>
public TimeSpan HeartbeatInterval { get; set; }