feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m32s
ci / java (push) Successful in 2m57s
ci / portable (push) Failing after 5m1s

The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.

- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
  and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
  WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
  MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
  worker settings use.
- The worker's production WorkerPipeSession reads them through
  WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
  ceiling not above the worker's 15 s watchdog grace, keeps the default rather
  than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
  send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
  the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:19:32 -05:00
parent 29d9510b4e
commit 2e52f9f5e8
13 changed files with 284 additions and 9 deletions
@@ -0,0 +1,72 @@
using System;
using System.Collections.Generic;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Worker.Ipc;
namespace ZB.MOM.WW.MxGateway.Worker.Tests.Ipc;
/// <summary>
/// Covers <see cref="WorkerPipeSessionOptions.FromEnvironment"/>: the gateway conveys the
/// worker's heartbeat send interval and watchdog stuck ceiling through the launch environment
/// (mxaccessgw#218), and a bad value must fall back rather than keep the worker from starting.
/// </summary>
public sealed class WorkerPipeSessionOptionsTests
{
/// <summary>Verifies that launcher-provided values replace the compiled defaults.</summary>
[Fact]
public void FromEnvironment_WithLauncherValues_UsesThem()
{
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup(
(WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, "3000"),
(WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, "240000")));
Assert.Equal(TimeSpan.FromSeconds(3), options.HeartbeatInterval);
Assert.Equal(TimeSpan.FromSeconds(240), options.HeartbeatStuckCeiling);
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatGrace, options.HeartbeatGrace);
options.Validate();
}
/// <summary>Verifies that an absent environment yields the compiled defaults.</summary>
[Fact]
public void FromEnvironment_WithNothingSet_KeepsDefaults()
{
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup());
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatInterval, options.HeartbeatInterval);
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling);
}
/// <summary>
/// Verifies that unusable values fall back to the defaults, including a ceiling that would
/// fail <see cref="WorkerPipeSessionOptions.Validate"/> by not exceeding the grace.
/// </summary>
/// <param name="interval">Send-interval environment value.</param>
/// <param name="ceiling">Stuck-ceiling environment value.</param>
[Theory]
[InlineData("0", "0")]
[InlineData("-5", "-5")]
[InlineData("abc", "1.5")]
[InlineData("", "15000")]
[InlineData("99999999999", "99999999999")]
public void FromEnvironment_WithUnusableValues_FallsBackToDefaults(string interval, string ceiling)
{
WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup(
(WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, interval),
(WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, ceiling)));
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatInterval, options.HeartbeatInterval);
Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling);
options.Validate();
}
private static Func<string, string?> Lookup(params (string Name, string Value)[] values)
{
Dictionary<string, string> map = new(StringComparer.Ordinal);
foreach ((string name, string value) in values)
{
map[name] = value;
}
return name => map.TryGetValue(name, out string? value) ? value : null;
}
}