diff --git a/docs/GatewayConfiguration.md b/docs/GatewayConfiguration.md index 842f97ef..a1364331 100644 --- a/docs/GatewayConfiguration.md +++ b/docs/GatewayConfiguration.md @@ -27,6 +27,8 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid. "HeartbeatCheckIntervalSeconds": 5, "HeartbeatGraceSeconds": 15, "HeartbeatStuckCeilingSeconds": 75, + "WorkerHeartbeatSendIntervalSeconds": 5, + "WorkerHeartbeatStuckCeilingSeconds": 75, "EventChannelFullModeTimeoutMilliseconds": 5000, "MaxMessageBytes": 16842752 }, @@ -148,10 +150,12 @@ launch CWD (SEC-01, SEC-33). | `MxGateway:Worker:WriteCompletionWaitMilliseconds` | `1500` | Bounded wait the worker holds a unary write reply (`Write`/`Write2`/`WriteSecured`/`WriteSecured2`; bulk writes excluded) for the matching MXAccess `OnWriteComplete` callback, so the reply's `statuses` carry the real commit outcome. `0` disables the wait (pure fire-and-forget replies). Must be `>= 0`. The gateway conveys the value to the worker via the `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` environment variable. Consumers that time their own writes must budget above this wait: OtOpcUa's GalaxyDriver wraps gateway writes in a 2 s Tier A resilience timeout, so a deployment raising this option past ~2000 must raise that driver `ResilienceConfig` write timeout in step or slow-but-successful commits surface as consumer-side failures. | | `MxGateway:Worker:EventQueueCapacity` | `10000` | Capacity, in events, of the worker's outbound MXAccess event queue. Must be between `1000` and `1000000`. This is burst headroom, not a throttle: the queue has no drop policy, so filling it records a `QueueOverflow` worker fault and faults the session. Raise it for sessions whose subscription set can outrun the drain loop (large advise sets, slow event consumers); the backing queue pre-allocates its slots, so the ceiling keeps a mistyped value from committing the 32-bit worker to an outsized allocation. The gateway conveys the value to the worker via the `MXGATEWAY_EVENT_QUEUE_CAPACITY` environment variable; a missing or unusable value leaves the worker on the 10000 default rather than failing the session. | | `MxGateway:Worker:ShutdownTimeoutSeconds` | `10` | Grace period for worker shutdown before the gateway treats shutdown as failed and may kill the worker process tree. | -| `MxGateway:Worker:HeartbeatCheckIntervalSeconds` | `5` | How often the gateway checks the age of the worker's last heartbeat against `HeartbeatGraceSeconds`. This is the gateway-side check cadence only; the worker sends heartbeats on its own fixed 5-second interval, which no gateway option reaches. Must be greater than zero. | +| `MxGateway:Worker:HeartbeatCheckIntervalSeconds` | `5` | How often the gateway checks the age of the worker's last heartbeat against `HeartbeatGraceSeconds`. This is the gateway-side check cadence only; the worker's send interval is `WorkerHeartbeatSendIntervalSeconds`. Must be greater than zero. | | `MxGateway:Worker:HeartbeatIntervalSeconds` | *(unset)* | **Deprecated** alias of `HeartbeatCheckIntervalSeconds` (mxaccessgw#192). The old name implied the worker's send interval, but the value only ever set the gateway's check cadence, which is what it still sets. Setting both keys fails validation. | -| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`. | -| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog keeps its separate, fixed 75-second ceiling. | +| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`, and strictly greater than `WorkerHeartbeatSendIntervalSeconds` — a grace at or below the worker's send interval declares a healthy worker dead between two heartbeats (mxaccessgw#218). | +| `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog has its separate ceiling, `WorkerHeartbeatStuckCeilingSeconds`. | +| `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` | `5` | Interval at which the worker sends heartbeats to the gateway. Must be between `1` and `86400`, and below `HeartbeatGraceSeconds`. Conveyed to the worker via `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS`; a missing or invalid value leaves the worker on 5 s. | +| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than the worker's fixed 15-second watchdog grace and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. | | `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. | | `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. | diff --git a/docs/MxAccessWorkerInstanceDesign.md b/docs/MxAccessWorkerInstanceDesign.md index 92f0e37e..8f8a5cee 100644 --- a/docs/MxAccessWorkerInstanceDesign.md +++ b/docs/MxAccessWorkerInstanceDesign.md @@ -100,6 +100,9 @@ MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS= MXGATEWAY_EVENT_QUEUE_CAPACITY= MXGATEWAY_ALARM_POLL_INTERVAL_MS= MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH= +MXGATEWAY_ALARM_ACK_CLIENT_NAME= +MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS= +MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS= ``` The nonce is protected bootstrap material. The remaining values are derived diff --git a/docs/WorkerProcessLauncher.md b/docs/WorkerProcessLauncher.md index 56d9aa5f..97890ea3 100644 --- a/docs/WorkerProcessLauncher.md +++ b/docs/WorkerProcessLauncher.md @@ -28,7 +28,7 @@ diagnostics can report the launch command without exposing the secret. All launch-variable names live in the shared Contracts `WorkerLaunchEnvironment` class and are consumed by both the launcher and the -worker. The launcher always exports all six values: +worker. The launcher always exports every value below: | Environment variable | Gateway source | Worker default when missing/invalid | |---|---|---| @@ -38,6 +38,9 @@ worker. The launcher always exports all six values: | `MXGATEWAY_EVENT_QUEUE_CAPACITY` | `MxGateway:Worker:EventQueueCapacity` | `10000` events | | `MXGATEWAY_ALARM_POLL_INTERVAL_MS` | `MxGateway:Alarms:PollIntervalMilliseconds` | `500` ms | | `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms | +| `MXGATEWAY_ALARM_ACK_CLIENT_NAME` | `MxGateway:Alarms:AckClientName` | `MxGateway` | +| `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS` | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` (× 1000) | `5000` ms | +| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's 15 s watchdog grace) | The write-completion wait applies to the four single-item write variants: `Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM diff --git a/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs b/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs index 0ea79886..dc474a56 100644 --- a/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs +++ b/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs @@ -28,4 +28,18 @@ public static class WorkerLaunchEnvironment /// Historian records it as the acknowledgement's OperatorNode. /// public const string AlarmAckClientName = "MXGATEWAY_ALARM_ACK_CLIENT_NAME"; + + /// + /// Interval, in milliseconds, at which the worker sends heartbeats to the gateway. Set + /// from MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds; the gateway's + /// HeartbeatGraceSeconds must exceed it or healthy workers are declared dead. + /// + public const string HeartbeatSendIntervalMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS"; + + /// + /// Ceiling, in milliseconds, after which the worker's own STA watchdog faults + /// StaHung even while a command is in flight. Set from + /// MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds. + /// + public const string HeartbeatStuckCeilingMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS"; } diff --git a/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs b/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs index c543a395..25b59397 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs @@ -16,6 +16,9 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase 0 + && options.HeartbeatGraceSeconds <= options.WorkerHeartbeatSendIntervalSeconds) + { + builder.Add( + "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds."); + } + + if (options.WorkerHeartbeatSendIntervalSeconds > MaximumWorkerHeartbeatSeconds) + { + builder.Add( + $"MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be at most {MaximumWorkerHeartbeatSeconds}."); + } + + if (options.WorkerHeartbeatStuckCeilingSeconds > MaximumWorkerHeartbeatSeconds) + { + builder.Add( + $"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most {MaximumWorkerHeartbeatSeconds}."); + } + + if (options.WorkerHeartbeatStuckCeilingSeconds <= WorkerOptions.WorkerWatchdogGraceSeconds) + { + builder.Add( + $"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's {WorkerOptions.WorkerWatchdogGraceSeconds}-second watchdog grace."); + } + AddIfNotPositive( options.EventChannelFullModeTimeoutMilliseconds, "MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds must be greater than zero.", diff --git a/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs b/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs index 3e619b00..559079a0 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs @@ -48,8 +48,8 @@ public sealed class WorkerOptions /// /// How often, in seconds, the gateway checks the age of the worker's last heartbeat /// against . This is the gateway-side check - /// cadence only: the worker sends heartbeats on its own fixed 5-second interval, which - /// this option does not reach. falls back to the deprecated + /// cadence only: the worker's own send interval is + /// . falls back to the deprecated /// key, then to /// ; read the resolved value through /// . @@ -81,11 +81,44 @@ public sealed class WorkerOptions /// is pending the gateway tolerates a stale heartbeat (the worker STA is busy in a COM /// call), but once the oldest pending command is older than this the heartbeat watchdog /// faults the worker anyway. Default 75; must be at least . - /// The worker's own watchdog keeps its separate, fixed 75-second ceiling. + /// The worker's own watchdog has its separate ceiling, + /// . /// public int HeartbeatStuckCeilingSeconds { get; init; } = (int)Workers.WorkerClientOptions.DefaultHeartbeatStuckCeiling.TotalSeconds; + /// Default for . + public const int DefaultWorkerHeartbeatSendIntervalSeconds = 5; + + /// Default for . + public const int DefaultWorkerHeartbeatStuckCeilingSeconds = 75; + + /// + /// The worker's own STA-watchdog grace, in seconds. Not configurable: the worker keeps its + /// compiled default (WorkerPipeSessionOptions.DefaultHeartbeatGrace). Mirrored here + /// only so validation can hold above it, + /// which the worker requires. + /// + public const int WorkerWatchdogGraceSeconds = 15; + + /// + /// Interval, in seconds, at which the worker sends heartbeats to the gateway. Default 5. + /// must be greater than this, or the gateway declares + /// a healthy worker dead between two heartbeats. Conveyed to the worker through the + /// MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS environment variable. + /// + public int WorkerHeartbeatSendIntervalSeconds { get; init; } = DefaultWorkerHeartbeatSendIntervalSeconds; + + /// + /// Ceiling, in seconds, on the worker's own STA watchdog: once STA activity has been stale + /// this long the worker faults StaHung even while a command is in flight. Default 75; + /// must be greater than the worker's fixed 15-second watchdog grace + /// (). Raise it, together with + /// , for deployments that run very long bulk + /// operations. Conveyed through MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS. + /// + public int WorkerHeartbeatStuckCeilingSeconds { get; init; } = DefaultWorkerHeartbeatStuckCeilingSeconds; + /// /// Resolves the effective heartbeat check interval: , /// else the deprecated , else diff --git a/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs b/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs index 1e4ace5d..0bb322ba 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs @@ -217,6 +217,10 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher _alarmsOptions.MaxAlarmsPerFetch.ToString(System.Globalization.CultureInfo.InvariantCulture); startInfo.Environment[WorkerLaunchEnvironment.AlarmAckClientName] = _alarmsOptions.AckClientName; + startInfo.Environment[WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds] = + (_workerOptions.WorkerHeartbeatSendIntervalSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture); + startInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds] = + (_workerOptions.WorkerHeartbeatStuckCeilingSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture); commandLine = new WorkerProcessCommandLine(executablePath, arguments); diff --git a/src/ZB.MOM.WW.MxGateway.Server/appsettings.json b/src/ZB.MOM.WW.MxGateway.Server/appsettings.json index 5cfce6c3..b3da90a1 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/appsettings.json +++ b/src/ZB.MOM.WW.MxGateway.Server/appsettings.json @@ -52,6 +52,8 @@ "HeartbeatCheckIntervalSeconds": 5, "HeartbeatGraceSeconds": 15, "HeartbeatStuckCeilingSeconds": 75, + "WorkerHeartbeatSendIntervalSeconds": 5, + "WorkerHeartbeatStuckCeilingSeconds": 75, "EventChannelFullModeTimeoutMilliseconds": 5000, "MaxMessageBytes": 16842752 }, diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs index a486376e..1b9d1237 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs @@ -43,6 +43,8 @@ public sealed class GatewayOptionsTests Assert.Equal(10, options.Worker.ShutdownTimeoutSeconds); Assert.Equal(5, options.Worker.ResolveHeartbeatCheckIntervalSeconds()); Assert.Equal(75, options.Worker.HeartbeatStuckCeilingSeconds); + Assert.Equal(5, options.Worker.WorkerHeartbeatSendIntervalSeconds); + Assert.Equal(75, options.Worker.WorkerHeartbeatStuckCeilingSeconds); Assert.Equal(5000, options.Worker.EventChannelFullModeTimeoutMilliseconds); Assert.Equal(15, options.Worker.HeartbeatGraceSeconds); // 16 MiB public gRPC cap plus the 64 KiB worker-frame envelope reserve for headroom. @@ -117,6 +119,12 @@ public sealed class GatewayOptionsTests [InlineData("MxGateway:Worker:HeartbeatCheckIntervalSeconds", "16", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than or equal to HeartbeatCheckIntervalSeconds.")] [InlineData("MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds", "0", "MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds must be greater than zero.")] [InlineData("MxGateway:Worker:HeartbeatStuckCeilingSeconds", "14", "MxGateway:Worker:HeartbeatStuckCeilingSeconds must be greater than or equal to HeartbeatGraceSeconds.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "0", "MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be greater than zero.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "15", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be at most 86400.")] + [InlineData("MxGateway:Worker:HeartbeatGraceSeconds", "5", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "15", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's 15-second watchdog grace.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most 86400.")] [InlineData("MxGateway:Sessions:DefaultLeaseSeconds", "0", "MxGateway:Sessions:DefaultLeaseSeconds must be greater than zero.")] [InlineData("MxGateway:Sessions:LeaseSweepIntervalSeconds", "0", "MxGateway:Sessions:LeaseSweepIntervalSeconds must be greater than zero.")] [InlineData("MxGateway:Sessions:DetachGraceSeconds", "-1", "MxGateway:Sessions:DetachGraceSeconds must be zero or greater (0 disables detach-grace retention).")] diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs index e7d238e0..256c8a85 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs @@ -23,6 +23,8 @@ public sealed class WorkerProcessLauncherTests WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds, WorkerLaunchEnvironment.MaxAlarmsPerFetch, WorkerLaunchEnvironment.AlarmAckClientName, + WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, + WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, ]; /// Verifies that a valid worker executable starts with correct bootstrap arguments and nonce environment variable. @@ -80,6 +82,15 @@ public sealed class WorkerProcessLauncherTests "MxGateway", processFactory.LastStartInfo.Environment[ WorkerLaunchEnvironment.AlarmAckClientName]); + // The worker's heartbeat send interval and its own watchdog ceiling (mxaccessgw#218). + Assert.Equal( + "5000", + processFactory.LastStartInfo.Environment[ + WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds]); + Assert.Equal( + "75000", + processFactory.LastStartInfo.Environment[ + WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]); Assert.All( RequiredLaunchEnvironmentVariables, variable => Assert.True( @@ -139,6 +150,35 @@ public sealed class WorkerProcessLauncherTests WorkerLaunchEnvironment.EventQueueCapacity]); } + /// + /// Verifies that configured worker heartbeat settings — not the shipped defaults — reach the + /// worker in milliseconds (mxaccessgw#218), so they are tunable without a worker rebuild. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task LaunchAsync_WithConfiguredWorkerHeartbeat_ExportsItToTheWorkerEnvironment() + { + using TestDirectory directory = TestDirectory.Create(); + string executablePath = directory.CreateWorkerExecutable(machine: 0x014c); + FakeWorkerProcessFactory processFactory = new(new FakeWorkerProcess(processId: 1234)); + WorkerProcessLauncher launcher = CreateLauncher( + executablePath, + processFactory, + new SucceedingStartupProbe(), + workerHeartbeatSendIntervalSeconds: 3, + workerHeartbeatStuckCeilingSeconds: 240); + + using WorkerProcessHandle handle = await launcher.LaunchAsync(CreateRequest()); + + Assert.NotNull(processFactory.LastStartInfo); + Assert.Equal( + "3000", + processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds]); + Assert.Equal( + "240000", + processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]); + } + /// Verifies that a failed startup probe kills and disposes the worker process. /// A task that represents the asynchronous operation. [Fact] @@ -263,7 +303,9 @@ public sealed class WorkerProcessLauncherTests GatewayMetrics? metrics = null, int startupTimeoutSeconds = 30, int eventQueueCapacity = 10000, - string ackClientName = "MxGateway") + string ackClientName = "MxGateway", + int workerHeartbeatSendIntervalSeconds = WorkerOptions.DefaultWorkerHeartbeatSendIntervalSeconds, + int workerHeartbeatStuckCeilingSeconds = WorkerOptions.DefaultWorkerHeartbeatStuckCeilingSeconds) { GatewayOptions options = new() { @@ -274,6 +316,8 @@ public sealed class WorkerProcessLauncherTests RequiredArchitecture = WorkerArchitecture.X86, StartupTimeoutSeconds = startupTimeoutSeconds, EventQueueCapacity = eventQueueCapacity, + WorkerHeartbeatSendIntervalSeconds = workerHeartbeatSendIntervalSeconds, + WorkerHeartbeatStuckCeilingSeconds = workerHeartbeatStuckCeilingSeconds, }, }; diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionOptionsTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionOptionsTests.cs new file mode 100644 index 00000000..80569913 --- /dev/null +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionOptionsTests.cs @@ -0,0 +1,72 @@ +using System; +using System.Collections.Generic; +using ZB.MOM.WW.MxGateway.Contracts; +using ZB.MOM.WW.MxGateway.Worker.Ipc; + +namespace ZB.MOM.WW.MxGateway.Worker.Tests.Ipc; + +/// +/// Covers : the gateway conveys the +/// worker's heartbeat send interval and watchdog stuck ceiling through the launch environment +/// (mxaccessgw#218), and a bad value must fall back rather than keep the worker from starting. +/// +public sealed class WorkerPipeSessionOptionsTests +{ + /// Verifies that launcher-provided values replace the compiled defaults. + [Fact] + public void FromEnvironment_WithLauncherValues_UsesThem() + { + WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup( + (WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, "3000"), + (WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, "240000"))); + + Assert.Equal(TimeSpan.FromSeconds(3), options.HeartbeatInterval); + Assert.Equal(TimeSpan.FromSeconds(240), options.HeartbeatStuckCeiling); + Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatGrace, options.HeartbeatGrace); + options.Validate(); + } + + /// Verifies that an absent environment yields the compiled defaults. + [Fact] + public void FromEnvironment_WithNothingSet_KeepsDefaults() + { + WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup()); + + Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatInterval, options.HeartbeatInterval); + Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling); + } + + /// + /// Verifies that unusable values fall back to the defaults, including a ceiling that would + /// fail by not exceeding the grace. + /// + /// Send-interval environment value. + /// Stuck-ceiling environment value. + [Theory] + [InlineData("0", "0")] + [InlineData("-5", "-5")] + [InlineData("abc", "1.5")] + [InlineData("", "15000")] + [InlineData("99999999999", "99999999999")] + public void FromEnvironment_WithUnusableValues_FallsBackToDefaults(string interval, string ceiling) + { + WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup( + (WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, interval), + (WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, ceiling))); + + Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatInterval, options.HeartbeatInterval); + Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling); + options.Validate(); + } + + private static Func Lookup(params (string Name, string Value)[] values) + { + Dictionary map = new(StringComparer.Ordinal); + foreach ((string name, string value) in values) + { + map[name] = value; + } + + return name => map.TryGetValue(name, out string? value) ? value : null; + } +} diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs index 7a128a14..200fb108 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs @@ -75,7 +75,7 @@ public sealed class WorkerPipeSession new WorkerFrameWriter(stream, options), options, () => Process.GetCurrentProcess().Id, - new WorkerPipeSessionOptions(), + WorkerPipeSessionOptions.FromEnvironment(), () => CreateDefaultRuntimeSession(WorkerCommandLogging.IsEnabled() ? logger : null), logger, stream) diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs index f8efe8df..35531bc9 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs @@ -1,4 +1,6 @@ using System; +using System.Globalization; +using ZB.MOM.WW.MxGateway.Contracts; namespace ZB.MOM.WW.MxGateway.Worker.Ipc; @@ -26,6 +28,56 @@ public sealed class WorkerPipeSessionOptions HeartbeatStuckCeiling = DefaultHeartbeatStuckCeiling; } + /// + /// Builds the production options from the launcher-provided environment + /// ( and + /// ), which the + /// gateway sets from MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds and + /// WorkerHeartbeatStuckCeilingSeconds. A missing, unparseable, or non-positive value + /// keeps its default, and so does a ceiling that would fail (not + /// above ): the gateway validates both at startup, and a + /// worker that refused to start over a stray environment value would turn a tuning + /// mistake into a session outage. The grace is not conveyed and keeps its default. + /// + /// + /// Environment lookup; reads the process environment. + /// + /// Options that always pass . + public static WorkerPipeSessionOptions FromEnvironment(Func? getEnvironmentVariable = null) + { + Func lookup = getEnvironmentVariable ?? Environment.GetEnvironmentVariable; + WorkerPipeSessionOptions options = new WorkerPipeSessionOptions(); + + if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, out TimeSpan interval)) + { + options.HeartbeatInterval = interval; + } + + if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, out TimeSpan ceiling) + && ceiling > options.HeartbeatGrace) + { + options.HeartbeatStuckCeiling = ceiling; + } + + return options; + } + + private static bool TryReadPositiveMilliseconds( + Func lookup, + string name, + out TimeSpan value) + { + value = TimeSpan.Zero; + if (!int.TryParse(lookup(name), NumberStyles.Integer, CultureInfo.InvariantCulture, out int milliseconds) + || milliseconds <= 0) + { + return false; + } + + value = TimeSpan.FromMilliseconds(milliseconds); + return true; + } + /// Gets or sets the heartbeat interval. public TimeSpan HeartbeatInterval { get; set; }