diff --git a/docs/GatewayConfiguration.md b/docs/GatewayConfiguration.md index a1364331..e27da7f3 100644 --- a/docs/GatewayConfiguration.md +++ b/docs/GatewayConfiguration.md @@ -155,7 +155,8 @@ launch CWD (SEC-01, SEC-33). | `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatCheckIntervalSeconds`, and strictly greater than `WorkerHeartbeatSendIntervalSeconds` — a grace at or below the worker's send interval declares a healthy worker dead between two heartbeats (mxaccessgw#218). | | `MxGateway:Worker:HeartbeatStuckCeilingSeconds` | `75` | Ceiling on the gateway's in-flight-command heartbeat skip. While a command is pending the gateway tolerates a stale heartbeat, because the worker STA is busy inside a COM call; once the oldest pending command is older than this, the watchdog faults the worker anyway so a truly stuck COM call cannot hide a dead worker. Must be greater than or equal to `HeartbeatGraceSeconds`. The worker's own watchdog has its separate ceiling, `WorkerHeartbeatStuckCeilingSeconds`. | | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` | `5` | Interval at which the worker sends heartbeats to the gateway. Must be between `1` and `86400`, and below `HeartbeatGraceSeconds`. Conveyed to the worker via `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS`; a missing or invalid value leaves the worker on 5 s. | -| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than the worker's fixed 15-second watchdog grace and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. | +| `MxGateway:Worker:WorkerHeartbeatGraceSeconds` | `15` | The worker's own STA-watchdog grace: how long STA activity may go stale before the worker faults `StaHung` (no command in flight). Must be between `1` and `86400` and below `WorkerHeartbeatStuckCeilingSeconds`; validation uses this configured value, not a copied constant (mxaccessgw#224). Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS`; a missing or invalid value, or a grace the ceiling does not exceed, leaves the worker on 15 s. | +| `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` | `75` | Ceiling on the worker's own STA watchdog: once STA activity has been stale this long, the worker faults `StaHung` even while a command is in flight. Must be greater than `WorkerHeartbeatGraceSeconds` and at most `86400`. Raise it together with `HeartbeatStuckCeilingSeconds` for deployments that run very long bulk operations. Conveyed via `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS`; a missing or invalid value leaves the worker on 75 s. | | `MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds` | `5000` | How long the gateway waits for a session's bounded event channel (`MxGateway:Events:QueueCapacity`) to accept a worker event before faulting the session with `QueueOverflow`. A consumer stalled for longer than this is treated as stuck rather than slow. Must be greater than zero. | | `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. | diff --git a/docs/MxAccessWorkerInstanceDesign.md b/docs/MxAccessWorkerInstanceDesign.md index 8f8a5cee..3f7f836d 100644 --- a/docs/MxAccessWorkerInstanceDesign.md +++ b/docs/MxAccessWorkerInstanceDesign.md @@ -102,6 +102,7 @@ MXGATEWAY_ALARM_POLL_INTERVAL_MS= MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH= MXGATEWAY_ALARM_ACK_CLIENT_NAME= MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS= +MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS= MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS= ``` diff --git a/docs/WorkerProcessLauncher.md b/docs/WorkerProcessLauncher.md index 97890ea3..b78739f4 100644 --- a/docs/WorkerProcessLauncher.md +++ b/docs/WorkerProcessLauncher.md @@ -40,7 +40,8 @@ worker. The launcher always exports every value below: | `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms | | `MXGATEWAY_ALARM_ACK_CLIENT_NAME` | `MxGateway:Alarms:AckClientName` | `MxGateway` | | `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS` | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` (× 1000) | `5000` ms | -| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's 15 s watchdog grace) | +| `MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS` | `MxGateway:Worker:WorkerHeartbeatGraceSeconds` (× 1000) | `15000` ms (also kept when the ceiling is not above it) | +| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's watchdog grace) | The write-completion wait applies to the four single-item write variants: `Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM diff --git a/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs b/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs index dc474a56..04037401 100644 --- a/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs +++ b/src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs @@ -42,4 +42,12 @@ public static class WorkerLaunchEnvironment /// MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds. /// public const string HeartbeatStuckCeilingMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS"; + + /// + /// Grace, in milliseconds, the worker's own STA watchdog allows STA activity to go stale + /// before faulting StaHung (when no command is in flight). Set from + /// MxGateway:Worker:WorkerHeartbeatGraceSeconds; the worker's stuck ceiling must + /// exceed it. + /// + public const string HeartbeatGraceMilliseconds = "MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS"; } diff --git a/src/ZB.MOM.WW.MxGateway.Server/Configuration/EffectiveWorkerConfiguration.cs b/src/ZB.MOM.WW.MxGateway.Server/Configuration/EffectiveWorkerConfiguration.cs index 5f7c75bd..970777ff 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Configuration/EffectiveWorkerConfiguration.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Configuration/EffectiveWorkerConfiguration.cs @@ -9,5 +9,8 @@ public sealed record EffectiveWorkerConfiguration( int HeartbeatCheckIntervalSeconds, int HeartbeatGraceSeconds, int HeartbeatStuckCeilingSeconds, + int WorkerHeartbeatSendIntervalSeconds, + int WorkerHeartbeatGraceSeconds, + int WorkerHeartbeatStuckCeilingSeconds, int EventChannelFullModeTimeoutMilliseconds, int MaxMessageBytes); diff --git a/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayConfigurationProvider.cs b/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayConfigurationProvider.cs index 416459f0..212c67d4 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayConfigurationProvider.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayConfigurationProvider.cs @@ -41,6 +41,9 @@ public sealed class GatewayConfigurationProvider(IOptions option HeartbeatCheckIntervalSeconds: value.Worker.ResolveHeartbeatCheckIntervalSeconds(), HeartbeatGraceSeconds: value.Worker.HeartbeatGraceSeconds, HeartbeatStuckCeilingSeconds: value.Worker.HeartbeatStuckCeilingSeconds, + WorkerHeartbeatSendIntervalSeconds: value.Worker.WorkerHeartbeatSendIntervalSeconds, + WorkerHeartbeatGraceSeconds: value.Worker.WorkerHeartbeatGraceSeconds, + WorkerHeartbeatStuckCeilingSeconds: value.Worker.WorkerHeartbeatStuckCeilingSeconds, EventChannelFullModeTimeoutMilliseconds: value.Worker.EventChannelFullModeTimeoutMilliseconds, MaxMessageBytes: value.Worker.MaxMessageBytes), Sessions: new EffectiveSessionConfiguration( diff --git a/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs b/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs index 25b59397..f8160c34 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs @@ -310,10 +310,23 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase MaximumWorkerHeartbeatSeconds) { builder.Add( - $"MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's {WorkerOptions.WorkerWatchdogGraceSeconds}-second watchdog grace."); + $"MxGateway:Worker:WorkerHeartbeatGraceSeconds must be at most {MaximumWorkerHeartbeatSeconds}."); + } + + // The worker refuses a ceiling at or below its own grace, so hold it above the configured + // grace (not a copied constant) -- mxaccessgw#224. + if (options.WorkerHeartbeatStuckCeilingSeconds <= options.WorkerHeartbeatGraceSeconds) + { + builder.Add( + "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than WorkerHeartbeatGraceSeconds."); } AddIfNotPositive( diff --git a/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs b/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs index 559079a0..ab8186cf 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs @@ -93,14 +93,17 @@ public sealed class WorkerOptions /// Default for . public const int DefaultWorkerHeartbeatStuckCeilingSeconds = 75; - /// - /// The worker's own STA-watchdog grace, in seconds. Not configurable: the worker keeps its - /// compiled default (WorkerPipeSessionOptions.DefaultHeartbeatGrace). Mirrored here - /// only so validation can hold above it, - /// which the worker requires. - /// - public const int WorkerWatchdogGraceSeconds = 15; + /// Default for . + public const int DefaultWorkerHeartbeatGraceSeconds = 15; + /// + /// The worker's own STA-watchdog grace, in seconds: how long STA activity may go stale + /// before the worker faults StaHung (no command in flight). Default 15. + /// must be greater than this, which + /// validation checks against the configured value rather than a copied constant. + /// Conveyed through MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS. + /// + public int WorkerHeartbeatGraceSeconds { get; init; } = DefaultWorkerHeartbeatGraceSeconds; /// /// Interval, in seconds, at which the worker sends heartbeats to the gateway. Default 5. /// must be greater than this, or the gateway declares @@ -112,8 +115,8 @@ public sealed class WorkerOptions /// /// Ceiling, in seconds, on the worker's own STA watchdog: once STA activity has been stale /// this long the worker faults StaHung even while a command is in flight. Default 75; - /// must be greater than the worker's fixed 15-second watchdog grace - /// (). Raise it, together with + /// must be greater than the worker's watchdog grace + /// (). Raise it, together with /// , for deployments that run very long bulk /// operations. Conveyed through MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS. /// diff --git a/src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/SettingsPage.razor b/src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/SettingsPage.razor index e98cb425..33bc0343 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/SettingsPage.razor +++ b/src/ZB.MOM.WW.MxGateway.Server/Dashboard/Components/Pages/SettingsPage.razor @@ -93,6 +93,9 @@ else Heartbeat grace@Snapshot.Configuration.Worker.HeartbeatGraceSeconds seconds Heartbeat check interval@Snapshot.Configuration.Worker.HeartbeatCheckIntervalSeconds seconds Heartbeat stuck ceiling@Snapshot.Configuration.Worker.HeartbeatStuckCeilingSeconds seconds + Worker heartbeat send interval@Snapshot.Configuration.Worker.WorkerHeartbeatSendIntervalSeconds seconds + Worker watchdog grace@Snapshot.Configuration.Worker.WorkerHeartbeatGraceSeconds seconds + Worker watchdog stuck ceiling@Snapshot.Configuration.Worker.WorkerHeartbeatStuckCeilingSeconds seconds Event channel full-mode timeout@Snapshot.Configuration.Worker.EventChannelFullModeTimeoutMilliseconds ms Default command timeout@Snapshot.Configuration.Sessions.DefaultCommandTimeoutSeconds seconds Max sessions@Snapshot.Configuration.Sessions.MaxSessions diff --git a/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs b/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs index 0bb322ba..a9c99db1 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs @@ -219,6 +219,8 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher startInfo.Environment[WorkerLaunchEnvironment.AlarmAckClientName] = _alarmsOptions.AckClientName; startInfo.Environment[WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds] = (_workerOptions.WorkerHeartbeatSendIntervalSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture); + startInfo.Environment[WorkerLaunchEnvironment.HeartbeatGraceMilliseconds] = + (_workerOptions.WorkerHeartbeatGraceSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture); startInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds] = (_workerOptions.WorkerHeartbeatStuckCeilingSeconds * 1000L).ToString(System.Globalization.CultureInfo.InvariantCulture); diff --git a/src/ZB.MOM.WW.MxGateway.Server/appsettings.json b/src/ZB.MOM.WW.MxGateway.Server/appsettings.json index b3da90a1..93489ca4 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/appsettings.json +++ b/src/ZB.MOM.WW.MxGateway.Server/appsettings.json @@ -53,6 +53,7 @@ "HeartbeatGraceSeconds": 15, "HeartbeatStuckCeilingSeconds": 75, "WorkerHeartbeatSendIntervalSeconds": 5, + "WorkerHeartbeatGraceSeconds": 15, "WorkerHeartbeatStuckCeilingSeconds": 75, "EventChannelFullModeTimeoutMilliseconds": 5000, "MaxMessageBytes": 16842752 diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs index 1b9d1237..8da8e75d 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs @@ -45,6 +45,7 @@ public sealed class GatewayOptionsTests Assert.Equal(75, options.Worker.HeartbeatStuckCeilingSeconds); Assert.Equal(5, options.Worker.WorkerHeartbeatSendIntervalSeconds); Assert.Equal(75, options.Worker.WorkerHeartbeatStuckCeilingSeconds); + Assert.Equal(15, options.Worker.WorkerHeartbeatGraceSeconds); Assert.Equal(5000, options.Worker.EventChannelFullModeTimeoutMilliseconds); Assert.Equal(15, options.Worker.HeartbeatGraceSeconds); // 16 MiB public gRPC cap plus the 64 KiB worker-frame envelope reserve for headroom. @@ -123,7 +124,10 @@ public sealed class GatewayOptionsTests [InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "15", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")] [InlineData("MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds must be at most 86400.")] [InlineData("MxGateway:Worker:HeartbeatGraceSeconds", "5", "MxGateway:Worker:HeartbeatGraceSeconds must be greater than WorkerHeartbeatSendIntervalSeconds.")] - [InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "15", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than the worker's 15-second watchdog grace.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "15", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than WorkerHeartbeatGraceSeconds.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatGraceSeconds", "75", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be greater than WorkerHeartbeatGraceSeconds.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatGraceSeconds", "0", "MxGateway:Worker:WorkerHeartbeatGraceSeconds must be greater than zero.")] + [InlineData("MxGateway:Worker:WorkerHeartbeatGraceSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatGraceSeconds must be at most 86400.")] [InlineData("MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds", "86401", "MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds must be at most 86400.")] [InlineData("MxGateway:Sessions:DefaultLeaseSeconds", "0", "MxGateway:Sessions:DefaultLeaseSeconds must be greater than zero.")] [InlineData("MxGateway:Sessions:LeaseSweepIntervalSeconds", "0", "MxGateway:Sessions:LeaseSweepIntervalSeconds must be greater than zero.")] diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SettingsPageTagVisibilityRenderTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SettingsPageTagVisibilityRenderTests.cs index 482e3bf0..09e1e9ab 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SettingsPageTagVisibilityRenderTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Dashboard/SettingsPageTagVisibilityRenderTests.cs @@ -120,6 +120,31 @@ public sealed class SettingsPageTagVisibilityRenderTests Assert.Contains("""Replay buffer max bytes987654321""", html, StringComparison.Ordinal); } + /// + /// The worker-side heartbeat send interval, watchdog grace and stuck ceiling (mxaccessgw#218, + /// #224) reach the page, each in its own labelled row, distinct from the gateway-side ones. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task SettingsPage_RendersWorkerHeartbeatOptions() + { + string html = await RenderAsync(new GatewayOptions + { + Worker = new WorkerOptions + { + HeartbeatStuckCeilingSeconds = 100, + WorkerHeartbeatSendIntervalSeconds = 3, + WorkerHeartbeatGraceSeconds = 31, + WorkerHeartbeatStuckCeilingSeconds = 241, + }, + }); + + Assert.Contains("""Worker heartbeat send interval3 seconds""", html, StringComparison.Ordinal); + Assert.Contains("""Worker watchdog grace31 seconds""", html, StringComparison.Ordinal); + Assert.Contains("""Worker watchdog stuck ceiling241 seconds""", html, StringComparison.Ordinal); + Assert.Contains("""Heartbeat stuck ceiling100 seconds""", html, StringComparison.Ordinal); + } + private static Task RenderAsync(DashboardOptions dashboard) => RenderAsync(new GatewayOptions { Dashboard = dashboard }); diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs index 256c8a85..41634073 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs @@ -24,6 +24,7 @@ public sealed class WorkerProcessLauncherTests WorkerLaunchEnvironment.MaxAlarmsPerFetch, WorkerLaunchEnvironment.AlarmAckClientName, WorkerLaunchEnvironment.HeartbeatSendIntervalMilliseconds, + WorkerLaunchEnvironment.HeartbeatGraceMilliseconds, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, ]; @@ -91,6 +92,10 @@ public sealed class WorkerProcessLauncherTests "75000", processFactory.LastStartInfo.Environment[ WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]); + Assert.Equal( + "15000", + processFactory.LastStartInfo.Environment[ + WorkerLaunchEnvironment.HeartbeatGraceMilliseconds]); Assert.All( RequiredLaunchEnvironmentVariables, variable => Assert.True( @@ -166,7 +171,8 @@ public sealed class WorkerProcessLauncherTests processFactory, new SucceedingStartupProbe(), workerHeartbeatSendIntervalSeconds: 3, - workerHeartbeatStuckCeilingSeconds: 240); + workerHeartbeatStuckCeilingSeconds: 240, + workerHeartbeatGraceSeconds: 30); using WorkerProcessHandle handle = await launcher.LaunchAsync(CreateRequest()); @@ -177,6 +183,9 @@ public sealed class WorkerProcessLauncherTests Assert.Equal( "240000", processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds]); + Assert.Equal( + "30000", + processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.HeartbeatGraceMilliseconds]); } /// Verifies that a failed startup probe kills and disposes the worker process. @@ -305,7 +314,8 @@ public sealed class WorkerProcessLauncherTests int eventQueueCapacity = 10000, string ackClientName = "MxGateway", int workerHeartbeatSendIntervalSeconds = WorkerOptions.DefaultWorkerHeartbeatSendIntervalSeconds, - int workerHeartbeatStuckCeilingSeconds = WorkerOptions.DefaultWorkerHeartbeatStuckCeilingSeconds) + int workerHeartbeatStuckCeilingSeconds = WorkerOptions.DefaultWorkerHeartbeatStuckCeilingSeconds, + int workerHeartbeatGraceSeconds = WorkerOptions.DefaultWorkerHeartbeatGraceSeconds) { GatewayOptions options = new() { @@ -318,6 +328,7 @@ public sealed class WorkerProcessLauncherTests EventQueueCapacity = eventQueueCapacity, WorkerHeartbeatSendIntervalSeconds = workerHeartbeatSendIntervalSeconds, WorkerHeartbeatStuckCeilingSeconds = workerHeartbeatStuckCeilingSeconds, + WorkerHeartbeatGraceSeconds = workerHeartbeatGraceSeconds, }, }; diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionOptionsTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionOptionsTests.cs index 80569913..af64f33e 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionOptionsTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionOptionsTests.cs @@ -26,6 +26,31 @@ public sealed class WorkerPipeSessionOptionsTests options.Validate(); } + /// Verifies that a launcher-provided grace replaces the compiled default (mxaccessgw#224). + [Fact] + public void FromEnvironment_WithGrace_UsesItAndStaysValid() + { + WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup( + (WorkerLaunchEnvironment.HeartbeatGraceMilliseconds, "30000"), + (WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, "120000"))); + + Assert.Equal(TimeSpan.FromSeconds(30), options.HeartbeatGrace); + Assert.Equal(TimeSpan.FromSeconds(120), options.HeartbeatStuckCeiling); + options.Validate(); + } + + /// Verifies that a grace the ceiling does not exceed is dropped rather than failing the worker. + [Fact] + public void FromEnvironment_WithGraceAtOrAboveCeiling_FallsBackToDefaultGrace() + { + WorkerPipeSessionOptions options = WorkerPipeSessionOptions.FromEnvironment(Lookup( + (WorkerLaunchEnvironment.HeartbeatGraceMilliseconds, "80000"))); + + Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatGrace, options.HeartbeatGrace); + Assert.Equal(WorkerPipeSessionOptions.DefaultHeartbeatStuckCeiling, options.HeartbeatStuckCeiling); + options.Validate(); + } + /// Verifies that an absent environment yields the compiled defaults. [Fact] public void FromEnvironment_WithNothingSet_KeepsDefaults() diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs index 35531bc9..a12e6ab8 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSessionOptions.cs @@ -37,7 +37,8 @@ public sealed class WorkerPipeSessionOptions /// keeps its default, and so does a ceiling that would fail (not /// above ): the gateway validates both at startup, and a /// worker that refused to start over a stray environment value would turn a tuning - /// mistake into a session outage. The grace is not conveyed and keeps its default. + /// mistake into a session outage. The watchdog grace is conveyed too + /// () and must stay below the ceiling. /// /// /// Environment lookup; reads the process environment. @@ -53,8 +54,25 @@ public sealed class WorkerPipeSessionOptions options.HeartbeatInterval = interval; } - if (TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, out TimeSpan ceiling) - && ceiling > options.HeartbeatGrace) + TimeSpan grace = options.HeartbeatGrace; + bool haveGrace = TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatGraceMilliseconds, out TimeSpan configuredGrace); + if (haveGrace) + { + grace = configuredGrace; + } + + bool haveCeiling = TryReadPositiveMilliseconds(lookup, WorkerLaunchEnvironment.HeartbeatStuckCeilingMilliseconds, out TimeSpan ceiling); + TimeSpan effectiveCeiling = haveCeiling ? ceiling : options.HeartbeatStuckCeiling; + + // Keep ceiling > grace so Validate() always passes. A grace the ceiling does not exceed + // is dropped (back to the default) rather than failing the worker; an unusable ceiling + // is dropped first when the default grace still leaves it valid. + if (effectiveCeiling > grace) + { + options.HeartbeatGrace = grace; + options.HeartbeatStuckCeiling = effectiveCeiling; + } + else if (haveCeiling && ceiling > options.HeartbeatGrace) { options.HeartbeatStuckCeiling = ceiling; }