Files
mxaccessgw/docs/WorkerProcessLauncher.md
T
Joseph Doherty 2e52f9f5e8
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m32s
ci / java (push) Successful in 2m57s
ci / portable (push) Failing after 5m1s
feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.

- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
  and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
  WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
  MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
  worker settings use.
- The worker's production WorkerPipeSession reads them through
  WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
  ceiling not above the worker's 15 s watchdog grace, keeps the default rather
  than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
  send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
  the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:24:05 -05:00

4.2 KiB
Raw Blame History

Worker Process Launcher

The gateway uses WorkerProcessLauncher to validate and start one worker process for a gateway session. The launcher owns process start semantics only; pipe handshaking and WorkerReady validation remain part of the worker client startup path.

Launch Inputs

WorkerProcessLaunchRequest carries the per-session bootstrap values:

  • SessionId,
  • PipeName,
  • ProtocolVersion,
  • Nonce,
  • optional PipeReservation cleanup handle.

The launcher passes SessionId, PipeName, and ProtocolVersion as command line arguments:

--session-id <sessionId> --pipe-name <pipeName> --protocol-version <version>

The launcher sets the nonce through the MXGATEWAY_WORKER_NONCE environment variable. The nonce is not included in WorkerProcessCommandLine so logs and diagnostics can report the launch command without exposing the secret.

All launch-variable names live in the shared Contracts WorkerLaunchEnvironment class and are consumed by both the launcher and the worker. The launcher always exports every value below:

Environment variable Gateway source Worker default when missing/invalid
MXGATEWAY_WORKER_NONCE WorkerProcessLaunchRequest.Nonce None; bootstrap fails because the nonce is required
MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS MxGateway:Worker:PipeConnectAttemptTimeoutMilliseconds 2000 ms
MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS MxGateway:Worker:WriteCompletionWaitMilliseconds 1500 ms
MXGATEWAY_EVENT_QUEUE_CAPACITY MxGateway:Worker:EventQueueCapacity 10000 events
MXGATEWAY_ALARM_POLL_INTERVAL_MS MxGateway:Alarms:PollIntervalMilliseconds 500 ms
MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH MxGateway:Alarms:MaxAlarmsPerFetch 1024 alarms
MXGATEWAY_ALARM_ACK_CLIENT_NAME MxGateway:Alarms:AckClientName MxGateway
MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds (× 1000) 5000 ms
MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds (× 1000) 75000 ms (also kept when the value is not above the worker's 15 s watchdog grace)

The write-completion wait applies to the four single-item write variants: Write, Write2, WriteSecured, and WriteSecured2. Bulk writes report COM acceptance per entry and do not wait for OnWriteComplete rows.

Validation And Cleanup

Before starting the process, the launcher validates that the configured worker path exists, has a .exe extension, contains a valid Windows Portable Executable header, and matches the configured RequiredArchitecture.

After the process starts, IWorkerStartupProbe waits for startup readiness. The default probe only verifies that the worker did not exit immediately. The worker client replaces this probe when pipe connection, hello, and WorkerReady handling are implemented.

The startup probe runs once, bounded by WorkerOptions.StartupTimeoutSeconds; it is not retried. The default probe only checks that the process is still alive, which cannot change on a retry, so the former StartupProbeRetryAttempts / StartupProbeRetryDelayMilliseconds options were removed (mxaccessgw#188). Stale keys in existing configuration are ignored by the binder. The worker pipe connect and handshake have their own retry loop.

The worker uses the exported pipe-attempt value inside its own bounded Polly retry loop.

If startup fails or exceeds WorkerOptions.StartupTimeoutSeconds, the launcher kills the worker process tree, disposes the process handle, disposes the optional pipe reservation, records a worker kill metric, and reports a WorkerProcessLaunchException.

Verification

Run the focused launcher tests after changing process launch behavior:

dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter WorkerProcessLauncherTests

Run the gateway build because the launcher is part of ZB.MOM.WW.MxGateway.Server:

dotnet build src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj