Files
mxaccessgw/docs/WorkerProcessLauncher.md
T
Joseph Doherty 2e52f9f5e8
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m32s
ci / java (push) Successful in 2m57s
ci / portable (push) Failing after 5m1s
feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.

- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
  and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
  WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
  MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
  worker settings use.
- The worker's production WorkerPipeSession reads them through
  WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
  ceiling not above the worker's 15 s watchdog grace, keeps the default rather
  than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
  send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
  the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:24:05 -05:00

93 lines
4.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Worker Process Launcher
The gateway uses `WorkerProcessLauncher` to validate and start one worker
process for a gateway session. The launcher owns process start semantics only;
pipe handshaking and `WorkerReady` validation remain part of the worker client
startup path.
## Launch Inputs
`WorkerProcessLaunchRequest` carries the per-session bootstrap values:
- `SessionId`,
- `PipeName`,
- `ProtocolVersion`,
- `Nonce`,
- optional `PipeReservation` cleanup handle.
The launcher passes `SessionId`, `PipeName`, and `ProtocolVersion` as command
line arguments:
```text
--session-id <sessionId> --pipe-name <pipeName> --protocol-version <version>
```
The launcher sets the nonce through the `MXGATEWAY_WORKER_NONCE` environment
variable. The nonce is not included in `WorkerProcessCommandLine` so logs and
diagnostics can report the launch command without exposing the secret.
All launch-variable names live in the shared Contracts
`WorkerLaunchEnvironment` class and are consumed by both the launcher and the
worker. The launcher always exports every value below:
| Environment variable | Gateway source | Worker default when missing/invalid |
|---|---|---|
| `MXGATEWAY_WORKER_NONCE` | `WorkerProcessLaunchRequest.Nonce` | None; bootstrap fails because the nonce is required |
| `MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS` | `MxGateway:Worker:PipeConnectAttemptTimeoutMilliseconds` | `2000` ms |
| `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` | `MxGateway:Worker:WriteCompletionWaitMilliseconds` | `1500` ms |
| `MXGATEWAY_EVENT_QUEUE_CAPACITY` | `MxGateway:Worker:EventQueueCapacity` | `10000` events |
| `MXGATEWAY_ALARM_POLL_INTERVAL_MS` | `MxGateway:Alarms:PollIntervalMilliseconds` | `500` ms |
| `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms |
| `MXGATEWAY_ALARM_ACK_CLIENT_NAME` | `MxGateway:Alarms:AckClientName` | `MxGateway` |
| `MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS` | `MxGateway:Worker:WorkerHeartbeatSendIntervalSeconds` (× 1000) | `5000` ms |
| `MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS` | `MxGateway:Worker:WorkerHeartbeatStuckCeilingSeconds` (× 1000) | `75000` ms (also kept when the value is not above the worker's 15 s watchdog grace) |
The write-completion wait applies to the four single-item write variants:
`Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM
acceptance per entry and do not wait for `OnWriteComplete` rows.
## Validation And Cleanup
Before starting the process, the launcher validates that the configured worker
path exists, has a `.exe` extension, contains a valid Windows Portable
Executable header, and matches the configured `RequiredArchitecture`.
After the process starts, `IWorkerStartupProbe` waits for startup readiness.
The default probe only verifies that the worker did not exit immediately. The
worker client replaces this probe when pipe connection, hello, and
`WorkerReady` handling are implemented.
The startup probe runs once, bounded by `WorkerOptions.StartupTimeoutSeconds`;
it is not retried. The default probe only checks that the process is still
alive, which cannot change on a retry, so the former
`StartupProbeRetryAttempts` / `StartupProbeRetryDelayMilliseconds` options were
removed (mxaccessgw#188). Stale keys in existing configuration are ignored by
the binder. The worker pipe connect and handshake have their own retry loop.
The worker uses the exported pipe-attempt value inside its own bounded Polly
retry loop.
If startup fails or exceeds `WorkerOptions.StartupTimeoutSeconds`, the launcher
kills the worker process tree, disposes the process handle, disposes the
optional pipe reservation, records a worker kill metric, and reports a
`WorkerProcessLaunchException`.
## Verification
Run the focused launcher tests after changing process launch behavior:
```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter WorkerProcessLauncherTests
```
Run the gateway build because the launcher is part of `ZB.MOM.WW.MxGateway.Server`:
```bash
dotnet build src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj
```
## Related Documentation
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
- [Worker Frame Protocol](./WorkerFrameProtocol.md)