feat(worker): log exception messages and per-command start/end (mxaccessgw#184)

R21-O3: WorkerApplication.Run's failure lines logged only the exception
type. Each now also carries exception_message, scrubbed by the new
WorkerLogRedactor.RedactMessage: the value of any name=value / name: value
pair whose name looks sensitive and any mxgw_ API key token are replaced,
line breaks are flattened and the text is capped at 512 characters. The
AppDomain unhandled-exception hook stays type-only, since an exception
from an arbitrary thread can carry command values.

R21-O7: StaCommandDispatcher.ExecuteQueuedCommandAsync logs
WorkerCommandStarted (correlation id, method, queued_ms) and
WorkerCommandCompleted (status, HRESULT, elapsed_ms) when a CommandLogger
is set. IWorkerLogger has no Debug level, so the lines are opt-in through
MXGATEWAY_WORKER_LOG_COMMANDS=1|true (WorkerCommandLogging), which the
worker inherits from the gateway process environment. No payloads or
diagnostic messages are logged. Docs updated. (mxaccessgw#184)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 12:56:29 -05:00
parent f49b7a0cc1
commit d9c60a78eb
11 changed files with 365 additions and 12 deletions
+8 -2
View File
@@ -124,7 +124,7 @@ public static WorkerBootstrapResult Failure(WorkerExitCode exitCode, IEnumerable
### Unhandled-exception hook
The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because they can carry tag values or credentials passed through a command, the same rule every other worker failure line follows. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global.
The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because an exception from an arbitrary thread (a COM callback, a command path) can carry tag values or credentials passed through a command. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global.
## Environment Abstraction
@@ -181,6 +181,12 @@ private void Write(
}
```
### Per-command start/end lines
`StaCommandDispatcher` can log one `WorkerCommandStarted` and one `WorkerCommandCompleted` line per STA command. Both carry `session_id`, `correlation_id`, and `command_method`; the start line adds `queued_ms` (time since the gateway enqueued the command) and the end line adds `protocol_status`, `hresult`, and `elapsed_ms` (time on the STA). Neither carries the command payload or the reply's diagnostic message, so tag values, secured-write credentials and provider detail stay out of the log.
The lines are off by default because a busy session dispatches many commands and `IWorkerLogger` has no Debug level to put them behind. Setting the environment variable `MXGATEWAY_WORKER_LOG_COMMANDS` to `1` or `true` turns them on (`WorkerCommandLogging`). The gateway launcher does not set it; a worker inherits it from the gateway process environment, so setting it on the gateway service enables it for every worker that service launches.
### What the redactor redacts and why
`gateway.md` "Security" requires that the worker never log raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations. The bootstrap nonce is also a credential: anyone who reads it can impersonate the worker to the gateway pipe. `WorkerLogRedactor` enforces this by replacing values whose field name contains any of these substrings (case-insensitive) with the literal `[redacted]`:
@@ -220,7 +226,7 @@ if (!result.Succeeded)
}
```
On success it logs `WorkerBootstrapSucceeded` with the session fields (the `nonce` field is redacted by `WorkerLogRedactor` because of its name), hands the `WorkerOptions` to `IWorkerPipeClient.RunAsync`, and waits synchronously. The `try/catch` around the pipe call maps a `WorkerFrameProtocolException` with `EndOfStream` to `PipeConnectionFailed` (a closed pipe is a lost connection, not a malformed frame), any other `WorkerFrameProtocolException` to `ProtocolViolation`, a `WorkerEventQueueFaultException` with category `QueueOverflow` to `EventQueueOverflow`, `IOException`/`TimeoutException` to `PipeConnectionFailed`, and any other exception to `UnexpectedFailure`, so every code path through `Run` returns one of the values in `WorkerExitCode`.
On success it logs `WorkerBootstrapSucceeded` with the session fields (the `nonce` field is redacted by `WorkerLogRedactor` because of its name), hands the `WorkerOptions` to `IWorkerPipeClient.RunAsync`, and waits synchronously. The `try/catch` around the pipe call maps a `WorkerFrameProtocolException` with `EndOfStream` to `PipeConnectionFailed` (a closed pipe is a lost connection, not a malformed frame), any other `WorkerFrameProtocolException` to `ProtocolViolation`, a `WorkerEventQueueFaultException` with category `QueueOverflow` to `EventQueueOverflow`, `IOException`/`TimeoutException` to `PipeConnectionFailed`, and any other exception to `UnexpectedFailure`, so every code path through `Run` returns one of the values in `WorkerExitCode`. Each of those failure lines carries `exit_code`, `exception_type`, and `exception_message`. The message is passed through `WorkerLogRedactor.RedactMessage` first, which scrubs the value of any `name=value` or `name: value` pair whose name looks sensitive (nonce, secret, password, token, credential, API key) and any `mxgw_` API key token, flattens line breaks, and caps the text at 512 characters. Field-name redaction alone cannot see inside free text.
## Related Documentation