diff --git a/docs/MxAccessWorkerInstanceDesign.md b/docs/MxAccessWorkerInstanceDesign.md index 307b84b9..92f0e37e 100644 --- a/docs/MxAccessWorkerInstanceDesign.md +++ b/docs/MxAccessWorkerInstanceDesign.md @@ -1162,7 +1162,9 @@ Worker logs should include: - protocol version, - gateway handshake result, - MXAccess COM creation result, -- command start/end with correlation id, +- command start/end with correlation id (`WorkerCommandStarted` / + `WorkerCommandCompleted`, opt-in through `MXGATEWAY_WORKER_LOG_COMMANDS`; see + `docs/WorkerBootstrap.md`), - HRESULT/status summary, - event family and sequence, - queue overflow, diff --git a/docs/WorkerBootstrap.md b/docs/WorkerBootstrap.md index 58557019..c66f23c8 100644 --- a/docs/WorkerBootstrap.md +++ b/docs/WorkerBootstrap.md @@ -124,7 +124,7 @@ public static WorkerBootstrapResult Failure(WorkerExitCode exitCode, IEnumerable ### Unhandled-exception hook -The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because they can carry tag values or credentials passed through a command, the same rule every other worker failure line follows. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global. +The production entry point, `WorkerApplication.Run(string[])`, subscribes `AppDomain.CurrentDomain.UnhandledException` before it parses arguments. The handler (`WorkerApplication.HandleUnhandledException`) logs one `WorkerUnhandledException` error line and calls `Environment.Exit(7)`. Without it, an exception on a thread that `Run` does not own ended the process with the CLR's crash code and no worker log line, so the gateway saw an unexplained exit. The line carries `exit_code`, `exception_type`, and `is_terminating` only: the exception message and stack are left out because an exception from an arbitrary thread (a COM callback, a command path) can carry tag values or credentials passed through a command. A failure inside the logger is swallowed so the exit always happens. The injectable `Run` overloads that tests call do not install the hook, because it is process-global. ## Environment Abstraction @@ -181,6 +181,12 @@ private void Write( } ``` +### Per-command start/end lines + +`StaCommandDispatcher` can log one `WorkerCommandStarted` and one `WorkerCommandCompleted` line per STA command. Both carry `session_id`, `correlation_id`, and `command_method`; the start line adds `queued_ms` (time since the gateway enqueued the command) and the end line adds `protocol_status`, `hresult`, and `elapsed_ms` (time on the STA). Neither carries the command payload or the reply's diagnostic message, so tag values, secured-write credentials and provider detail stay out of the log. + +The lines are off by default because a busy session dispatches many commands and `IWorkerLogger` has no Debug level to put them behind. Setting the environment variable `MXGATEWAY_WORKER_LOG_COMMANDS` to `1` or `true` turns them on (`WorkerCommandLogging`). The gateway launcher does not set it; a worker inherits it from the gateway process environment, so setting it on the gateway service enables it for every worker that service launches. + ### What the redactor redacts and why `gateway.md` "Security" requires that the worker never log raw credential values for `AuthenticateUser`, `WriteSecured`, or related secured operations. The bootstrap nonce is also a credential: anyone who reads it can impersonate the worker to the gateway pipe. `WorkerLogRedactor` enforces this by replacing values whose field name contains any of these substrings (case-insensitive) with the literal `[redacted]`: @@ -220,7 +226,7 @@ if (!result.Succeeded) } ``` -On success it logs `WorkerBootstrapSucceeded` with the session fields (the `nonce` field is redacted by `WorkerLogRedactor` because of its name), hands the `WorkerOptions` to `IWorkerPipeClient.RunAsync`, and waits synchronously. The `try/catch` around the pipe call maps a `WorkerFrameProtocolException` with `EndOfStream` to `PipeConnectionFailed` (a closed pipe is a lost connection, not a malformed frame), any other `WorkerFrameProtocolException` to `ProtocolViolation`, a `WorkerEventQueueFaultException` with category `QueueOverflow` to `EventQueueOverflow`, `IOException`/`TimeoutException` to `PipeConnectionFailed`, and any other exception to `UnexpectedFailure`, so every code path through `Run` returns one of the values in `WorkerExitCode`. +On success it logs `WorkerBootstrapSucceeded` with the session fields (the `nonce` field is redacted by `WorkerLogRedactor` because of its name), hands the `WorkerOptions` to `IWorkerPipeClient.RunAsync`, and waits synchronously. The `try/catch` around the pipe call maps a `WorkerFrameProtocolException` with `EndOfStream` to `PipeConnectionFailed` (a closed pipe is a lost connection, not a malformed frame), any other `WorkerFrameProtocolException` to `ProtocolViolation`, a `WorkerEventQueueFaultException` with category `QueueOverflow` to `EventQueueOverflow`, `IOException`/`TimeoutException` to `PipeConnectionFailed`, and any other exception to `UnexpectedFailure`, so every code path through `Run` returns one of the values in `WorkerExitCode`. Each of those failure lines carries `exit_code`, `exception_type`, and `exception_message`. The message is passed through `WorkerLogRedactor.RedactMessage` first, which scrubs the value of any `name=value` or `name: value` pair whose name looks sensitive (nonce, secret, password, token, credential, API key) and any `mxgw_` API key token, flattens line breaks, and caps the text at 512 characters. Field-name redaction alone cannot see inside free text. ## Related Documentation diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerApplicationTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerApplicationTests.cs index 3d915791..3380ca33 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerApplicationTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerApplicationTests.cs @@ -102,6 +102,47 @@ public sealed class WorkerApplicationTests Assert.Equal((int)WorkerExitCode.ProtocolViolation, exitCode); Assert.Equal("WorkerPipeProtocolFailure", logger.Entries[1].EventName); + Assert.Equal("Bad nonce.", logger.Entries[1].Fields["exception_message"]); + } + + /// + /// Every failure line in Run carries the exception message, scrubbed by + /// so a credential embedded in it never + /// reaches the log (mxaccessgw#184). + /// + /// Which catch block the thrown exception lands in. + [Theory] + [InlineData("io")] + [InlineData("protocol")] + [InlineData("endofstream")] + [InlineData("overflow")] + [InlineData("unexpected")] + public void Run_WhenPipeClientFails_LogsRedactedExceptionMessage(string exceptionKind) + { + const string message = "pipe failed password=hunter2 for mxgw_dev_s3cr3t"; + Exception exception = exceptionKind switch + { + "io" => new System.IO.IOException(message), + "protocol" => new WorkerFrameProtocolException(WorkerFrameProtocolErrorCode.InvalidEnvelope, message), + "endofstream" => new WorkerFrameProtocolException(WorkerFrameProtocolErrorCode.EndOfStream, message), + "overflow" => new WorkerEventQueueFaultException(WorkerFaultCategory.QueueOverflow, message), + _ => new InvalidOperationException(message), + }; + MemoryWorkerLogger logger = new(); + + ZB.MOM.WW.MxGateway.Worker.WorkerApplication.Run( + ValidArgs(), + CreateEnvironment("nonce-secret"), + logger, + new ThrowingPipeClient(exception)); + + MemoryWorkerLogEntry entry = logger.Entries[1]; + Assert.Equal("Error", entry.Level); + string logged = Assert.IsType(entry.Fields["exception_message"]); + Assert.Contains("pipe failed", logged); + Assert.Contains("password=[redacted]", logged); + Assert.DoesNotContain("hunter2", logged); + Assert.DoesNotContain("s3cr3t", logged); } /// diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerLogRedactorTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerLogRedactorTests.cs index b7a93dfd..d7064c23 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerLogRedactorTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Bootstrap/WorkerLogRedactorTests.cs @@ -44,4 +44,35 @@ public sealed class WorkerLogRedactorTests Assert.Equal(WorkerLogRedactor.RedactedValue, WorkerLogRedactor.RedactValue("password_value", "secret")); Assert.Equal(WorkerLogRedactor.RedactedValue, WorkerLogRedactor.RedactValue("secured_write_token", "secret")); } + + /// + /// Verifies scrubs credential values and API key + /// tokens out of free text while keeping the rest of the message. + /// + [Fact] + public void RedactMessage_ScrubsSensitiveAssignmentsAndApiKeyTokens() + { + string redacted = WorkerLogRedactor.RedactMessage( + "Login failed: Password=\"hunter 2\", api_key: abc123; token=xyz for mxgw_dev_s3cr3t on tag Area1.Tank"); + + Assert.DoesNotContain("hunter", redacted); + Assert.DoesNotContain("abc123", redacted); + Assert.DoesNotContain("xyz", redacted); + Assert.DoesNotContain("s3cr3t", redacted); + Assert.Contains("Password=[redacted]", redacted); + Assert.Contains("api_key: [redacted]", redacted); + Assert.Contains("Area1.Tank", redacted); + } + + /// Verifies messages are flattened to one line, capped, and null-safe. + [Fact] + public void RedactMessage_FlattensLineBreaksCapsLengthAndHandlesNull() + { + Assert.Equal(string.Empty, WorkerLogRedactor.RedactMessage(null)); + Assert.Equal("a b c", WorkerLogRedactor.RedactMessage("a\r\nb\nc")); + + string capped = WorkerLogRedactor.RedactMessage(new string('x', WorkerLogRedactor.MaxMessageLength + 100)); + Assert.StartsWith(new string('x', WorkerLogRedactor.MaxMessageLength), capped); + Assert.EndsWith("...[truncated]", capped); + } } diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Sta/StaCommandDispatcherTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Sta/StaCommandDispatcherTests.cs index ee189c6d..e3dffbef 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Sta/StaCommandDispatcherTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Sta/StaCommandDispatcherTests.cs @@ -5,7 +5,9 @@ using System.Runtime.InteropServices; using System.Threading; using System.Threading.Tasks; using ZB.MOM.WW.MxGateway.Contracts.Proto; +using ZB.MOM.WW.MxGateway.Worker.Bootstrap; using ZB.MOM.WW.MxGateway.Worker.Sta; +using ZB.MOM.WW.MxGateway.Worker.Tests.Bootstrap; using ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport; namespace ZB.MOM.WW.MxGateway.Worker.Tests.Sta; @@ -191,6 +193,85 @@ public sealed class StaCommandDispatcherTests await Task.WhenAll(current, pending); } + /// + /// With a command logger set, each STA command logs a start line and an end line carrying its + /// correlation id, method, status summary and latency (mxaccessgw#184). + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task DispatchAsync_WithCommandLogger_LogsStartAndEndWithCorrelationIdAndLatency() + { + using StaRuntime runtime = CreateRuntime(); + runtime.Start(); + MemoryWorkerLogger logger = new(); + StaCommandDispatcher dispatcher = new(runtime, new RecordingCommandExecutor()) + { + CommandLogger = logger, + }; + + await dispatcher.DispatchAsync(CreateCommand("correlation-1", MxCommandKind.Register)); + + Assert.Equal(2, logger.Entries.Count); + MemoryWorkerLogEntry started = logger.Entries[0]; + Assert.Equal("WorkerCommandStarted", started.EventName); + Assert.Equal("Information", started.Level); + Assert.Equal("correlation-1", started.Fields["correlation_id"]); + Assert.Equal("Register", started.Fields["command_method"]); + Assert.Equal("session-1", started.Fields["session_id"]); + MemoryWorkerLogEntry completed = logger.Entries[1]; + Assert.Equal("WorkerCommandCompleted", completed.EventName); + Assert.Equal("correlation-1", completed.Fields["correlation_id"]); + Assert.Equal("Register", completed.Fields["command_method"]); + Assert.Equal(ProtocolStatusCode.Ok.ToString(), completed.Fields["protocol_status"]); + double elapsed = Assert.IsType(completed.Fields["elapsed_ms"]); + Assert.True(elapsed >= 0); + } + + /// + /// The end line of a failed command carries the status summary and HRESULT but never the + /// exception's message, which can echo provider detail. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task DispatchAsync_WithCommandLogger_WhenExecutorThrows_LogsStatusWithoutMessage() + { + using StaRuntime runtime = CreateRuntime(); + runtime.Start(); + MemoryWorkerLogger logger = new(); + StaCommandDispatcher dispatcher = new( + runtime, + new ThrowingCommandExecutor(new COMException("provider detail", unchecked((int)0x80070057)))) + { + CommandLogger = logger, + }; + + await dispatcher.DispatchAsync(CreateCommand("correlation-1", MxCommandKind.Register)); + + MemoryWorkerLogEntry completed = Assert.Single(logger.Entries, entry => entry.EventName == "WorkerCommandCompleted"); + Assert.Equal(ProtocolStatusCode.MxaccessFailure.ToString(), completed.Fields["protocol_status"]); + Assert.Equal("0x80070057", completed.Fields["hresult"]); + Assert.DoesNotContain( + logger.Entries.SelectMany(entry => entry.Fields.Values), + value => value?.ToString()?.Contains("provider detail") == true); + } + + /// Verifies the opt-in environment value is parsed strictly. + /// Raw environment value. + /// Whether logging should be enabled. + [Theory] + [InlineData(null, false)] + [InlineData("", false)] + [InlineData("0", false)] + [InlineData("false", false)] + [InlineData("yes", false)] + [InlineData("1", true)] + [InlineData("true", true)] + [InlineData(" TRUE ", true)] + public void WorkerCommandLogging_IsEnabledValue_AcceptsOnlyOneOrTrue(string? value, bool expected) + { + Assert.Equal(expected, WorkerCommandLogging.IsEnabledValue(value)); + } + private static StaCommand CreateCommand( string correlationId, MxCommandKind kind, diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerCommandLogging.cs b/src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerCommandLogging.cs new file mode 100644 index 00000000..d386e30e --- /dev/null +++ b/src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerCommandLogging.cs @@ -0,0 +1,36 @@ +using System; + +namespace ZB.MOM.WW.MxGateway.Worker.Bootstrap; + +/// +/// Gate for the per-command WorkerCommandStarted / WorkerCommandCompleted log +/// lines. has no Debug level, and a pair of lines for every STA +/// command is too much for the default log, so the lines are opt-in through an environment +/// variable. The gateway's launcher does not set it; the worker inherits it from the gateway +/// process environment, so setting it on the gateway service turns the lines on for every worker +/// that service launches. +/// +public static class WorkerCommandLogging +{ + /// + /// Environment variable that enables per-command logging when set to 1 or + /// true (case-insensitive). Any other value, or none, leaves it off. + /// + public const string EnvironmentVariable = "MXGATEWAY_WORKER_LOG_COMMANDS"; + + /// Gets whether per-command logging is enabled for this process. + /// true when enables it. + public static bool IsEnabled() + { + return IsEnabledValue(Environment.GetEnvironmentVariable(EnvironmentVariable)); + } + + /// Interprets a raw value. + /// The variable's value, or null when unset. + /// true for 1 or true; otherwise false. + public static bool IsEnabledValue(string? value) + { + string trimmed = value?.Trim() ?? string.Empty; + return trimmed == "1" || string.Equals(trimmed, "true", StringComparison.OrdinalIgnoreCase); + } +} diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerLogRedactor.cs b/src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerLogRedactor.cs index cfbef582..dbecbfab 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerLogRedactor.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Bootstrap/WorkerLogRedactor.cs @@ -1,5 +1,6 @@ using System; using System.Collections.Generic; +using System.Text.RegularExpressions; namespace ZB.MOM.WW.MxGateway.Worker.Bootstrap; @@ -24,6 +25,50 @@ public static class WorkerLogRedactor "api_key", ]; + /// + /// Longest exception message passes through; the rest is cut and + /// marked, so one runaway message cannot flood the worker's stderr log. + /// + public const int MaxMessageLength = 512; + + // A "name=value" or "name: value" pair whose name contains a sensitive part, e.g. + // "password=hunter2" or "ApiKey: mxgw_k_s". The name is kept; the value is replaced. + private static readonly Regex SensitiveAssignmentPattern = new( + @"(?[\w.-]*(?:nonce|secret|password|token|credential|apikey|api_key)[\w.-]*)(?\s*[=:]\s*)(?""[^""]*""|'[^']*'|[^\s,;]+)", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant | RegexOptions.Compiled); + + // A gateway API key token (mxgw__) anywhere in the text. + private static readonly Regex ApiKeyTokenPattern = new( + @"mxgw_[A-Za-z0-9_-]+", + RegexOptions.CultureInvariant | RegexOptions.Compiled); + + /// + /// Makes an exception message safe to log. Field-name redaction cannot see inside a free-text + /// message, so this scrubs the content: the value of any name=value or + /// name: value pair whose name looks sensitive, and any gateway API key token. Line + /// breaks are flattened so the message stays on one log line, and the result is capped at + /// characters. + /// + /// Exception message to scrub; null yields an empty string. + /// The scrubbed message. + public static string RedactMessage(string? message) + { + if (string.IsNullOrEmpty(message)) + { + return string.Empty; + } + + string redacted = SensitiveAssignmentPattern.Replace( + message, + match => match.Groups["name"].Value + match.Groups["sep"].Value + RedactedValue); + redacted = ApiKeyTokenPattern.Replace(redacted, RedactedValue); + redacted = redacted.Replace("\r\n", " ").Replace('\r', ' ').Replace('\n', ' '); + + return redacted.Length <= MaxMessageLength + ? redacted + : redacted.Substring(0, MaxMessageLength) + "...[truncated]"; + } + /// /// Redacts sensitive field values from a log field dictionary. /// diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs index 8b2804f7..799616bb 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs @@ -76,7 +76,7 @@ public sealed class WorkerPipeSession options, () => Process.GetCurrentProcess().Id, new WorkerPipeSessionOptions(), - () => new MxAccessStaSession((eq, affinity, comFactory, pump) => new AlarmCommandHandler(eq, () => new WnWrapAlarmConsumer(), affinity, comFactory, standbyFactory: null, ackPumpStep: pump)), + () => CreateDefaultRuntimeSession(WorkerCommandLogging.IsEnabled() ? logger : null), logger, stream) { @@ -98,7 +98,7 @@ public sealed class WorkerPipeSession options, processIdProvider, new WorkerPipeSessionOptions(), - () => new MxAccessStaSession((eq, affinity, comFactory, pump) => new AlarmCommandHandler(eq, () => new WnWrapAlarmConsumer(), affinity, comFactory, standbyFactory: null, ackPumpStep: pump)), + () => CreateDefaultRuntimeSession(commandLogger: null), logger: null) { } @@ -138,6 +138,19 @@ public sealed class WorkerPipeSession _sessionOptions.Validate(); } + /// + /// Creates the production runtime session: MXAccess on its own STA with the wnwrap alarm + /// consumer. receives the per-command start/end lines when + /// non-null; the caller decides that through . + /// + private static MxAccessStaSession CreateDefaultRuntimeSession(IWorkerLogger? commandLogger) + { + return new MxAccessStaSession((eq, affinity, comFactory, pump) => new AlarmCommandHandler(eq, () => new WnWrapAlarmConsumer(), affinity, comFactory, standbyFactory: null, ackPumpStep: pump)) + { + CommandLogger = commandLogger, + }; + } + /// Runs the worker session, completing the handshake and processing messages until cancellation. /// Token to cancel the asynchronous operation. /// A task that represents the asynchronous operation. diff --git a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessStaSession.cs b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessStaSession.cs index c66baaa4..505c533e 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessStaSession.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessStaSession.cs @@ -251,6 +251,14 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession /// internal Action? AlarmPollWarningSink { get; set; } + /// + /// Logger handed to the STA command dispatcher for the per-command start/end lines, or + /// null to leave them off (see ). Set + /// before , which creates the + /// dispatcher. + /// + internal Bootstrap.IWorkerLogger? CommandLogger { get; set; } + /// /// Resolves the write-completion wait from the launcher-provided /// environment variable; a missing or invalid value falls back to @@ -363,7 +371,10 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession // closes over staRuntime so it always pumps the pump // tied to the apartment that owns this session. pumpStep: () => staRuntime.PumpPendingMessages(), - writeCompletionTimeout: WriteCompletionTimeout)); + writeCompletionTimeout: WriteCompletionTimeout)) + { + CommandLogger = this.CommandLogger, + }; return session.CreateWorkerReady(workerProcessId); }, diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Sta/StaCommandDispatcher.cs b/src/ZB.MOM.WW.MxGateway.Worker/Sta/StaCommandDispatcher.cs index 834c49d7..9ef6a74f 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Sta/StaCommandDispatcher.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Sta/StaCommandDispatcher.cs @@ -1,7 +1,9 @@ using System; using System.Collections.Generic; +using System.Diagnostics; using System.Threading.Tasks; using ZB.MOM.WW.MxGateway.Contracts.Proto; +using ZB.MOM.WW.MxGateway.Worker.Bootstrap; using ZB.MOM.WW.MxGateway.Worker.Conversion; namespace ZB.MOM.WW.MxGateway.Worker.Sta; @@ -72,6 +74,14 @@ public sealed class StaCommandDispatcher this.maxPendingCommands = maxPendingCommands; } + /// + /// Gets or sets the logger for the per-command WorkerCommandStarted / + /// WorkerCommandCompleted lines. null (the default) turns them off: they are one + /// pair per STA command, so they are opt-in rather than always-on (see + /// ). Set it before commands are dispatched. + /// + public IWorkerLogger? CommandLogger { get; set; } + /// /// Gets the count of pending commands in the queue. /// @@ -263,22 +273,90 @@ public sealed class StaCommandDispatcher } SetCurrentCommand(command.CorrelationId); + IWorkerLogger? logger = CommandLogger; + Stopwatch? stopwatch = null; + if (logger is not null) + { + LogCommandStarted(logger, command); + stopwatch = Stopwatch.StartNew(); + } + + MxCommandReply completedReply; try { MxCommandReply reply = await staRuntime .InvokeAsync(() => commandExecutor.Execute(command)) .ConfigureAwait(false); - queuedCommand.Complete(NormalizeReply(command, reply)); + completedReply = NormalizeReply(command, reply); } catch (Exception exception) { - queuedCommand.Complete(CreateExceptionReply(command, exception)); + completedReply = CreateExceptionReply(command, exception); } finally { SetCurrentCommand(string.Empty); } + + if (logger is not null && stopwatch is not null) + { + LogCommandCompleted(logger, command, completedReply, stopwatch.Elapsed); + } + + queuedCommand.Complete(completedReply); + } + + // Identity and timing only: never the command payload, whose values (and WriteSecured + // credentials) must not reach the log. + private static void LogCommandStarted(IWorkerLogger logger, StaCommand command) + { + Dictionary fields = new() + { + ["session_id"] = command.SessionId, + ["correlation_id"] = command.CorrelationId, + ["command_method"] = command.MethodName, + }; + if (command.EnqueueTimestamp is not null) + { + double queuedMilliseconds = (DateTime.UtcNow - command.EnqueueTimestamp.ToDateTime()).TotalMilliseconds; + fields["queued_ms"] = Math.Max(0, Math.Round(queuedMilliseconds, 3)); + } + + TryLog(() => logger.Information("WorkerCommandStarted", fields)); + } + + // Status summary only: the protocol status code and HRESULT, not the diagnostic message, + // which can echo provider detail. + private static void LogCommandCompleted( + IWorkerLogger logger, + StaCommand command, + MxCommandReply reply, + TimeSpan elapsed) + { + Dictionary fields = new() + { + ["session_id"] = command.SessionId, + ["correlation_id"] = command.CorrelationId, + ["command_method"] = command.MethodName, + ["protocol_status"] = (reply.ProtocolStatus?.Code ?? ProtocolStatusCode.Unspecified).ToString(), + ["hresult"] = reply.HasHresult ? $"0x{reply.Hresult:X8}" : string.Empty, + ["elapsed_ms"] = Math.Round(elapsed.TotalMilliseconds, 3), + }; + + TryLog(() => logger.Information("WorkerCommandCompleted", fields)); + } + + // A diagnostics line must never fail the command it describes. + private static void TryLog(Action log) + { + try + { + log(); + } + catch + { + } } private void SetCurrentCommand(string correlationId) diff --git a/src/ZB.MOM.WW.MxGateway.Worker/WorkerApplication.cs b/src/ZB.MOM.WW.MxGateway.Worker/WorkerApplication.cs index 28e676d5..f7a55d6e 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/WorkerApplication.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/WorkerApplication.cs @@ -40,10 +40,14 @@ public static class WorkerApplication /// CLR's crash code. /// /// - /// Logs the exception type only — never its message or stack, which - /// can carry tag values or credentials passed through a command — the - /// same rule every other worker failure line follows. A logging - /// failure is swallowed so the exit always happens. + /// Logs the exception type only — never its message or stack. An + /// exception reaching this hook can come from any thread, including a + /// COM callback or a command path whose message can carry tag values + /// or credentials, so it gets none of the content that + /// 's + /// own catch blocks log through + /// (those see only pipe-session failures). A logging failure is + /// swallowed so the exit always happens. /// /// Worker logger. /// The unhandled exception object (not necessarily an ). @@ -171,6 +175,7 @@ public static class WorkerApplication ["exit_code"] = WorkerExitCode.PipeConnectionFailed, ["error_code"] = exception.ErrorCode, ["exception_type"] = exception.GetType().FullName, + ["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message), }); return (int)WorkerExitCode.PipeConnectionFailed; @@ -182,6 +187,7 @@ public static class WorkerApplication ["exit_code"] = WorkerExitCode.ProtocolViolation, ["error_code"] = exception.ErrorCode, ["exception_type"] = exception.GetType().FullName, + ["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message), }); return (int)WorkerExitCode.ProtocolViolation; @@ -193,6 +199,7 @@ public static class WorkerApplication { ["exit_code"] = WorkerExitCode.EventQueueOverflow, ["exception_type"] = exception.GetType().FullName, + ["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message), }); return (int)WorkerExitCode.EventQueueOverflow; @@ -203,6 +210,7 @@ public static class WorkerApplication { ["exit_code"] = WorkerExitCode.PipeConnectionFailed, ["exception_type"] = exception.GetType().FullName, + ["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message), }); return (int)WorkerExitCode.PipeConnectionFailed; @@ -213,6 +221,7 @@ public static class WorkerApplication { ["exit_code"] = WorkerExitCode.UnexpectedFailure, ["exception_type"] = exception.GetType().FullName, + ["exception_message"] = WorkerLogRedactor.RedactMessage(exception.Message), }); return (int)WorkerExitCode.UnexpectedFailure;